Developing for non-x86 targets using QEMU

Introduction

Emulation allows even casual hobbyist developers to build and test the software they write on multiple hardware platforms from the comfort of their own laptop.

QEMU is rapidly becoming a category killer in open source emulation software, capable of not only booting a Knoppix CD in a window but booting Linux systems built for ARM, MIPS, PowerPC, SPARC, sh4, and more.

This talk covers application vs system emulation, native vs cross compiling (and combining the two with distcc), using QEMU, setting up an emulated development environment, real world scalability issues, using the Amazon EC2 Cloud, and building a monster server for under $3k.

Advantages of cross-compiling

• Why would you want to cross-compile?
• Fast
  • About as fast as native compiling on the host
  • Same SMP Scalability
• Prerequisites are common
  • Just software, common PC Hardware and Linux OS
• Unavoidable
  • Someone somewhere has to do a certain amount of cross-compiling in order to get a new target up and running
  • Minimal subset must be supported for each target
  • Can you build a current Linux 2.6 Kernel on an Alpha Machine running a 2.2 Kernel with GCC 2.95? If not, you need to cross-compile from a newer system.

Disadvantages of cross-compiling

• Keeping Multiple Build Contexts Straight
  • Information leaks between the target and the host
  • Each toolchain has its own headers and libraries
  • Different things in $PATH depending on target or host
  • Hard to remove things the host has, but the target doesn't
    • Install gzip on the host, but not in the target
  • uname -m
    • Machine type given by the host may not equal target
  • Environment Variables
  • Looking at /proc
• Lying to Configure
  • The design of configure is fundamentally wrong for cross-compiling
  • Asking questions about the host to build programs for the target is crazy when the host and the target are not the same
  • libtool, pkg-config, finding Python, wrong signal numbers for Perl...
• Hard to test the result
  • The test suite is often part of the source. You didn't build the source on the target, so you have to copy the source tarball to the target hardware. Then to run its test suite you need to cross-compile tcl/expect and install it to the target's root filesystem.
• Do you want to ship that?
  • Will you bother?
• Bad Error Reporting
  • Using the wrong "strip" mostly works... until you build for sh4 target.
  • The Perl thing mentioned earlier shipped to a customer. It built, it ran, it just didn't work in this one area.
• Not all packages cross-compile
  • Most developers barely care about native compiling on non-x86, let alone cross-compiling
    • Implementing cross compiling complicates the heck out of the build system for everybody, to serve less than 10% of the userbase
    • Less than 10% of the userbase will ever test it, so it'll break a lot
• The package developer can't reproduce your bug.
  • Developers haven't got a cross compiling build environment set up for every possible target. Installing a binary-only toolchain requiring a root login on their development machine isn't appealing
  • Developers haven't got a test environment to run the result. Building a binary they can't run proves nothing and isn't very interesting.
  • The packages that do cross compile don't do it the same way, it's not just "./configure; make; make install".
• Maintenance
  • Each new release breaks
  • Discourages upgrading
  • MIPS was broken in the 2.6.30 kernel
  • sh4 is broken in 2.6.31
  • The kernel is the poster child for many eyeballs
  • Non-x86 gets much less attention than x86 at the best of times
  • Now add 100 different build environments you could be cross compiling from. Add enough variables (which toolchain, which build scripts, which C library) and you may be the only person with your particular setup.
• Little or no regression testing means bit-rot.
• Open source community only cares if you're using current version
  • Debugging old versions doesn't help the project
  • Fixes only go upstream if applied to current
• Upshot
  • Most embedded projects get stuck using old packages, even when you can get all the source.
• Hairball Builds
  • everything has to know about everything else
  • buildroot wasn't initially intended to be a distro (embedded variant of Zawinski's law)
  • non-orthogonal, hard to mix and match toolchain, build scripts, packaging (rpm, dpkg, ipkg/opkg, portage, tarballs), system imaging
    • Build system generally picks one of each and requires you to use its choices together
• Hidden dependencies
  • The host has to be set up specially
    • This is never properly documented, and the docs bit-rot if they exist
  • It builds for this engineer but not that one
  • Try to reproduce 6 months later and something's changed
• Previous mention of installing a target version of gzip into your toolchain, not just into your target system, so later packages can build against it
  • Version in toolchain and target must match
• Pain to set up
  • Knowing how to build a working cross compiler toolchain yourself is a black art
    • Even using an existing toolchain build system ala crosstool-ng requires extensive configuration
    • Lots of toolchain build systems aren't standalone, using their toolchain for other purposes is an afterthought
    • Slightest mistake results in leaky toolchain
  • Prebuilt binaries may or may not work for you unless you use the recommended version of the recommended distro
    • Linux has never been big into binary portability
  • Lots of absolute paths hardwired into gcc. Can't just extract into your home directory, need root access to install it as a package
  • Most developers get a prebuilt BSP (Board Support Package, I.E. build system) and use that as a black box. (See hairball, previously)
    • Can't report bugs upstream because package developers won't set up 15 different BSPs. Which leads into...
• Hard to get package bugs fixed
  • Can't report bugs upstream because package developers don't have your cross compiling environment set up and don't have the hardware to test the result anyway
• Restricted package selection
  • About 200-600 packages cross compile, depending on which platform you're targeting, how much effort you're willing to put in, and your definition of success
  • Debian has >30,000 packages. That's less than 2% of the total

Advantages of native compiling on real hardware

• Native compiling is simpler
  • No special support required in build
  • One context, no host/target split to leak information from
  • ./configure works as designed
  • Pushing bugfixes upstream takes less explanation
• For something like an iPhone, doing this is almost feasible.
  • There are also high-end PPC and ARM workstations.
    • This is how Fedora for ARM is built, using a build cluster of high-end ARM systems.

Disadvantages of native compiling on real hardware

• Need to obtain and install a bootable development environment
  • Chicken and egg problem
  • Either cross compile from host or use prebuilt binaries
• Is the device powerful enough?
  • Fast CPU, enough memory, enough disk, console access
    • 200 Mhz CPU with 32 Megs of RAM and a jffs2 flash root filesystem is not an ideal modern build environment
      • Flash has limited write cycles, building on it shortens device life.
    • Building on NFS is painful
      • Timestamp granularity and skew due to caching
• Did you bring enough for everybody?
  • These systems can be expensive and hard to share.
  • Can you afford to buy one for each engineer?
  • Lots of money into rapidly depreciating non-general-purpose asset.
  • Is it really cheaper to manage resource contention when engineers try to share 'em?
  • Even when you can ssh into it and it's something high-end like an iPhone or a game console, does it have enough resources to run two builds at once?
    • If not, sharing it between multiple developers is cumbersome
• Headless box UI issues
  • Need lots of infrastructure before you can run "hello world"
    • Appropriately configured/packaged/installed: toolchain, hardware installer (JTAG?), bootloader, kernel, console (serial port? Network?), root filesystem, C library, init program
    • If any of that fails, you may not get any output. Good luck.
• Easy to brick it
  • Need to reboot, re-flash, configure serial console. (See "sharing" above. If it's in the server room, you walk in there a lot or set up expensive infrastructure to automate power cycling, which has its own layer of control infrastructure.)
  • Can't necessarily just stick in a boot CD and start over
• Reduces flexibility
  • You have to make your hardware decisions early in your development cycle, you can't do much software development until you have the hardware
  • You can always get it running first on x86 and then port it, which has its own issues
    • Since x86 is the default platform everything gets developed on, "it works on x86" doesn't prove much
• Still tricky to send stuff upstream.
  • Package maintainers are less likely to have your hardware than they are to have/install your BSP. They can't build/test your issues.
• Less portable
  • A laptop can come with you on a plane, or work from home
  • Not all targets can be powered from USB or battery
• Not easily replaced
  • In case of a coffee spill you can get another PC and set it up same day

What do we want?

• It's possible to cope with the above, and a lot of people do. None of these problems are insurmountable, with enough engineering time and effort.
  • But it's not ideal
  • What have we identified so far that would bring us closer to an ideal build system?
    • Scalable
      • Scale to large development teams without significant resource contention
      • Run on cheap personal machines like hobbyists have, so open source hobbyists can use/develop with it. (Minimally usable on a $250 Thinkpad.)
      • Reasonably fast
        • Take advantage of SMP (possibly even clustering for build servers) so you have the option to throw money at it to make it faster
      • Capable of building complicated projects without collapsing into a tangle or requiring days for each build cycle
    • Reproducible
      • Easy to modify and debug
      • Breaking pieces you can't rebuild from source sucks
        • When something does break, you can make it happen again, take it apart and examine it, add logging, attach gdb to it, etc.
    • Maintainable
      • Something you can archive and retry years from now without having to dig up a copy of Red Hat 7.2 and try to get it to work on current hardware behind enough of a firewall it won't immediately get pwned by the ambient flora on the department's Exchange server.
    • Simple
      • Understandable, easy to learn, debug, extend.
      • If it doesn't exist, it can't break.
        • Well, almost. Piggy's zero byte file story
    • Orthogonal
      • Don't tie pieces together. Pick and choose.
        • Toolchain from here, build scripts from here, package management system from here...
        • Too many "accidental distros", like buildroot. Hairball is the natural state of cross compiling.
        • If you wind up writing your own in-house (or forking an upstream BSP), maintenance snowballs. You'll wind up rewriting it from scratch every 5 years or so, without improving much.
        • Something you can re-use for your next embedded project.
        • Don't let the tool take over your project
      • Not stuck to a single implementation.
      • Upgradeable
        • Drop in new upstream components as they release.
    • Portable
      • Move build to new machines
      • Take build home, to coffee shop, on a bus, on a plane
      • What if your developer's laptop is a Mac or Windows machine?
    • Self-contained
      • Something you can download/run without root access would be nice
        • Dear hobbyist developer, I found a bug in your code. To reproduce it, log into your laptop as root and go...
      • Easy enough to point upstream package managers at so they can reproduce your bug, diagnose the problem themselves, and merge a fix.
    • Scriptable
      • Automated regression testing is nice.
      • Cron job, nightly build and test.
        • Including upstream component projects.
    • Cheap
      • Free software running on cheap commodity x86-64 hardware.
    • Easy transition to your deployment environment (final hardware).

Compiling under emulation

QEMU Application Emulation

• QEMU has two modes:
  • Application and System Emulation
• This mode runs a userspace program, intercepting system calls and translating them to/from the host OS.
• Good for "smoke testing" your toolchain and C library: run a statically linked "hello world" program to make sure your toolchain (compiler, linker, C library, kernel headers) is set up correctly for your target.
• Some projects (such as some versions of scratchbox) set up the kernel's "misc binary support" to call QEMU application emulation to run target binaries.
  • This is to placate ./configure, and builds that aren't cross compile aware.
  • This is only a partial fix, leaving many problems unaddressed. The resulting build is still fairly brittle, very complicated, and requires root access on the host to set up, but it's a viable approach.
    • Also needs a version of uname that lies about the host, and some fiddling with shared library paths to run non-static target binaries, and remember to export HOSTTYPE and MACHTYPE environment variables...
• QEMU Application Emulation is nice but limited

QEMU System Emulation

• Standalone build system isolated from the host.
  • The rest of this talk describes this approach.
• Advantages
  • See "What We Want" previously. All that applies here
• Disadvantages
  • As with native hardware build, need to find/create a development system
    • Install's a lot easier though
  • Emulation overhead, slower than native compiling
    • Needs extra work to take advantage of SMP
  • Sometimes building on native hardware wins (faster, more convenient, more accurate, etc.)
    • There exists hardware with no good emulation
    • Running on the emulator doesn't always prove it'll work properly on the real hardware
    • You can always do both. (See "orthogonal" above, even QEMU can be swapped out.)

Obtaining a development environment for QEMU

• Root filesystem and native toolchain for target hardware, plus kernel configured specifically for one of QEMU's system emulations.
• First question: Use prebuilt binaries or build from source?

Using Prebuilt Binaries

• Prebuilt binaries are easier but less flexible.
  • They save time up-front at the risk of costing you a lot of time if anything goes wrong
  • If your intention is to put together a new embedded system, putting together your own development environment is good practice
  • Tailoring a system to save space/power/latency means eliminating unnecessary packages/libraries/ daemons
• There are lots of prebuilt distros to choose from.
  • Your hardware vendor probably has a BSP you can repackage for QEMU.
    • They may not offer a native toolchain, though.
  • Angstrom Linux, Gentoo Embedded, Emdebian, Openwrt, Openmoko, Fedora for ARM, Buildroot, and many many more.
• No guarantee any of them will meet your needs out of the box.
  • May need to repackage them for QEMU if they don't explicitly target it.
    • New kernel, drivers
    • ext3 filesystem image (with native toolchain and prereq packages)
    • QEMU command line.
• Even if you're not personally going to build a development environment from source, it's helpful to understand how one is put together.
  • Learning how it works can only help
    • Unless you're using Windows, which stops working when you collapse its quantum state.

Building a development environment from source

• Using a build system without understanding how it works isn't that much better than just grabbing prebuilt binaries.
  • But it does give you a fall back position if something goes wrong, you can then start learning about the build system when you need to.
  • Thinking ahead, you might want to pick a build system that isn't too hard to learn if you do need to take it apart
• The simpler your build system is, the less there is to go wrong
  • Less to keep up-to-date
  • Easier to learn
  • If cross compiling is inherently harder than native compiling, then the most maintainable system is probably one that does as little cross compiling as possible before switching over to a native build
• This is similar to the way tailoring an embedded system to save space/power/latency means eliminating unnecessary packages/libraries/daemons.
  • Less is more
• That said, the less the build system starts with, the more you'll have to add to it to meet your needs. dropbear, strace, gdb, gzip....

Understanding your build environment

Reinventing the wheel

• Many warnings (Danger Will Robinson)
  • On reinventing the wheel, and the hairball problem
    • Just about every new distro or build system in the past decade has started with somebody writing a shell script to automate the Linux From Scratch steps, and then tweaking it. You are not the first, and most of the others have a 10 year headstart on you.
• If you do want to create your own embedded build system anyway, expect it to take about a year to get something reasonably usable. (It's a great learning experience.)
• Maintaining your own build system is enormously time consuming.

Package Management and Accidental Distros

• Linux From Scratch intentionally does not cover package management.
• Package management two parts: Using a package manager (rpm, dpkg, ipkg/opkg, portage) to install/uninstall/track packages, and creating a repository of build scripts and package dependencies.
• Creating a repository of package dependencies and build options is a huge undertaking and requires constant maintenance as new packages are released upstream.
• Fedora, Debian, Gentoo, and others already have perfectly acceptable repositories you can leverage. Creating your own is a huge time sink.
• Despite this, it's very easy to fall into the trap of creating your own Linux distro by mistake, at which point complexity explodes out of control.
  • Maintaining your own distro is enormously time consuming.
  • The more packages your build system builds, the more likely this is to happen.
• Defending yourself from accidental distros.
  • Divide the hairball into orthogonal layers. Toolchain selection has nothing to do with package management
  • Needing to change one is no excuse for accepting responsibility to maintain the other.
  • Delegate everything you can to existing projects.
    • Push your changes upstream, which makes them somebody else's problem.
• Figure out what your goals are and what you're not going to do
  • You can't stay focused if you can't say no

Buildroot Example

• The buildroot project was originally just a uClibc toolchain creator and test harness. The easy way to test that a package built against uClibc was to add it to buildroot, and since it never had a clear design boundary allowing it to say "no" to new features, this quickly grew out of hand.
• The project maintainer (Erik Andersen) and several of his senior developers had so much of their time taken up by buildroot they stopped contributing to the original project, uClibc.
• The uClibc mailing list was swamped by buildroot traffic for years until Rob created a new buildroot list and kicked the traffic over there, but uClibc development still hasn't fully recovered.
• This is a significant contributing factor to uClibc's TLS/ NPTL support being feature complete for its first architecture in 2006 (there was an OLS paper on it) but still not having shown up in an actual uClibc release 3 years later.
• They do the distro thing badly: no package manager, no repository.
  • Buildroot didn't use a package manager (like rpm, dpkg, ipkg/opkg, or portage), instead it encoded its dependency resolution in menuconfig's kconfig files, and the package build logic in a series of nested makefiles, then stored the whole thing in source control. Thus the builtroot develpers took a while to notice it was becoming a distro.
• Lots of open source build systems forked off of buildroot, just as lots of in-house build systems forked off of board support packages. They quickly diverge enough to become independent projects capable of absorbing enormous amounts of time.
• Moral: Original project stalled because its developers sucked away, wound up doing the distro thing badly with no package manager, no repository, no stable releases.
  • Don't let this happen to you.

Orthogonal layers

• The following is based on what our Aboriginal Linux project does.
  • Other build systems need to do the same things, most of them just aren't as careful to separate the layers.
• Each of these layers can be swapped out
  • Use somebody else's cross compiler, use our cross compiler to build somebody else's root filesystem, package an arbitrary directory with our packaging script, etc.
  • We try to make it as easy as possible to NOT use our stuff, and show it here only as an example.
• This build is intentionally implemented as a simple series of bash scripts, so you can read the scripts to see what it's doing.

Downloading Source

• Download source code (download.sh)
  • Calls wget on a bunch of URLs to copy files into "packages" directory.
  • You can just cp 'em there yourself instead if you like
• Does security/integrity sha1sum checks on existing tarballs (if any)
  • Some build systems (such as crosstool-ng) give the weirdest errors if you re-run the build after an interrupted download, and you have to figure out the problem and fix it by hand
  • The GNU ftp server has been cracked before
• Cache tarballs so once you've run this step you no longer need net access
  • Falls back to a couple mirrors if the source site is down
    • PREFERRED_MIRROR environment variable points to a mirror to try first if you have a local copy. Note that QEMU's 10.0.2.2 tunnels through to the host's 127.0.0.1, so if your build host runs a web server on loopback it can easily provide source tarballs to a build inside QEMU without bogging the public servers.
• Track all the source URLs and versions in one place
• Only this file has version info in it, making drop-in upgrades a two-line change.
  • Non-drop-in upgrades require debugging, of course

Host Tools

• Entirely optional step, can be skipped if your host is set up right, but that's seldom a good idea.
• Serves same purpose as LFS chapter 5: an airlock preventing leakage.
  • LFS chroots, Aboriginal adjusts the $PATH to remove stuff.
  • Remember, adding stuff is easy. Removing stuff the host already has so cross compiling doesn't accidentally find and leak it into your build is one of the hard parts.
• Provides prerequisites automatically, no need to install packages by hand as root
  • Nothing the Aboriginal build system does requires root access on the host. That's an explicit design goal.
• Isolate your build from variations in the host distro
  • Provide known versions of each tool
• Smoke test and regression test
  • Building everything from here on with the same tools we're going to in the final system (busybox, etc) demonstrates that the build system we're putting together can reproduce itself.
  • And if it can't (regression) we find out early

Create a Cross-Compiler

• Four packages: binutils, gcc, linux kernel headers, uclibc
  • In that order
    • gcc depends on binutils, building uClibc depends on all three previous packages.
  • Later, we add a fifth package: uClibc++, for c++ support.
• Use a compiler wrapper (ccrap.c) based on the old uClibc wrapper to make it all work together
  • The wrapper parses the gcc command line and then rewrites it to start with --nostdinc and --nostdlib, then explicitly adds in every library and header path gcc needs, pointing to a known location determined relative to where the wrapper binary currently is.
  • This prevents the host toolchain's files (in /usr/include and /lib and such) from leaking into the target toolchain's builds.
  • It also means the toolchain is "relocatable", I.E. it can be extracted into any directory and run from there, so it can live in a user's home directory and does not require root access to install.
  • Just add the toolchain's "bin" subdirectory to your $PATH and use the appropriately prefixed $ARCH-gcc name to build.
• The simple compiler above can be created without prerequisites
• Thus creating a simple cross compiler is always the first step
• For Aboriginal releases we build a second cross compiler that's statically linked against uClibc on the host, which is a two-stage process
  • First build an i686-gcc compiler, which outputs 32-bit x86 binaries
  • Then rebuild the cross compilers as --static binaries using that as the host compiler
  • This increases portability and decreases size. You can run the resulting binaries on 32 bit Red Hat 9 and on 64-bit Ubuntu 9.04.
  • This one also builds uClibc++, an embedded replacement for libstdc++
    • The first toolchain can build C++, but has no standard C++ library to link them against
• --disable-shared vs --enable-shared
  • History of ccwrap
    • uClibc compiler wrapper
  • evolution of buildroot
  • Why building libstdc++ is painful

Cross-compile a root filesystem

• Long story short: seven packages (busybox, uClibc, linux, binutils, gcc, make, bash), plus some empty directories and misc files like an init script.
  • This creates a directory full of files, suitable to chroot into.
• Create directory layout
  • tmp proc sys dev home usr
  • Symlinks into usr/{bin,sbin,lib,etc} from top level.
    • Historical note where "usr" came from in 1971.
• Install uClibc
  • Build this first so everything else can build against it.
  • Cross compiler may not come with libc we're installing, so use ccwrap on existing cross compiler here.
    • This is redundant when using our compiler, but necessary for orthogonality.
      • Allows you to use arbitrary existing cross compiler
• Build busybox
  • Provides almost all POSIX/SUSv4 command line utilities
  • Switch off a few commands that don't build on non-x86
  • Add a few supplemental commands (getent, patch, netcat, oneit) from "toybox" or as shell scripts. Mostly optional.
    • Biggest issue is that busybox patch can't apply hunks at an offset, which makes it too brittle to use in a development environment
• Install native toolchain
  • Same packages as cross compiler, plus uClibc++
  • Note, Aboriginal Linux builds a statically linked native compiler, which you can extract and run on an arbitrary target system.
    • Same theory as the statically linked cross compiler, only for native builds on systems that don't come with a uClibc toolchain.
• Install make, bash, and (optionally) distcc.
  • You can't do ./configure; make; make install without "make". (While cross compiling, used the host's version of make.)
  • Build bash because busybox ash isn't quite good enough yet
    • Getting pretty close, need to retest and push fixes upstream
    • Building bash 2 not bash 3, because it's 1/3 the size, and doesn't require ncurses
• Install init script and a few misc files
  • Copy sources/native/* to destination
  • Includes an init script
  • An etc/resolf.conf for QEMU
  • Some "hello world" C source programs in usr/src...
• The init script does the following
  • Mount sysfs on /sys, and tmpfs on /dev and /tmp.
    • tmpfs is a swap-backed ramfs, writeable even if root filesystem is read-only.
  • populate /dev from /sys (using busybox mdev instead of udev)
  • Configure eth0 with default values for QEMU's built-in vpn.
  • Set clock with rdate if unix time < 1000 seconds.
    • This means this emulation has no battery backed up clock.
  • Uses host's inetd on 127.0.0.1 to get current time.
  • If we have /dev/hdb or /dev/sdb, mount it on /home.
  • It echos "Type exit when done.", which means the boot was successful and gives "expect" something to respond to.
  • Use "oneit" program to run a command shell.
    • This gives us a shell with a controlling tty (ctrl-c works), and the system shuts down cleanly when that process exits.
• Optimizations
  • Strip binaries, delete info and man pages, and gcc's "install-tools".
  • Build without a native toolchain
    • Means you can also delete /usr/include and /usr/lib/*.a from uClibc.
    • Don't need make or bash
  • Build static or dynamic
    • When building static and not installing a native toolchain, we can skip this step entirely. Just use whichever libc the cross compiler has, and don't bother to install it on the target.

Package a system image

• qemu-system-* needs a bootable kernel and a filesystem image.
  • So does real hardware.
• Build a bootable Linux kernel
  • Needs correct CPU and board layout (device tree or equivalent), drivers for serial console, hard drive, network card, clock, ext2/ext3/squashfs/tmpfs
• Which kernel .config do you build?
  • The kernel has default configuration files for a lot of boards.
  • Look in the Linux kernel source for an arch/$ARCH/configs directory
    • Or a single arch/$ARCH/defconfig file for the less-diverse architectures
• We use the "miniconfig" technique
  • Uses miniconfig.sh to create one
  • make allnoconfig KCONFIG_ALLCONFIG=filename
  • Portability, Readability
  • Also works with uClibc and busybox .configs.
• Use mksquashfs, gene2fs, or cpio/gzip to create standalone filesystem image.
  • Possibly rebuild kernel to include initramfs (gzipped cpio image).
    • Note initramfs is size-limited by most kernels' memory mappings, and seldom has enough space for a dozen megabytes of toolchain.

Alternatives to packaging

• All sadly inferior, but know your options
• QEMU Application Emulation
  • It can run programs out of root-filesystem.
    • Requires root access to use the -chroot option (and to populate /dev).
    • Not much faster than full system emulation, but significantly more brittle.
  • Strangely, application emulation is a harder problem to solve than system emulation.
    • System emulation starts from scratch and fakes something simple, a CPU, some attached memory (with MMU), and a half-dozen well-documented device chips (clock, serial, hard drive, etc).
  • QEMU application emulation must intercept literally thousands of syscalls and ioctl structures (each of which may have a dozen members) and translate every one its own special case, dealing with endianness and alignment and sometimes different meanings of the fields on different architectures.
  • Signal numbers differ.
  • Page sizes differ, which affects mmap behavior in several ways (padding the end of the file with zeroes for how long?)
  • It's fiddly.
  • QEMU application emulation is a nice smoke test, but you wouldn't want to develop there.
    • Run a static "hello world" to see if your compiler's working.
    • Fewer prerequisites between you and a simple shell prompt.
      • No need to build a kernel, let alone get serial console working.
• QEMU can create virtual FAT disk images
  • Option "-hda fat:/path"
  • This exports a directory as a virtual block device containing a read-only vfat filesystem constructed from the files at path.
    • Remember to mount it read-only: if Linux tries to write to it lost interrupts and driver unhappiness ensue.
  • Standard "booting from vfat" issues
    • Case insensitivity, no ownership, incomplete permissions...
    • Doable, but fiddly.
  • Do not change contents of host directory while QEMU is running, it'll get very confused.
  • Assembling a virtual FAT with lots of little files can take lots of time and memory
    • Doesn't scale. QEMU may grind for quite a while before launch with even a relatively small filesystem.
• Exporting a directory via a network filesystem is another way to provide a root filesystem to QEMU or to real hardware.
  • May be OK for read-only root filesystem, not so good for writeable development space you compile stuff in.
  • Login management unpleasant, make sure you do this behind a firewall or via loopback (10.0.2.2 in qemu is 127.0.0.1 on host).
• Linux's most commonly used network filesystem (NFS, Samba, TFTP) all suck in various ways.
  • NFS sucks rocks
    • The words "stateless" and "filesystem" do not belong in the same sentence. (Thanks Sun!)
    • Requires root access on the host to export anything
    • Fiddly and brittle to automate, so usually done by hand
    • Building on non-readonly NFS is unreliable due to dentry cacheing issues
      • Make gets very confused when timestamps go backwards.
      • General problem: either you have network round trip latency for each directory lookup or you have local dentry caches that can get out of sync with the server's timestamp info.
      • At least most things don't care about atime.
  • Samba sucks almost as many rocks
    • It's a Windows protocol, constructed entirely out of weird non-POSIX behavior and corner cases.
      • Case insensitive, many builds don't like this.
      • Dentry info translated (can confuse make), doing things like mmap (and even symlinks) hacked on.
      • Share naming and account management are their own little world.
    • Finding the server can be a pain
      • Domain server and Active Directory and such: there be dragons
    • But at least it's got a userspace server that can export a filesystem running as a normal user, and a lot of people already understand it quite well.
    • QEMU has a built in -smb option, which launches samba on the host for you to export the appropriate directory, visible inside the emulator on 10.0.2.4 as "\\smbserver \qemu".
• TFTP
  • Only mentioned because QEMU has a built in -tftp option, which exports a directory of files to the emulator via a built-in (emulated) tftp server.
• Filesystem in Userspace (sshfs and friends)
  • Can't boot from FUSE, need another root filesystem (generally initramfs) for setup, so doesn't solve this problem.

Booting a system under QEMU

QEMU Command Line Options

• Command line options to qemu-system-$ARCH:
  • Use "-nographic" to make qemu scriptable from the host.
  • Disables video card emulation, and instead connects the first emulated serial port to the QEMU process's stdin/stdout.
  • This gives you an emulator shell prompt about like ssh would, in a normal terminal window where cut and paste work properly.
• You might want to build and use "screen" inside the emulator, or set $TERM, $ROWS and $COLUMNS for things like busybox "vi".
• This means you can run QEMU as a child process and script it with "expect" (or similar).
  • This is marvelous for automating builds, regression testing, etc.
  • Wrap a GUI (or emacs) around the QEMU invocation.
    • QEMU runs on non-Linux hosts, no reason your GUI couldn't run on a Mac
• Note, some versions of QEMU are built with a broken curses support that craps ANSI escape sequences randomly to stdout, which will confuse expect. If you encounter one of these, rebuild it from source with --disable-curses.
  • In theory this shouldn't happen if stdin is not a tty, but there are buggy versions out there that don't check.
• You can also cat a quick and dirty shell script into QEMU's stdin for the emulated Linux's shell prompt to execute.
  • The reason this is "quick and dirty" is it isn't 100% reliable, because the input is presented to stdin before the kernel has a chance to boot.
  • Serial port initialization eats a variable amount of data, generally around the UART buffer size (16 bytes-ish for a 16550a, but there's some race conditions in there so it's not quite contstant).
  • This is a Linux kernel serial/tty issue, not a QEMU issue.
  • You can work around this by starting your script with a line of space characters and a newline, and then it'll work ~99% of the time. But if you use it a lot (automated regression, nightly builds, etc) the lack of flow control can inject the occasional spurious failure because the script didn't read right.
• Using "expect" (or similar) is more reliable. The kernel doesn't discard data when it's listening for data. The tty driver and various programs just get a bit careless with the buffer if you get too far ahead when they're not listening for data.
• Use "-no-reboot" so QEMU exits when told to shut down by the kernel.
  • Helps with scriptability.
• Use "-kernel FILENAME" to specify which kernel image to boot.
  • Avoids the need for grub or u-boot to launch emulated system.
  • Uses a very simple built-in bootloader to load kernel image into QEMU's emulated physical memory and start it.
• QEMU has code to boot an ELF format linux kernel
  • vmlinux in top level kernel build
    • Alas, not enabled for all architectures
  • Can also handle bzImage and various target-specific variants
• "-hda FILENAME"
  • Provides virtual ATA/SCSI block devices for board emulation
    • Specify more with -hda, -hdb, -hdc...
    • First one can be a read-only root filesystem, second writeable scratch space to compile stuff in, third extra data such as source tarballs...
    • Yes the option is "-hda" even when the board is emulating SCSI and thus the device it emulates becomes "/dev/sda".
    • No need to partition these, Linux can mount /dev/hda just fine
• "-append 'OPTIONS'"
  • The Linux kernel command line
    • In theory this is extra stuff added to the default kernel command line, but in practice it's generally the whole kernel command line.
  • Relevant options you probably want to specify:
    • "console=ttyS0" (or platform equivalent device)
      • Tell the kernel to send boot messages (and /dev/console, thus the stdin/stdout of PID 1) to a serial device
      • I.E. "serial console"
    • "root=/dev/hda"
      • Specify where the root filesystem lives
      • Also specify "rw" so root filesystem is read/write
      • Don't need either if booting from initramfs
    • "init=/usr/sbin/init.sh"
      • Specify what executable file to launch as the initial process. The "init task" (PID 1) is special. It has signals blocked, the kernel panics if it ever exits, it gets sigexit for orphanedchild processes and has to reap their zombies, etc
      • Note that just using init=/bin/sh will give you a quick and dirty shell prompt, but you won't have a controlling tty, thus ctrl-c won't work.
      • Meaning you won't be able to break out of simple things like "ping". Very easy to lose control, gets old fast.
      • Also need to set up /proc yourself, populate /dev, ifconfig...
    • "panic=1"
      • This tells the kernel to reboot one second after panicking
        • Which -no-reboot above turns into QEMU exiting
• Put it all together and you have something like:

qemu -nographic -no-reboot -kernel zImage-i686 -hda image-i686.sqf -append "root=/dev/hda rw init=/usr/sbin/init.sh panic=1 PATH=/usr/bin console=ttyS0"

Troubleshooting

• Do you see kernel boot messages on QEMU's stdout? If not:
  • Your serial console might not work
    • Linux serial driver
      • static driver, not a module. * instead of m.
      • console= line on kernel command line
      • Is QEMU's board emulation giving you the serial device you expect?
• Is my toolchain even producing the right output?
  • Build a statically linked "hello world" and run that on the host using QEMU's application emulation.
    • If it won't run, use the "file" command on the binary (and the .o and .a files) and check that it the kind of binary you think it is.
    • If the toolchain you're using uses ccwrap, try setting the environment variable WRAPPER_DEBUG=1 to see all the component files linked into the binary, and check them with "file".
• Kernel may not be booting up far enough to spit anything out
  • Is it the right configuration?
    • Enable EARLY_PRINTK in the kernel
  • Is it packaged the right way?
    • vmlinux, bzImage, chrp...
• Do you see boot messages but no shell prompt?
  • Ok, where does it stop?
  • Seeing some output is always better than seeing no output
  • You can always stick more printk() calls into the kernel source code
  • Is it hanging on a driver before it tries to launch init?
    • Make sure the board emulation and kernel .config agree.
    • If it's a target that needs a "device tree", what does that say?
  • Not finding the root filesystem?
    • Is the root= line correct?
    • Correct block driver?
    • Correct filesystem format driver?
    • Make sure those two drivers are static, not modules.
    • Try an initramfs?
  • Mounting but unable to run init?
    • Check your filesystem image (via loopback mount) to make sure the binary you expect really is there.
    • Note that an initramfs uses "rdinit=" instead of "init=".
  • Try pointing init= at a statically linked "hello world" program.
    • Aboriginal Linux contains one at /bin/hello-static
    • If this works, your dynamic linker is probably at fault.
  • Then try a dynamically linked hello world. If that fails:
    • Is your library loader (/lib/ld-uClibc.so.0 or similar) where you think it is?
      • ldd is target specific but "readelf" is fairly generic.
    • Make sure all the libraries it tries to link to are there.
      • Remember: Shared libraries link to other shared libraries
      • run ldd/readelf on all
  • Run "file" on everything to make sure no host binaries have leaked into your root filesystem.
  • Maybe dynamic loading isn't supported with that libc on that target? (*cough* uClibc on Sparc *cough*)
• If it launches init but you get no further output.
  • Is your init trying to redirect /dev/console and missing the serial console?
    • Work your way up from init=/helloworld through a shell prompt to whatever your init program is doing.
      • Add our prebuilt static strace binaries to your root filesystem and run your init under that.
• strace turns random hangs into "it made it this far, and was trying to do that"
  • Only useful if you're getting output from "hello world".
• Target-specific issues
  • ARM, MIPS, PowerPC, SPARC, sh4, x86, x86_64, M68k...

Using your emulated development environment

• The emulated development environment gives you two things
  • A scriptable build environment, so you can run nightly automated regression testing cron jobs to build and test your software
    • Not a substitute for testing on real hardware, but a good automated smoke test
  • A shell prompt at which you can do fresh development
• As with all new development environments, you may need to build/install lots of prerequisite packages to get the development environment you really want
  • gzip, ncurses, openssl, strace, dropbear, screen...
• Remember: what your development machine needs and what your deployment environment needs aren't necessarily the same thing.
  • Your development image has a toolchain. Probably won't ship that.
  • Quite possibly other stuff you'll develop with but not ship.
    • Needing to build it != needing to ship it.
• Editing source on your host system is a lot more comfortable.
  • Trying to add too much UI to the emulators is probably a waste of time. Your host is pretty and user friendly already
  • busybox has a "vi" implementation, so if you really need to you can edit source inside the emulator.
  • Using "screen" (which requires ncurses) makes things much less painful.
    • Probably need to export $TERM, $LINES, and $COLUMNS by hand to get it to work, ncurses can't query a serial console for tty info
• You probably still want to do most of your editing and source control and such on your host, in an IDE, with multiple xterms and source control and so on.
  • So the problem becomes getting source from your host into your build system (and updating it), and getting results back out.

Getting data in/out of the emulator

Debugging

• gdbserver (part of the gdb package) allows you to debug things remotely.
  • Runs on the target to trace an executable
  • Lets you use gdb without the overhead of running gdb itself on the target, which isn't always even possible (it's a big piece of GNU bloatware).
• gdbserver runs on the target to trace an executable, talks to gdb via a serial protocol.
  • Serial protocols tunnel well over ssh or netcat
    • Run QEMU to -redirect a port (such as 9876) to the host loopback
    • Inside QEMU: gdbserver 10.0.2.15:9876 FILENAME
    • Run gdb on host using the "pipe to process" option:
      • file FILENAME
      • target remote | netcat 127.0.0.1 9876
      • set height 0
• Note that QEMU has a "-s" option that lets you attach gdb to the emulated hardware, the way a JTAG would
  • Uses the same gdbserver protocol.
  • This debugs the OS, not the applications.
• Either way requires a host version of gdb that understands your target's instruction set.
  • Most distro versions configured host-only.

Interacting with upstream package maintainers

• You have a bug you would like fixed
  • Offering them a test environment in which they can reproduce the problem.
  • Offering a development environment in which they can test their fixes.
  • Self-contained QEMU development environment images small enough to download, running locally without root access are good for both.

Performance considerations