Documentation for Firmware Linux


What is Firmware Linux?

Firmware Linux is a toolkit for building custom virtual machines. It lets you boot virtual PowerPC, ARM, MIPS and other exotic systems on your x86 laptop, and do development in them.

The name "Firmware Linux" reflects FWL's origins as a tool for embedded development. It provides an easy way to get started with that, building your own code against uClibc and testing it on various hardware platforms. But it has other uses as well, including cross-platform regression testing, portability auditing, and toolchain debugging.

This documentation uses the name "Firmware Linux" (or abbreviation "FWL") to refer to the build system, and calls the output of the build a "system image". The build system is implemented as a series of bash scripts and configuration files which compile a Linux development environment for the specified target system and package it into a bootable binary image.

The base development environment is built from seven source packages: busybox, uClibc, gcc, binutils, make, bash, and the Linux kernel. This is the smallest environment that can rebuild itself entirely from source code, and thus the minimum a host system must cross compile in order to create a fully independent native development environment for a target.

Booting a development system image under an emulator such as QEMU allows fully native builds for supported target platforms to be performed on cheap and powerful commodity PC hardware. You can then build and install additional packages (zlib, bison, openssl...) within the virtual machine's native development environment, without having to do any additional cross compiling.

FWL currently includes full support for arm, mips, powerpc, x86, x86-64 targets, and several other more exotic platforms; see the screenshots page for a complete list. The goal for the FWL 1.0 release is to support every target QEMU can emulate in "system" mode.

Firmware Linux is licensed under GPL version 2. Its component packages are redistributed under their respective licenses (mostly GPL and LGPL).

Optional extras

Intermediate stages of the build (such as the cross compiler and the unpackaged root filesystem directory) may also be useful to Linux developers, so tarballs of them are saved during the build.

By default the build cross-compiles some optional extra packages (toybox, distcc, uClibc++) and preinstalls them into the target filesystem. This is just a convenience; these packages build and install natively within the minimal development system image just fine.)


Using system images

If you want to jump straight to building your own software natively for embedded targets, you can download a prebuilt binary image instead of running the build scripts to produce your own.

Here are the different types of output produced by the build:

system-image-*.tar.bz2

System images boot a complete linux system under an emulator. Each system-image tarball contains an ext2 root filesystem image, a Linux kernel configured to run under the emulator QEMU, and scripts launch the virtual system under the emulator in various configurations.

The steps to test boot a system image under QEMU are:

This boots the system image under the appropriate emulator, with the emulated Linux's /dev/console hooked to stdin and stdout of the emulator process. (I.E. the shell prompt the script gives you after the boot messages scroll past is for a shell running inside the emulator. This lets you pipe the output of other programs into the emulator, and capture the emulator's output.)

Type "cat /proc/cpuinfo" to confirm you're running in the emulator, then play around and have fun. Type "exit" when done.

Inside a system image, you generally wget source code from some URL and compile it. (For example, you can wget the FWL build, extract it, and run it inside one of its own system images to trivially prove it can rebuild itself.) If you run a web server on your host's loopback interface, you an access it inside QEMU using the special address "10.0.2.2". Example build scripts are available in the /usr/src directory.

Extra space and speed

The system images by themselves are fairly small (64 megabytes), and don't have a lot of scratch space for building or installing other packages. If a file named "hdb.img" exists in the current directory, run-emulator.sh will automatically designate it as a second virtual hard drive and attempt to mount the whole unpartitioned device on /home inside the emulator.

Some optional command line arguments to run-emulator.sh provide extra space and extra speed for compiling more software:

Running an armv4l system image with the cross compiler installed in the user's home directory, using a hard drive image in the user's home directory (to be created with a size of 2 gigabytes if it doesn't already exist) might look like:

./run-emulator.sh --make-hdb 2048 --with-hdb ~/blah.img --with-distcc ~/cross-compiler-armv4l

mini-native-*.tar.bz2

These tarballs contain the same root filesystem as the corresponding system images, just in an archive instead of packaged into a filesystem image.

If you want to boot your own system image on real hardware instead of an emulator, the appropriate mini-native tarball is a good starting point. If all you want is a native uClibc development environment for your host, try:

chroot mini-native-x86_64 /usr/chroot-setup.sh

The boot script /usr/qemu-setup.sh or /usr/chroot-setup.sh performs minimal setup for the appropriate environment, mounting /proc and /sys and such. It starts a single shell prompt, and automatically cleans up when that process exits.

If you're interested in building a more complex development environment within this one (adding zlib and perl and such before building more complicated packages), the best way to learn how is to read Linux From Scratch.

Note that mini-native is just one potential filesystem layout; the FWL build scripts have several other configurations available when you build from source.

cross-compiler-*.tar.bz2

The cross compilers created during the FWL build are relocatable C compilers for each target platform. The primary reason for offering each cross compiler as a downloadable binary is to implement the distcc accelerator trick. Using them to cross compile additional software is supported, but not recommended.

If you'd like to use one for something other than distcc, this documentation mostly assumes you already know how. Briefly:

Also, stock up on asprin and clear a space to beat your head against; you'll need both. See why cross compiling sucks for more details.

Note that although this cross compiler has g++, it doesn't have uClibc++ in its lib or include subdirectories, which is required to build most c++ programs. If you need extra libraries, it's up to you to cross-compile and install them into those directories.

How do I build my own customized system images from source code?

To build your own root filesystem and system images from source code, download and run the FWL build scripts. You'll probably want to start with the most recent release version, although once you've got the hang of it you might want to follow the development version.

For a quick start, download the tarball, extract it, cd into it, and run "./build.sh". This script takes one argument, which is the target to build for. Run it with no arguments to list available targets.

This should produce all the tarballs listed in the previous section in the the "build" directory. To perform a clean build, "rm -rf build" and re-run build.sh.

How building from source works

The build system is a series of shell scripts which download, compile, install, and use the appropriate source packages to generate a system image. These shell scripts are designed to be easily read and modified, acting both as tools to perform a build and as documentation on how to build these packages.

The build.sh script is a simple wrapper which calls the following other scripts in sequence:

  1. download.sh
  2. host-tools.sh
  3. cross-compiler.sh $TARGET
  4. mini-native.sh $TARGET
  5. package-mini-native.sh $TARGET

In theory, the stages are othogonal. If you have an existing cross compiler, you can add it to the $PATH and skip cross-compiler.sh. Or you can use _just_ cross-compiler.sh to create a cross compiler, and then go build something else with it. The host-tools.sh stage can often be skipped entirely.

Build stages

The following files control the individual stages of the build. Each may be called individually, from in the top level directory of FWL:

  • cross-compiler.sh - Build a cross compiler for the target, for use by mini-native.sh and the distcc accelerator.

    In order to build binaries for the target, the build must first create a cross compiler to build those target binaries with. This script creates that cross compiler. If you already have a cross compiler, you can supply it here (the easy way is to create a build/cross-compiler-$TARGET/bin directory and put "$TARGET-gcc" style symlinks in it) and skip this step.

    This script takes one argument: the architecture to build for. It produces a cross compiler that runs on the host system and produces binaries that run on the target system. This cross compiler is created using the source packages binutils, gcc, uClibc, the Linux kernel headers, and a compiler wrapper to make the compiler relocatable.<

    The reason for the compiler wrapper is that by default, gcc hardwires lots of absolute paths into itself, and thus only runs properly in the directory it was built in. The compiler wrapper rewrites its command line to prevent gcc from using its built-in (broken) path logic.

    The build requires a cross-compiler even if the host and target system use the same processor because the host and target may use different C libraries. If the host has glibc and the target uses uClibc, then the (dynamically linked) target binaries the compiler produces won't run on the host. (Target binaries that won't run on the host are what distinguishes cross-compiling from native compiling. Different processors are just one reason for it: glibc vs uClibc is another, ELF vs binflat or a.out executable format is a third...)

    This script produces produces a working cross compiler in the build directory, and saves a tarball of it as "cross-compiler-$TARGET.tar.bz2" for use outside the build system. This cross compiler is fully relocatable (because of the compiler wrapper), so any normal user can extract it into their home directory, add cross-compiler-$TARGET/bin to their $PATH, and run $TARGET-gcc to create target binaries.

  • mini-native.sh - Use the cross compiler to create a minimal native build environment for the target platfrom.

    This script takes one argument: the architecture to build for.

    This script uses the cross compiler found at build/cross-compiler-$ARCH/bin (with $ARCH- prefixes) to build a root filesystem for the target, as well as a target Linux kernel configured for use with qemu. A usable cross compiler is left in the build directory by the cross-compiler.sh script, or you can install your own.

    The basic root filesystem consists of busybox and uClibc. If the configuration variable NATIVE_TOOLCHAIN is set (this is enabled by default), this script adds a native compiler to the target, consisting of linux kernel headers, gcc, binutils, make, and bash. It also adds distcc to potentially distribute work to cross compilers living outside the emulator. This provides a minimal native development environment, which may be expanded by building and installing more packages under the existing root filesystem.

  • package-mini-native.sh - Create an ext2 filesystem image of the native root filesystem.

  • This script takes one argument: the architecture to package.

    This uses genext2fs to create an ext2 filesystem image from the build/mini-native-$ARCH directory left by running mini-native.sh, and creates a system-image-tarball containing the result. It first compiles genext2fs and adds it to build/host if the host system hasn't already got a copy.

    This script also generates a run-emulator.sh script to call the appropriate emulator, using the architecture's configuration information.

  • run-from-build.sh - Runs a system image you compiled from source.

    Calls run-emulator.sh in the appropriate build/system-image-$TARGET directory, with a 2 gigabyte hdb.img for /home and distcc connected to build/cross-compiler/$TARGET. Between runs it calls e2fsck on the system image's root filesystem.

    This is not technically a build stage, as it isn't called from build.sh, but it's offered as a convenience for users. It uses the existing cross-compiler and system-image directories in build/ and doesn't mess with the tarballs that were created from them.

  • The following generally aren't called directly, but are used by the rest of the build.


    How is Firmware Linux implemented?

    Directory layout

    The top level directory of FWL contains the user interface of FWL (scripts the user is expected to call or edit), and the "sources" directory containing code that isn't expected to be directly called by end users.

    Important directories under sources include:

    Output files from running the build scripts, and all temporary files, go in the "build" subdirectory. This entire directory can be deleted between builds.

    Shared infrastructure

    The top level file for the behind-the-scenes plumbing is sources/include.sh. This script is not run directly, but is instead included from the other scripts. It does a bunch of things:

  • It parses the "config" file at the top directory, reading in the user defined configuration variables. (You can also supply these as environment variables, if you want to specify them for just one run.)

  • It sets several other environment variables, specifying things like the $SOURCE and $BUILD directories, and detecting the number of $CPUS. Many of these are set using the export_if_blank function, which keeps any existing value of the variable, allowing them to be externally overridden.

  • It adjusts the $PATH. If build/host exists and contains a busybox executable (meaning host-tools.sh did its thing already), $PATH is set to just that directory. If build/wrapdir exists, that's used instead for command line logging via sources/more/record-commands.sh.

  • If host-tools.sh ran after record-commands.sh, it sets the $PATH to point to the logging wrapper directory. ($WRAPPY_LOGPATH specifies where the logging wrapper should write its log file, and $WRAPPY_REALPATH says where to find the actual commands the logging wrapper hands off to.)

  • It also reads sources/functions.sh, which provides shell functions used by the rest of the build, including: