Mercurial > hg > aboriginal
changeset 5:2907d5193cf6
A real web page, and description of the build process.
author | Rob Landley <rob@landley.net> |
---|---|
date | Mon, 27 Nov 2006 19:36:31 -0500 |
parents | 9f52b6d1c4ab |
children | e039588b3189 |
files | www/build-process.html www/index.html |
diffstat | 2 files changed, 287 insertions(+), 2 deletions(-) [+] |
line wrap: on
line diff
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/www/build-process.html Mon Nov 27 19:36:31 2006 -0500 @@ -0,0 +1,68 @@ +<html> +<title>The Firmware Linux build process</title> + +<p>FWL builds a cross-compiler and then uses it to build a minimal system with +a native compiler, BusyBox and uClibc. Then it runs this minimal system +under an emulator (QEMU) and natively builds the final system. It then +packages the resulting system (kernel, initramfs, and root filesystem) into +a single file that can boot and run (using a modified version of LILO on +x86).</p> + +<p>Firmware Linux builds in stages:</p> + +<h2>Stage 1: Build a cross-compiler.</h2> + +<p>The first stage builds a cross-compiler, +which runs on the host system and produces binaries that run on the target +system. (See my <a href=/writing/docs/cross-compiling.html>Introduction to +cross compiling</a> if you're unfamiliar with this.)</p> + +<p>We have to cross-compile even if the host and target system are both +x86, because the host probably use different C libraries. If the host has +glibc and the target uses uClibc, then the (dynamically linked) target binaries +we produce won't run on the host. This is what distinguishes cross-compiling +from native compiling: different processors are just one reason the binaries +might not run. Of course, as long as we've got the cross-compiling support +anyway, we might as well support building for x86_64, arm, mips, or ppc +targets...</p> + +<p>Building a cross-compiler toolchain requires four packages. The bulk of +it is binutils, gcc, and uClibc, but building those requires header files from +the Linux kernel which describe the target system.</p> + +<h2>Stage 2: Use the cross-compiler to build a native build environment +for the target.</h2> + +<p>Because cross-compiling is persnickety and difficult, we do as little of +it as possible. We use the cross-compiler to generate a native build +environment for the target, and then run the rest of the build under an +emulator.</p> + +<p>The minimal build environment you can boot into and build a complete Linux +system under is the Linux kernel, binutils, gcc, uClibc, BusyBox, make, and +bash. The emulator we use to run this is QEMU, so we build that too.</p> + +<h2>Stage 3: Run the target's native build environment under an emulator to +build the final system.</h2> + +<p>Running a native build under QEMU is about 1/3 the speed of cross-compiling, +but it's a lot easier and more reliable.</p> + +<p>A trick to accelerate the build is to use distcc to call out to the +cross-compiler, feeding the results back into the emulator through the virtual +network. This is still a TODO item.</p> + +<p>Stage 3 is a fairly straightforward +<a href=http://www.linuxfromscratch.org>Linux From Scratch</a> approach, +except that we use BusyBox and uClibc in place of a couple dozen other +packages.</p> + +<h2>Stage 4: Package the system into a firmware file.</h2> + +<p>The reason for the name Firmware Linux is that the entire operating system +(kernel, initramfs, and read-only squashfs root filesystem) are glued together +into a single file. A modified version of LILO is included which can boot and +run this file on x86.</p> + +</body> +</html>
--- a/www/index.html Mon Nov 27 19:34:36 2006 -0500 +++ b/www/index.html Mon Nov 27 19:36:31 2006 -0500 @@ -1,6 +1,223 @@ <html> +<title>Firmware Linux</title> <body> -<p>Nothing to see yet, try <a href="http://landley.net/code/firmware">the old site</a> and email me if you're actually interested.</p> +<h1>Firmware Linux</h1> + +<b><h2>What is it?</h2></b> + +<p>Firmware Linux is an embedded Linux distribution builder. It's basically +a shell script that builds a complete Linux system from source code.</p> + +<p>FWL builds a cross-compiler and then uses it to build a minimal system +containing a native compiler, BusyBox and uClibc. Then it runs this minimal +system under an emulator (QEMU) and natively builds the final system. Finally +it packages the resulting system (kernel, initramfs, and root filesystem) into +one big bootable file.</p> + +<p>Here is a description of <a href=design.html>the design of Firmware +Linux</a>. That's the new (QEMU-based, capable of cross-compiling for non-x86) +design I'm working on now. The old (UML-based, x86 only) design is described +below.</p> + +<b><h2>Download</h2></b> + +<p>The current stuff is available from <a href=/hg/firmware>the mercurial +repository. That's the new (QEMU-based, capable of cross-compiling for +different hardware platforms) design I'm working on now, and where new +development happens.</p> + +<p>The old (UML-based, x86 only) design is still available from <a href=old>the +old website</a>, which is hideously out of date but contains a working +(ancient) version.</p> + +<b><h2>Documentation</h2></b> + +<p>Here's a quick <a href=build-process.html>overview of the Firmware Linux +build process</a>.</p> + +<p>Here is a description of <a href=design.html>the design of Firmware +Linux</a>.</p> + +<b><h2>Status</h2></b> + +<p>I've been working on this project on and off since 1999, it's what +got me into BusyBox and uClibc and compilers and so on. Now it's where I put +together everything else I'm doing (like toybox and tinygcc) to see what +actually works and give it a good stress-test. (Eating your own dogfood, +and all that.)</p> + +<p>The project stalled while I was BusyBox maintainer (2005-2006) due +to lack of time, and since then most of my spare programming time has gone +into launching toybox. But sinice one of the main goals of toybox is to +replace BusyBox in Firmware Linux, as toybox matures it'll naturally lead +to spending more time working on FWL.</p> + +<p>This server does not currently run on Firmware Linux. Making it do so +is a TODO item. After that, I'd like to get it to the point where I can +use it on my laptop. :)</p> + +<hr> + +<p>Firmware Linux is a bootable single file linux system, based on busybox and +uClibc. After downloading the +<a href=downloads/firmware-build-0.8.9.tar.bz2>source code</a>, extract +it, read the README, and run "./build.sh". This will (eventually) +create a self-contained firmware-uml executable you can use to try out an +emulated version of Firmware Linux.</p> + +<p>Prebuilt versions are available: The <a href=downloads/base-uml>basic +build</a> is 2.5 megabytes (which includes the linux kernel, all command line +utilities, and the minimal set of shared libraries). The +<a href=downloads/devel-uml>full development environment</a> is 15 megabytes +(the base system plus enough development tools for Firmware Linux to rebuild +itself from source code).</p> + +<h2>News</h2> +<p>Read <a href=/notes.html>my development log</a> to see what I've been up +to on this project.</p> + +<p>Here's the <a href=lilo-length.patch>length patch</a> I'm using on lilo.</p> + +<p>I wrote a <a href=/writing/docs/UML.html>Quick and Dirty User Mode Linux HOWTO</a> if you've never played with UML before.</p> + +<h2>What is it?</h2> + +<h2>Firmware Linux is a bootable single file linux system.</h2> + +<p>Firmware Linux is one file containing a kernel, initramfs, read-only root +filesystem, and cryptographic signature. You can boot Linux from this file +as if it was a normal kernel image (a slightly modified LILO is required, +patches for GRUB and other bootloaders are a to-do item). You can upgrade +your entire OS (and any applications in the root filesystem) atomically, by +downloading a new file and pointing your bootloader at it.</p> + +<h2>Firmware Linux is a Linux distro using busybox and uClibc as the basis for +a self-hosting development environment.</h2> + +<p>When the Firmware Linux project started, busybox applets like sed and sort +weren't powerful enough to handle the "./configure; make; make install" of +packages like binutils or gcc. Busybox was usable in an embedded router or +rescue floppy, but trying to get real work done with it revealed numerous +bugs and limitations.</p> + +<p>Busybox has now been fixed, and in Firmware Linux Busybox functions as an +effective replacement for bzip2, coreutils, e2fsprogs, file, findutils, gawk, +grep, inetutils, less, modutils, net-tools, patch, procps, sed, shadow, +sysklogd, sysvinit, tar, util-linux, and vim. (Eventually, it should be +capable of replacing bash and diffutils as well, but it's not there yet.)</p> + +<p>The base system consists of uClibc-0.9.28, busybox-1.1-pre1, and +linux-2.6.13.2. (Currently, bash-2.05 is also included due to limitations in +the busybox built-in shell. This is a temporary measure until Busybox +is further improved.)</p> -<p>Some <a href=design.html>design notes</a>.</p> +<p>The build toolchain uses the base system plus binutils, gcc-core, bison, +linux-libc-headers, and make.</p> + +<p>The install software uses lilo, which needs bin86 and as86 to build it.</p> + +<p>The full development system (which creates a development environment +sufficient for Firmware Linux to rebuild itself from source) adds +m4, flex, bison, diffutils, zlib, bin86, and nasm.</p> + +<p>Busybox is effectively replacing all the following packages: +bzip2, coreutils, e2fsprogs, file, findutils, gawk, grep, inetutils, less, +modutils, net-tools, patch, procps, sed, shadow, sysklogd, sysvinit, tar, +util-linux, and vim. (Eventually, it should be capable of replacing bash +and diffutils as well, but it's not there yet.)</p> + +<h2>Design</h2> + +<p>The single file packaging combines a linux kernel (either a bootable kernel +or User Mode Linux executable), initramfs, squashfs partition, and +cryptographic signature.</p> + +<p>In 2.6, the kernel and initramfs are already combined into a single file. +At the start of this file is either the obsolete floppy boot sector (just +a stub in 2.6), or an ELF header which has 12 used bytes followed by 8 unused +bytes. Either way, we can use the 4 bytes starting at offset 12 to store the +original length of the kernel image, then append a squashfs root partition +to the file, followed by a whole-file cryptographic signature.</p> + +<p>A User Mode Linux executable should still run just fine (loading is +controlled by the ELF segments, the appended data is ignored). Note: don't +strip the file or the appended data will be lost.</p> + +<p>Loading the bootable kernel image requires a modified boot loader that +can be told the original size of the kernel, rather than querying the current +file length which would be too long. Hence the patch to Lilo allowing +a "length=xxx" argument in the config file.</p> + +<p>Upon boot, the kernel runs the initramfs code which finds the firmware +file. In the case of User Mode Linux, the symlink /proc/self/exe points +to the path of the file. A bootable kernel needs a command line argument +of the form firmware=device:/path/to/file (it can lookup the device in +/sys/block and create a temporary device node to mount it with; this is +in expectation of dynamic major/minor happening sooner or later). +Once the file is found, /dev/loop0 is bound to it with an offset (losetup -o, +with a value extracted from the 4 bytes stored at offset 12 in the file), and +the resulting squashfs is used as the new root partition.</p> +<p>The cryptographic signature can be verified on boot, but more importantly +it can be verified when upgrading the firmware. New firmware images can +be installed beside old firmware, and LILO can be updated with boot options +for both firmware, with a default pointing to the _old_ firmware. The +lilo -R option sets the command line for the next boot only, and that can +be used to boot into the new firmware. The new firmware can run whatever +self-diagnostic is desired before permanently changing the default. If the +new firmware doesn't boot (or fails its diagnostic), power cycle the machine +and the old firmware comes up. (Note that grub does not have an equivalent +for LILO's -R option; which would mean that if the new firmware doesn't run, +you have a brick.)</p> + +<h2>Notes</h2> + +<p>Currently, the version of gcc it builds and uses only has a C compiler, not +c++. This restricts the packages I can build with it, and you'd be amazed +what kind of things need c++. (Python is the one I'm really missing at the +moment.) Possibly uclibc++ could help here, but I'm also looking at tcc +instead of gcc and binutils. (tcc doesn't quite build an unmodified Linux +kernel yet, and who knows what other packages would need to be tweaked, +but it's definitely worth a look once version 1.0 comes out. Perhaps +some kind of front-end could make it do c++?)</p> + +<p>User Mode Linux is used during the build for a number of reasons. The +kernel headers used to build the C library may be newer than the kernel the +system doing the build is using, and this may result in programs linked against +this C library trying to use new features the existing kernel doesn't have. +This tends to result in programs segfaulting. Building a UML kernel to +run these programs under during the build solves this problem, by translating +the calls into ones the host system understands.</p> + +<p>UML also avoids the need to run the build as root. The build needs to +mount partitions, associate files with looback devices, create device nodes, +create absolute paths requiring new entries in the root directory, chroot, +and so on. Doing all of this within the emulated UML environment avoids the +need for root permissions on the host.</p> + +<p>That said, if you're running the same kernel version the Firmware Linux +build is using, and you have root access, you can skip the UML wrapper to +speed up the build and make things more easily debuggable. I need to make a +wrapper script for this, but basically in sources/scripts, stage 0.0, 1.1, +and 2.2 are still needed, then 2.3 to package the final result (although 2.3 +depends on an executable built in 0.1).</p> + +<h2>How to build it</h2> + +<p>Run "./build.sh". This runs all the stages (numbered files in +sources/scripts) in sequence. It'll start by downloading all the source code +needed to build everything (which it'll keep around in the sources/packages +directory for future builds). If you just want to download the source, +run "sources/scripts/0.0-*". The 1-* stages create a cross-compile environment +independent of the parent system. The 2-* stages build the final system +by using that cross-compile environment.</p> + +<h2>Contact</h2> + +<p>My name is Rob Landley and my email address is rob@landley.net. +My <a href=notes.html>development log</a> is probably the best way to keep +track of what I'm working on, although I'll start a mailing list if enough +people pester me.</p> + +</body> +</html>