--- a/www/about.html	Wed Nov 12 14:56:26 2008 -0600
+++ b/www/about.html	Wed Nov 12 14:57:05 2008 -0600
@@ -2,11 +2,22 @@
 
 <h2>What is Firmware Linux?</h2>
 
-<p>Firmware Linux (FWL) is an embedded Linux build system implemented as a
-series of shell scripts.  The full build produces a complete
-<a href=downloads/image>bootable linux system</a> (based on uClibc and
-BusyBox/toybox) for various hardware platforms (including arm, mips, ppc, x86,
-and x86-64).</p>
+<p>Firmware Linux (FWL) is an embedded Linux build system designed to
+eliminate the need for cross compiling, replacing it with native
+compiling under emulation instead.</p>
+
+<p>FWL consists of a series of shell scripts which create a complete
+<a href=downloads/image>bootable linux system</a> for a given target hardware
+platform.  This provides a minimal native Linux build environment for the
+target, loosely based on <a href=http://linuxfromscratch.org>Linux From
+Scratch</a> (stripped down to an embedded variant built from busybox, uClibc,
+gcc, binutils, make, bash, and the Linux kernel).  Running this native build
+environment under an emulator such as QEMU (or on real target hardware if
+applicable) allows fully native builds for arbitrary target platforms to
+be performed on cheap and powerful commodity PC hardware.</p>
+
+<p>The currently supported target hardware platforms include arm, mips, powerpc,
+x86, and x86-64, with partial support for sh4, mips, and sparc.</p>
 
 <p>Firmware Linux is loosely based on <a href=http://linuxfromscratch.org>Linux
 From Scratch</a>, stripped down and rebased on busybox and uClibc.</p>
@@ -19,42 +30,44 @@
 platform, eliminating the need for cross compiling in favor of
 <a href="#native_compiling">native compiling under emulation</a>.</p>
 
-<h2>License</h2>
-
-<p>Firmware Linux is licensed under GPL version 2.  This means that the build
-scripts are licensed under GPL version 2, and each of the component packages
-of the default "mini-native" root filesystem may be redistributed under the
-terms of GPLv2 (and in the case of uClibc, LGPLv2).</p>
-
-<p>Adding your own packages to the resulting system is generally "mere
-aggregation" (see the last paragraph of section 2), and thus not my
-problem.  This license statement is not intended to apply to your packages
-as long as they are not derived works of existing GPLv2 code.</p>
-
 <h2>Using the prebuilt binaries</h2>
 
-<h3>qemu-image-*.tar.bz2</h3>
+<h3>system-image-*.tar.bz2</h3>
 
-<p>Each qemu-image tarball contains an ext2 root filesystem image, a kernel
+<p>Each system-image tarball contains an ext2 root filesystem image, a kernel
 configured to run under qemu, and shell scripts to run the system (in various
-modes) under the emulator <a href=http://qemu.org>qemu</a>.</p>
+modes) under the emulator <a href=http://bellard.org/qemu/>qemu</a>.</p>
 
 <p>To test boot an FWL target system under qemu 0.9.1, download the appropriate
 <a href=downloads/image>prebuilt binary tarball</a> for the target you're
-interested in, extract it, cd into it, and execute the "./run-emulator.sh"
-script in that directory.</p>
+interested in, extract it, cd into it, and execute one of the following
+shell scripts:</p>
+
+<ul>
+<li><p><b>./run-emulator.sh</b> - This boots the target Linux system under the
+appropriate emulator, with /dev/console hooked to the emulator's stdin and
+stdout.  (I.E. the shell prompt the script gives you after the boot messages is
+running in the emulator.)  Type "cat /proc/cpuinfo" to confirm you're running
+in the emulator, then play around and have fun.</p></li>
 
-<p>This boots the target Linux system under the emulator, with /dev/console
-hooked to the emulator's stdin and stdout.  (I.E. the shell prompt the script
-gives you after the boot messages is running in the emulator.)  Type "cat
-/proc/cpuinfo" to confirm you're running in the emulator, then play around and
-have fun.</p>
+<li><p><b>./run-with-home.sh</b> - Wrapper for run-emulator.sh which adds
+a second virtual hard drive mounted on /home.  (If the file "hdb.img"
+doesn't exist on the host, this script automatically creates a 2 gigabyte
+sparse file and formats it ext3.)</p></li>
+
+<li><p><b>./run-with-distcc.sh</b> - Wrapper for run-with-home.sh which
+sets up the appropriate magic so the emulator will automatically call out to
+the cross compiler on the host system via distcc.  This can significantly speed
+up build times under the emulator, while still avoiding most of the complexity
+of cross compiling.  (<a href="#distcc_trick">More info</a>)</p>
+</ul>
 
 <p>The FWL boot script for qemu (/tools/bin/qemu-setup.sh) populates /dev
 from sysfs, sets up an emulated (masquerading) network (so you can wget
 source packages or talk to <a href="#distcc_trick">distcc</a>), and creates
 a few symlinks needed to test build normal software packages (such as making
-/lib point to /tools/lib).</p>
+/lib point to /tools/lib).  It also mounts /dev/hdb (or /dev/sdb) on /home
+if a second emulated drive is present.</p>
 
 <p>For most platforms, exiting the command shell will exit the emulator.
 (Some, such as powerpc, don't support this yet.  For those you have to kill
@@ -394,4 +407,17 @@
 (ext2, initramfs, jffs2).</li>
 </ol>
 
+<h2>License</h2>
+
+<p>Firmware Linux is licensed under GPL version 2.  This means that the build
+scripts are licensed under GPL version 2, and each of the component packages
+of the default "mini-native" root filesystem may be redistributed under the
+terms of GPLv2 (and in the case of uClibc, LGPLv2).</p>
+
+<p>Adding your own packages to the resulting system is generally "mere
+aggregation" (see the last paragraph of section 2), and thus not my
+problem.  This license statement is not intended to apply to your packages
+as long as they are not derived works of existing GPLv2 code.</p>
+
+
 <!--#include file="footer.html" -->

--- a/www/design.html	Wed Nov 12 14:56:26 2008 -0600
+++ b/www/design.html	Wed Nov 12 14:57:05 2008 -0600
@@ -2,46 +2,233 @@
 
 <h1>The Firmware Linux build process</h1>
 
-<p>Firmware Linux is an embedded Linux distribution builder, which creates a
-bootable single file Linux system based on uClibc and BusyBox/toybox.  It's
-basically a shell script that builds a complete Linux system from source code.</p>
+<h1>Overview</h1>
+
+<h2>User Interface</h2>
+
+<p>The script "./<b>build.sh</b>" calls the other scripts in the correct
+sequence to perform a full build for one or more architectures.  Its argument
+is the architecture to build, and running it without arguments lists the
+available architectures.  (Configuration information for each architecture is
+in the sources/targets directory.)</p>
+
+<p>Each stage can also be run (or re-run) individually.  The individual stages
+are described in the next section.</p>
+
+<p>
+
+<p>The script "<b>sources/forkbomb.sh</b>" builds all architectures at once.
+Its command line arguments specify whether to build them in series (--nofork) or in
+parallel (--fork).  The output of each architecture's build is saved in a log
+file named "out-$ARCH.txt".  A quick scan of each log for success or failure
+is available with "./forkbomb.sh --stat".  Run it with no arguments to see
+all available options.</p>
+
+<p>Setting the following environment variables to non-blank values can modify
+the behavior of the build:</p>
+
+<ul>
+<li><p><b>FWL_NO_TOOLCHAIN</b> - This tells mini-native.sh not to include a
+compiler toolchain (binutils, gcc, bash, make, and distcc), but instead just
+build a small uClibc/busybox system.</p>
+
+<p>Setting FWL_NO_TOOLCHAIN="headers" will leave the libc and kernel header
+files in the appropriate include directory, for use by a compiler such as
+pcc, llvm/clang, or tinycc.  (Building and installing additional tools
+such as "make" remains your problem.)</p>
+</li>
+
+<li><p><b>FWL_TOPDIR</b> - Path to the top level directory within the
+native filesystem.  If this is blank, configure sets this to a default value
+of "/tools", to act as a Linux From Scratch style chroot environment.  For
+a more traditional filesystem layout, set this to "/" or to "/usr".</p></li>
+
+<li><p><b>FWL_RECORD_COMMANDS</b> - Records all command lines used to build each
+package.</p>
+
+<p>Tells host-tools.sh to build a logging wrapper (sources/toys/wrappy.c) and
+populate a directory (build/wrapper) with symlinks to that wrapper for each
+command name in $PATH.  This allows later build stages to write log files in
+the build directory (named "cmdlines.${STAGE_NAME}.${PACKAGE_NAME}") recording
+each command run.  (When build/wrapper exists, include.sh sets the wrapper
+control variables $WRAPPY_LOGDIR, $WRAPPY_LOGPATH, and $WRAPPY_REALPATH,
+then adjusts $PATH to point to the wrapper directory, which makes the wrapper
+append to the appropriate log file before calling the actual command.)</p>
+
+<p>Afterwards, the script "sources/toys/report_recorded_commands.sh" can
+generate a big report on which commands were used to build each package for
+each architecture.  To get a single list of the command names used by
+everything, do:</p>
+
+<blockquote>
+<p>echo $(find build -name "cmdlines.*" | xargs awk '{print $1}' | sort -u)</p>
+</blockquote>
+
+<p>(Note: this will miss things which which call executables at absolute values
+instead of checking $PATH, but the only interesting ones so far are the
+#!/bin/bash type lines at the start of shell scripts.)</p>
+</li>
+
+<li><p><b>FWL_BUILD_STATIC</b> - Tells cross-compiler.sh to statically link all
+binaries in the cross compiler toolchain it creates.</p>
+
+<p>The prebuilt binary versions in the download directory are statically linked
+against uClibc, by building a mini-native environment and re-running the build
+under that with BUILD_STATIC=1.</p></li>
+
+<li><p><b>FWL_PREFERRED_MIRROR</b> - Tells download.sh to try to download
+packages from this URL first, before falling back to the normal mirror list.
+For example, "FWL_PREFERRED_MIRROR=http://landley.net/code/firmware/mirror".</p></li>
+
+<li><p><b>FWL_USE_TOYBOX</b> - Tells the host-tools.sh and mini-native.sh to
+install the <a href=http://landley.net/code/toybox>toybox</a> implementation
+of commands (where available) instead of the busybox versions.</p></li>
+</ul>
+
+<h1>Implementation</h1>
 
-<p>FWL builds a cross-compiler and then uses it to build a minimal system
-containing a native compiler, BusyBox and uClibc.  Then it runs this minimal
-system under an emulator (QEMU) to natively build the final system.  Finally it
-packages the resulting system (kernel, initramfs, and root filesystem) into
-a single file that can boot and run (on x86 by using a modified version of LILO).</p>
+<p>The top level wrappers (<b>build.sh</b> or <b>forkbomb.sh</b>) call the
+other stages in sequence.  The other stages are <b>download.sh</b>,
+<b>host-tools.sh</b>, <b>cross-compiler.sh</b>, <b>mini-native.sh</b>, and
+<b>package-mini-native.sh</b>.  Each script sources the common file
+<b>include.sh</b>.</p>
+
+<p>In theory, the stages are othogonal.  If you have an existing cross
+compiler, you can add it to the $PATH and skip cross-compiler.sh.  Or you
+can use _just_ cross-compiler.sh to create a cross compiler, and then go build
+something else with it.  The host-tools.sh stage can often be skipped
+entirely.</p>
+
+<h2>Stage 0: Setup</h2>
+
+<p>Before building anything we're going to keep, we need to do some setup
+work.</p>
+
+<ul>
+<li><p><b>include.sh</b> - header file defining common environment variables
+and functions, and performing other miscelanous setup.</li>
+
+<p>This script is not run directly, instead it's included from all the other
+scripts.</p>
+
+<p>For building packages, this file defines the functions:</p>
+<ul>
+<li><p><b>setupfor</b> - extracts a source package (named in the first
+argument) into a temporary directory, and changes the current directory
+to there.</p>
+
+<p>Source code is cached, meaning each package's source tarball is only
+actually extracted once (into build/sources) and the temporary copies
+are directories full of hard links to the cached source.</p>
+</li>
+
+<li><p><b>cleanup</b> - delete temporary copy of source code after build.</p>
+</li>
+</ul>
+
+<li><p><b>download.sh</b> - Download source code packages from the web.</p>
+
+<p>This file is a series of calls to the <b>download</b> function (defined in
+include.sh).  If an existing copy of the tarball matching the sha1sum $SHA1
+doesn't currently exist in the sources/packages directory, it uses wget to
+fetch it from $URL (or a series of fallback mirrors).</p>
 
-<p>Firmware Linux builds in stages:</p>
+<p>A blank value for $SHA1 will accept any file as correct, ignoring its
+contents.</p>
+
+<p>After downloading all tarballs, the function <b>cleanup_oldfiles</b> deletes
+any old files (meaning any files in sources/packages with a timestamp
+before the call to download.sh, such as previous versions left over
+after a package upgrade).</p>
+
+<p>Running this stage with the argument "--extract-all" will extract all
+the tarballs, to preopulate the cache used by setupfor.  (This is primarily
+used to avoid race conditions in forkbomb.sh --fork.)</p>
+
+<li><p><b>host-tools.sh</b> - Set up a known environment on the host</p>
+
+<p>This script populates the <b>build/host</b> directory with
+host versions of the busybox and toybox command line tools (the same ones
+that the target's eventual root filesystem will contain), plus symlinks to the
+host's compiler toolchain.</p>
+
+<p>This allows the calling scripts to trim the $PATH to point to just this
+one directory, which serves several purposes:</p>
+
+<ul>
+
+<li><p><b>Isolation</b> - This prevents the ./configure stages of the source
+packages from finding and including unexpected dependencies on random things
+installed on the host.</p></li>
+
+<li><p><b>Portability</b> - Using a known set of command line utilities
+insulates the build from variations in the host's Linux distribtion (such as
+Ubuntu's /bin/echo lacking suport for the -e option).</p></li>
+
+<li><p><b>Testing</b> - It ensures the resulting system can rebuild itself
+under itself, since the initial build was done with the same tools we
+install into the target's root filesystem.  The initial build acts as a smoke
+test of most of the packages used to create the resulting system, and
+restricting $PATH ensures that no necessary commands are missing.  (Variation
+can still show up between x86/arm/powerpc versions, of course.)</p></li>
+
+<li><p><b>Enumeration</b> - The RECORD_COMMANDS functionality (see
+environment variables, above) starts here and continues into the later build
+scripts.</p></li>
+</ul>
+
+<p>This stage is optional.  You don't need to run this stage if you don't
+want to.  If the build/host directory doesn't exist (or doesn't contain
+a "busybox" executable), the build will use the host's original $PATH.</p>
+</ul>
 
 <h2>Stage 1: Build a cross-compiler.</h2>
 
-<p>The first stage builds a cross-compiler,
-which runs on the host system and produces binaries that run on the target
-system.  (See my <a href=/writing/docs/cross-compiling.html>Introduction to
-cross compiling</a> if you're unfamiliar with this.)</p>
+<ul>
+<li><p><b>cross-compiler.sh</b> - build a cross compiler for a target.</p></li>
 
-<p>We have to cross-compile even if the host and target system are both
-x86, because the host probably use different C libraries.  If the host has
+<p>This script builds a cross-compiler, which runs on the host system and
+produces binaries that run on the target system.  (See my
+<a href=http://landley.net/writing/docs/cross-compiling.html>Introduction to
+cross compiling</a> if you're unfamiliar with cross compiling.)</p>
+
+<p>The build requires a cross-compiler even if the host and target system are
+both x86, because the host usually uses different C libraries.  If the host has
 glibc and the target uses uClibc, then the (dynamically linked) target binaries
-we produce won't run on the host.  This is what distinguishes cross-compiling
-from native compiling: different processors are just one reason the binaries
-might not run.  Of course, as long as we've got the cross-compiling support
-anyway, we might as well support building for x86_64, arm, mips, or ppc
-targets...</p>
+we produce won't run on the host.  (Target binaries that won't run on the
+host are what distinguishes cross-compiling from native compiling.  Different
+processors are just one reason for it.)</p>
+
+<p>Building a cross-compiler toolchain requires four packages: binutils,
+gcc, uClibc, and the linux kernel (for header files).</p>
+
+</ul>
 
-<p>Building a cross-compiler toolchain requires four packages.  The bulk of
-it is binutils, gcc, and uClibc, but building those requires header files from
-the Linux kernel which describe the target system.</p>
+<h2>Stage 2: Cross compile a bootable target system.</h2>
 
-<h2>Stage 2: Use the cross-compiler to build a native build environment
-for the target.</h2>
+<ul>
+<li><p><b>mini-native.sh</b> - Build a minimal native development environment
+for the target system.</p>
+
+<p>This script uses the cross compiler from the previous step to build a kernel
+and root filesystem for the target, including a native compiler toolchain.
+The resulting system should boot and run on the target, or under an
+appropriate emulator.</p>
 
 <p>Because cross-compiling is persnickety and difficult, we do as little of
 it as possible.  Instead we use the cross-compiler to generate the smallest
 possible native build environment for the target, and then run the rest of the
 build in that environment, under an emulator.</p>
 
+<p>This script should perform all the cross compiling anyone ever
+needs to do.  It uses the cross-compiler to generate the simplest possible
+native build environment for the target which is capable of rebuilding itself
+under itself.</p>
+
+<p>Anything else that needs to be built for the target can then be built natively,
+by running this kernel and root filesystem under an emulator and building
+new packages there, bootstrapping up to a full system if necessary.</p>
+
 <p>The emulator we use is QEMU.  The minimal build environment powerful enough
 to boot and compile a complete Linux system requires seven packages: the Linux
 kernel, binutils, gcc, uClibc, BusyBox, make, and bash.  It's packaged
@@ -49,6 +236,9 @@
 /tools directory approach, staying out of the way so the minimal build
 environment doesn't get mixed into the final system.</p>
 
+</li>
+</ul>
+
 <h2>Stage 3: Run the target's native build environment under an emulator to
 build the final system.</h2>
 
@@ -440,4 +630,22 @@
 either /var/root or /home/root, change the passwd entry to do this), and life
 is good.</p>
 
+
+<!--
+<p>Firmware Linux is an embedded Linux distribution builder, which creates a
+bootable single file Linux system based on uClibc and BusyBox/toybox.  It's
+basically a shell script that builds a complete Linux system from source code
+for an arbitrary target hardware platform.</p>
+
+<p>The FWL script starts by building a cross-compiler for the appropriate
+target.  Then it cross-compiles a small Linux system for the target, which
+is capable of acting as a native development environment when run on the
+appropriate hardware (or under an emulator such as QEMU).  Finally the
+build script creates an ext2 root filesystem image, and packages it with
+a kernel configured to boot under QEMU and shell scripts to invoke qemu
+appropriately.</p>
+-->
+
+
+
 <!--#include file="footer.html" -->