News


February 13, 2011

The repository is back up, and tweaked so it generates correct URLs without superfluous cgi names in them.

I see from the release RSS feed (go to repository viewer, then click "tags" and then "rss") that we're due for the next release in the first half of March.

I'd better get on that. (I'm aiming for a version of uClibc with NPTL working on at least x86/64, arm, mips, and powerpc. There's some work to do between now and then...)


February 8, 2011

The website was down again while we switched web hosts, but we're back up on dreamhost now, which has buckets more bandwidth and lets us have a mailing list again.

I still need to figure out how to enable mercurial on the new host, so the repository is down at the moment, but I should get that fixed up soon.


January 2, 2011

It's a month late, but Aboriginal Linux 1.0.1 is finally out, based on hg commit 1318, using Linux 2.6.36, uClibc 0.9.31, and BusyBox 1.18.0.

Automated native build control images

The big news this time is the native build control images, especially the one to build Linux From Scratch 6.7.

These control images drive the automated build infrastructure (first introduced back around release 0.9.8 and substantially improved since then). They allow you to fire up a system image under QEMU and run a completely automated native build.

To use a control image, cd into a system-image directory and run "./native-build.sh path/to/control-image.hdc". The result should wind up in the "uploads" subdirectory. To build control images from source, run "more/build-control-images.sh" and look in "build/control-images" afterwards. The build scripts live in "sources/control-images".

The lfs-bootstrap.hdc image builds most of Linux From Scratch 6.7. It doesn't replace the toolchain or C library, and stubs out internationalization support, but builds everything else (about 50 packages total). This build runs to completion reliably on i686, x86_64, and armv5l. Making it do that smoked out a lot of subtle bugs and configuration issues in the build environment, resulting in much improved system images. This LFS build sometimes finishes on mips and powerpc, and sometimes randomly dies for various asynchronous reasons that seem like something wrong in uClibc's pthreads locking code, although that's just a guess. (Since the uClibc developers (finally!) shipped a 0.9.32-rc1 with NPTL support on all targets, I expect the fix will involve upgrading. Part of the delay in this release was me playing around with that, but I'm releasing a checkpoint of the existing code before switching over to the new threading implementation.)

What else is new?

I fixed the static "dropbear" binary to call "ssh" out of $PATH instead of expecting dbclient at a fixed location, fixed the guest account's /etc/passwd and /etc/group entries so dropbear can use it, and added a /root directory to the system images so you can add a .ssh directory to it to make dropbear happy (possibly by mounting a tmpfs on it first).

Build infrastructure tweaks: sources/download_functions.sh is now its own file, and various fixes (only apply patches that end in ".patch") . The root filesystem's sbin/init.sh now mounts /dev/pts. Bugfix for SIMPLE_ROOT_OVERLAY, and multiple bugfixes in ccwrap.

Lots of tweaks to the native-build stuff. Redo control image generation, add bootstrap-skeleton with common code for bootstrapping distros (it has its own README), and rebase the half-finished gentoo-bootstrap on top of that. Upgrade "bisectinate" debugging script to take its test from the command line.

Several tweaks to uClibc to build LFS packages, and tweaks to the base kernel config to support NFSv3 and devtmpfs.

The new more/chroot-splice.sh script combines a root filesystem directory and a control image directory into a combined version you can chroot into and run a build from. (It looks for each in the location the build puts them, build/root-filesystem-$ARCH and build/control-images/$IMAGENAME respectively.) The new more/zapchroot.sh script unmounts everything under a directory, which is useful when playing with a root-filesystem via chroot.

Target support

I've fallen a bit behind on regression testing the targets, and hope to catch up next release. (Upgrading the kernel and QEMU breaks things that used to work.)

The i486, i586, i686, x86_64, powerpc, armv4l, armv4tl, and armv5l targets all work (for me) without qualification.

Last I checked, qemu still didn't support armv4eb or m68k, so I can't test those, but they build. (The kernel .config is almost certainly wrong for any target you'll actually be able to run them on, but the toolchain and root filesystem are probably ok.)

Updating the kernel broke armv6l: attempting to boot it under qemu exits with an "unimplemented cp15 register write" error. I'm not sure whether this is a kernel bug, a qemu bug the upgrade triggered, or some kind of misconfiguration. The proper fix is probably to look at the other arm boards QEMU can emulate for newer processors, and switch to one of those.

QEMU had a MIPS bug that prevented the system images from working for a version or two (such as the one in Ubuntu 10.04). It's fixed in current qemu-git, and isn't actually a problem with the mips, mipsel, or mips64 system images, they work fine under older or newer versions of qemu.

Sparc broke again (failure to launch init), and Oracle's patent trolling makes it harder than ever to care. It's on the todo list...

The only system image that doesn't current compile is sh4. (As noted in the 0.9.11 release notes, the sh4 maintainer broke it on purpose, which also makes it hard for me to care much about that target.)

The problem with sh4 is once again the kernel, this time a build break:

arch/sh/kernel/process_32.c:303: error: conflicting types for 'sys_execve'
/home/landley/play/two/build/temp-sh4/linux/arch/sh/include/asm/syscalls_32.h:24: error: previous declaration of 'sys_execve' was here

And even if I bothered to fix that, another change in the sh4 code changed the serial devices so QEMU's serial console no longer produces output. (Note that the sh4 test image on qemu.org uses a 2.6.29 kernel, which may be the last time the thing actually worked for anybody who isn't in a financial relationship with Renesas.)

Last I checked, QEMU still didn't support armv4eb or m68k, so I can't test those. I have patches to make m68k work on aranym and armv6eb work on actual big endian arm hardware, but haven't merged them yet.


November 3, 2010

My apologies for the lack of notice, the impactlinux.com domain unexpectedly went away saturday. The landley.net mirror is still up, and is becoming the main site again for the moment despite its' limited bandwidth. I'll see if I can recover the mailing list subscriptions, but I wouldn't count on it.

Still working on the next release, currently scheduled for around the start of December. Until we get a new mailing list, the source control log shows what's changing, and you can always email me directly, rob at landley dot net.


September 5, 2010

There's an old saying, "eventually you need to shoot the engineers and go into production". Aboriginal Linux can now bootstrap a native development environment on an arbitrary target, and build packages under the result. There's always more to do, but it does what it set out to do. And thus:

Aboriginal Linux 1.0 is out, based on hg commit 1238, using Linux 2.6.35, uClibc 0.9.31, and BusyBox 1.17.2.

Yes, it's been over five months since the last release. I didn't want to ship yet another intermediate release before doing 1.0, but I was trying to run out of todo items before shipping. (Yeah, I should know better. Sorry for the delay.)

The following are release notes, not proper documentation. Proper documentation (including an introduction to the whole thing) is the next big TODO item.

Targets

The arm, mips, powerpc, and x86 targets all work. Sparc works statically linked, but dynamic linking is still broken. Khem Raj fixed mips64 dynamic linking, but the native compiler still doesn't work. The m68k target builds again, although QEMU still doesn't emulate it. The sh4 target and big endian arm are are still broken.

The snapshots page needs updating. The documentation needs updating.

System image updates

The native-build.sh script moved into each system image. This means you can now download a system image tarball and a control image, then run a native build without any dependencies on the Aboriginal Linux build scripts. This script is a wrapper around dev-environment.sh which takes one argument, the name of the control image file.

The build only puts dev-environment.sh and run-from-build.sh into a system image when the system image includes native development tools.

Control Images

This release adds prebuilt binary build control images, generated by more/build-control-images.sh and shipped in the control-images directory. These are squashfs filesystems containing source code and an init script to build it and upload it to the host (using the $FTP_SERVER and $FTP_PORT variables, set by native-build.sh).

If these images are loaded into /dev/hdc of the emulated system, the system image's init script will mount that on /mnt and run /mnt/init instead of a shell prompt, so the build is completely automated.

Control images are provided to build hello world, build static dropbear and strace binaries, run the busybox test suite, and an in-progress script to bootstrapping Gentoo Stage 1 does about the first 2/3 of the job.

Root filesystem improvements

The root filesystem broke into two stages: simple-root-filesystem just contains a simple "defconfig busybox plus uClibc" system, enough to boot to a shell prompt. The native-compiler stage is like cross-compiler only unprefixed, built to run on the target, and including make, bash, and distcc. The root-filesystem stage combines the two into a single root filesystem with native development tools.

If the root filesystem is mounted read only, the init script will now mount a tmpfs on /home if there's no /dev/hdb to provide writeable scratch space. Chroot environments should now use Google's 8.8.8.8 nameserver by default. The distcc binaries moved out of /bin to the /distcc directory.

The "oneit" binary became a standalone program, and uses the / directory as PID 1's working directory, allowing you to umount /home if necessary. Toybox's "patch" command went upstream into busybox, and busybox's defconfig once again supports netcat server mode, so toybox was removed. We're now using a defconfig busybox, so configuring it properly is their problem.

Build scripts

The kernel configuration got collated to sources/baseconfig-linux (just like baseconfig-uClibc), and the various packages were migrated to use that. The kernel config is now the baseconfig plus the settings entry $LINUX_CONFIG. You can still supply a miniconfig-linux file to override this, and some architectures still do that. The ones that have been migrated to use a shared baseconfig are a lot more regular now, and the differences between architectures broken out and isolated.

The build stages were broken up to be more granular. The root-filesystem.sh stage now merely collates the output of two previous stages: simple-root-filesystem.sh (which produces just enough of a root filesystem to boot to a shell prompt, based on uClibc and busybox plus a few init scripts and config files), and native-compiler.sh (a portable native toolchain).

The system-image.sh stage now generates the dev-environment.sh script. It only puts dev-environment.sh and native-build.sh scripts in the tarball when NO_NATIVE_COMPILER is blank (and thus the build includes native development tools), or if root-filesystem isn't available in which case it'll automatically use simple-root-filesystem.

Several more package builds were moved to sources/sections.

The new "more" subdirectory collects together all the user-runnable scripts that aren't build stages. (Many of these were previously in sources/more.) New scripts in this directory include:

Website

A new cron job is populating the snapshots directory. Currently only the stable versions, but the unstable versions should wander back in eventually. (When the bisectinate and migrate-kernel scripts catch up to what they need to do to properly test unstable packages.)

More documentation: The "about" page got redone, and the FAQ was fluffed out with a new debugging section. Michael S. Zick wrote up mips docs for the architecture page, and the build scripts grew even more comments.

General Infrastructure

The control image generation was upgraded so each control image source can have its own patch directory and file directory. (This needs more work.)

Environment variable sanitization: the sanitize_environment function unsets any environment variable it doesn't recognize, either from a whitelist or from config. Clearing all unrecognized environment variables insulates the build from some of the strange environments out there. To disable this behavior, export NO_SANITIZE_ENVIRONMENT=1.

New config variables:

ccwrap improvements: Handle unexported $PATH better. It doesn't need to search $PATH when argc[0] is a path (either absolute or relative to the current directory). Also use Scrt1.o when building a shared library. (Oddly this never caused a problem before, but do the right thing anyway.)

Upgrades to BUILD_STATIC: it now takes a comma separated list of packages, with the special value "all" to build everything static, and "none" to build nothing static. The default is to build static versions of busybox and the toolchain packages (gcc, binutils, make, and ccwrap) both due to the increased performance this provides under qemu (which has to repeatedly re-translate executable code pages that modify themselves, such as when dynamic linking patches per-process jump addresses), and also due to the increased portability this provides the native compiler tarball.

(The host-tools stage always builds dynamic packages, because the glibc maintainer has a personal dislike of static linking, meaning glibc's static linking support (in addition to creating 400k "hello world" binaries) is both broken[LINK] and deprecated[LINK]. Some distros have even taken to breaking out the *.a format libraries used for static linking into a separate package, and not installing them by default. The host-tools stage now tests for this, and refuses to continue in a dynamic-only build environment unless you set BUILD_STATIC=none. You can rebuild under the resulting target system to natively compile static binaries, which uClibc supports quite well, but you can't generate them from the host.)

The old FROM_ARCH is now HOST_ARCH, and the infrastructure using it (cross-compiler.sh and native-compiler.sh) has been cleaned up. The HOST_UTILS variable was also removed, that's now built for the full cross and native compilers, but not the simple cross compiler.

The more/smoketest.sh file no longer creates hdb.img.

The host-tools stage has been upgraded to work around the utter stupidity of Ubuntu 10.04, which turned "gcc" into a perl script. (Yes really. No, I don't know what they were smoking.)

Build zlib in host-tools (for squashfs) and in static-tools for dropbear's -C. This is necessary for reproducing the build under a system-image, because the default system image type (squashfs) requires zlib to build.

Update the command prompt in system images to show the amount of parallelism available via distcc. (When it says 1 you haven't got distcc, and when it says a number greater than that you have.) The prompt now looks like:

(powerpc:4) /home #

The make, bash, and distcc builds are now part of the native-compiler stage (and their build scripts moved to sources/sections). This means that a native toolchain now includes make, because it's hard to build make without make. This stage also records the uClibc and linux config files for posterity.

The build was weaned off of toybox. (Broke out oneit, sent patch upstream to busybox, and got netcat fixed in defconfig so native-build.sh could use it.)

Several small bugfixes and cosmetic cleanups. For example, we now delete empty build/temp-* directories when we're done with them, and several config options such as NO_NATIVE_COMPILER and NO_CLEANUP should be more reliable now. Don't build hello-dynamic on a BUILD_STATIC=all system. Clean up debris files the kernel's headers_install leaves lying around. Minor upgrades to FORK=1 support, and QUIET is now part of FORK instead of a separate control. Generating a package manifest should no longer produce error messages. Empty temp directories get deleted on the way out. Better support for building older tools as alt-packages (tested binutils 2.14 and gcc 3.4). Comment out default UNSTABLE= URLs in download.sh, supply your own if you want that. Binutils build now honors existing $CFLAGS (if any). The dienow error message goes to stderr. More reliably track package cache state when patches fail to apply. Simplify system-image build (don't try to package root filesystem and build kernel in parallel anymore, squashfs and make both have internal SMP support). BUILD_VERBOSE is now more verbose. Reliability improvement for record-commands.sh. New shell functions blank_workdir and extract_package. Consistently use name-name in scripts and name_name in functions.

It's been a long 5 months...


May 2, 2010

The name of the project is changing from Firmware Linux to Aboriginal Linux. The new URL is "http://landley.net/aboriginal".

Some reasons to move away from the old name are listed here, and some reasons to pick the new name are covered in this thread.

Some links may be broken while we switch over. Let us know what we've missed on the mailing list...


March 29, 2010

Version 0.9.11 is out, based on hg commit 1020. We're closing in on a 1.0 release, but not quite there yet.

This version upgrades to Linux 2.6.33, uClibc 0.9.30.3, and BusyBox 1.16.1.

Improved native builds

Most targets now have prebuilt binaries for dropbear, strace, busybox. These are statically linked against uClibc, so just download, set the executable bit, and run.

These dropbear and strace binaries are built using the automated three-disk native build capability introduced back in November, which has been improved and genericized this release. Now one script creates a build image (see examples in sources/native-builds) and another script launches a system image with that build image attached as the third hard drive (/dev/hdc), telling the system image to automatically build those source packages and upload the result to the host via ftp.

So to reproduce the dropbear and strace binaries for a supported ARCH, you could do:

./build.sh ARCH
sources/native-builds/static-tools.sh tools.hdc
sources/more/native-build.sh ARCH tools.hdc output

When the build exits, the results should be in the host's "output" directory.

Targets

The arm, mips, x86, x86-64 targets that worked last time still work. The armv4eb and m68k targets still don't work.

The powerpc and sparc targets have been upgraded. Powerpc now works with the 3-disk automated native build (although you'll need qemu 0.12.4 to take advantage of this). The sparc dynamic linker still doesn't work in uClibc, but the statically linked system is now working well enough to build packages natively.

New targets this release are i486 (thanks David Seikel) and mips64 (which mostly works, but the native toolchain isn't quite properly configured to build stuff with yet).

Finally, the sh4 target was broken by a cosmetic cleanup of the startup assembly code in Linux 2.6.33 that rendered any binutils version older than 4 months unusable to build sh4. While it's possible to patch around this, it hasn't yet been worth effort. The Linux-sh4 maintainer is aware of the issue, but explicitly doesn't care about users other than his employer, and so far nobody else seems to care enough about supporting the discontinued Sega Dreamcast game console to argue with him. (Might revisit this next release, but m68k support is a higher priority.)

Infrastructure

The system-image's "run-emulator.sh" is now a much smaller and simpler script, consisting primarily of the qemu command line. (And if you export DEBUG=1 it prints the emulator command line.) The distcc and hdb setup code moved to dev-environment.sh.

The screenshots page is working again, if a bit stale.

The bisectinate.sh script (to automatically git bisect a build issue) has been updated, more to come next release.

The download.sh script has been updated to understand FORK=1 and download packages in parallel. (And to extract them in parallel when EXTRACT_ALL=1.)

The new export_if_blank shell function allows several more environment variables to be overridden, including TOP, SOURCES, SRCDIR, PATCHDIR, BUILD, SRCTREE, HOSTTOOLS, WRAPDIR, CC, and STAGE_NAME.

I'd also like to thank the debugging efforts of Wolfgang Denk, who hit as many bugs trying to use this project as I normally do with other people's. He eventually got discouraged and left, but thanks to his efforts:


February 2, 2010

Version 0.9.10 is out, based on hg commit 966.

Yeah, I know 1.0 is overdue for a release, here's a resync point with the 2.6.32 kernel and uClibc 0.9.30.2.

There are some regressions this time around, which I plan to fix up in the next release. The 2.6.32 kernel broke strace, and upgrading to the new version didn't fix it. (I know how, I just haven't yet.) The m68k target isn't currently building due to the uClibc upgrade (yeah, "bugfix only", I know), and the armv6l target seems to be broken two different ways. I screwed up the screenshots page (which is halfway converted to Vladimir's new fancy version with style sheets, but "halfway" doesn't actually display right). The cron job building nightly snapshots got taken down when the server's hard drive filled up, and now it's in the process of being upgraded with a couple new scripts:

The new sources/more/migrate_kernel.sh script does a "make oldconfig" between the stable and "USE_UNSTABLE=linux" kernels, and creates a new alt-miniconfig-linux for the specified target with any new symbols that showed up. (This means the snapshots should break less, but adding the symbols to the native configs will still require manual filtering.)

Also, the new sources/more/bisectinate.sh script automatically does a git bisect between a known good and known bad (generally HEAD) version of one of the unstable packages (linux, uClibc, or busybox). It can test that the build completes, or that the system image boots and successfully builds the static native dropbear binary. (In theory this will allow the nightly snapshots to automatically isolate and report the first commit that broke them.)

In other general infrastructure news: some extensive upgrades to the compiler wrapper (ccwrap), such as no longer needing GIMME_AN_S at compile time and instead autodetecting the presence of libgcc_s.so vs libgcc.a and behaving appropriately. A largish cleanup/refactoring of the compiler build (described in commit 944) added a new "native-compiler.sh" script, which handles building the improved "cross-compiler" tarball that includes thread support and uClibc++ and is statically linked against uClibc on the host. The simple cross compiler is now "simple-cross-compiler.sh" (which is good enough to build a system image, but isn't very useful in other contexts).

The host-tools.sh stage now only builds e2fsprogs or squashfs if the build is actually going to use them (based on the SYSIMAGE_TYPE config symbol). The sources/sections scripts now come in two flavors: an *.build version with setup/cleanup called automatically and *.sh files which do their own setup/cleanup within the script. Each system image's run-emulator.sh will once again add /sbin:/usr/sbin to the $PATH if it can't find mke2fs.

The --extract command line option went away from download.sh because everything else works via environment variables. Use EXTRACT_ALL=1 instead.

Fixed a few regressions that snuck into previous versions: the config-{linux,uClibc,busybox} should once again be in the root filesystem's usr/src directory, buildall.sh should now try to build the static native dropbear/strace binaries for all targets (not just the last one) and it should properly be running smoketest.sh again.


December 8, 2009

Version 0.9.9 is out, based on hg commit 921.

Just a checkpoint on the way to 1.0, which is still planned for around new years, but there's been some schedule slippage already.


November 7, 2009

Version 0.9.8 is out, based on hg commit 876. (If you want to see all the changes in this release, look at commits 810 through 876.)

This release upgrades linux to 2.6.31.4 and busybox to 1.15.2. (The uClibc version is still 0.9.30.1, they haven't had a new release yet.)

The old #firmware irc channel on freenode.net has been merged into #edev, so go there now to ask questions about this project. (A _lot_ of knowledgeable, helpful people hang there who can answer your embedded development questions.)

Bug tracker and roadmap

The new bug tracker has a roadmap showing what we plan to implement for the next release or two.

It's also a place to file bugs, although sending them to the mailing list is still the fastest way to get a fix.

New screenshots page

This page lists every target, with a "screenshot" of the target's boot messages under qemu, and associated links to all the various prebuilt binaries for that target.

Fun with static linking

The cross-compiler-$ARCH tarballs are all compiled for i686 and statically linked against uClibc for maximum portability.

The busybox binary in the root filesystem images is now built statically by default, providing a 20% speed improvement in ./configure times under qemu (due to the reduced page translation overhead). You can switch this back to dynamic with "BUILD_STATIC=none".

Static dropbear and strace binaries for each target are also available, built by the new sources/more/native-static-build.sh script.

System Image interface changes

The run-from-build.sh script used to set up a development environment out of the build directory, but didn't provide an easy way to do so based on downloaded tarballs. (You had to work out and provide extensive command line arguments to the run-emulator.sh script.) This has been fixed.

The run-emulator.sh script in each system interface tarball is now much simpler. It now looks for distccd and $ARCH-cc in its $PATH, and sets up distcc acceleration automatically if it finds them. (It announces whether or not it has managed to do so, and init.sh announces whether or not it thinks it has distcc acceleration enabled just before launching a command prompt.) So setting up the distcc accelerator should be less of a pain.

Also, run-emulator.sh no longer takes any command line arguments. Instead the environment variable $HDB indicates a /dev/hdb disk image to mount on /home, $HDBMEGS specifies the size of a (sparse ext2) HDB image to create if it doesn't already exist, and QEMU_MEMORY indicates how many megabytes of physical memory the emulator should allocate for the virtual system.

Each system-image also provides a simple wrapper dev-environment.sh which calls run-emulator.sh with HDBMEGS=2048, HDB=hdb.img, and QEMU_MEMORY=256. (These are the values run-from-build.sh used to provide, and give a reasonable build environment with 256 megs of physical memory and 2 gigabytes of writeable /home space.) The run-from-build.sh script itself now merely adds the specified build/cross-compiler-$ARCH directory to the $PATH, changes directory to the appropriate build/system-image-$ARCH, and calls ./dev-environment.sh. It's 3 lines.

The system images can now also automatically mount a third disk image ($HDC, specifying an /dev/hdc image to mount onto the /mnt directory), and if that filesystem contains an executable "init" program in its root directory the sbin/init.sh script will run that instead of launching a command prompt.

The new native-static-build.sh uses this HDC capability to automatically run some native package builds without having to drive the build through /dev/console via a wrapper script (the way smoketest.sh does). See sources/more/setup-native-static-build.sh for the script which generates the hdc.sqf image for this.

Note that some targets (such as sh4 and powerpc) don't yet support /dev/hdc due to QEMU board emulation limitations.

Build interface changes

The new "sources/more" directory contains scripts the end user can call, but which aren't important enough to clutter up the top level directory with. The buildall.sh and smoketest-all.sh scripts moved there, as did the cronjob.sh used to build the nightly snapshots.

The new sources/more/native-static-build.sh natively compiles dropbear and strace under the emulator, and copies them out to the host through the virtual network (into the build/cron-temp directory) using busybox ftpd on the host. This provides an example of how to use the new HDC /mnt/init functionality to perform automatic builds under the emulator, and copy the results out through the network. It uses the new sources/timeout.sh wrapper to detect hung builds (defined as builds that haven't produced a line of output within a given number of seconds).

The new BINARY_PACKAGE_TARBALLS variable (in the file "config") tells the build to create a separate tarball of the binaries produced building each package. (These more granular tarballs aren't shipped yet, but may be in a future release.)

The new SKIP_STRIP variable tells the build not to strip binaries. (Add in CFLAGS=-g for enormous binaries full of debug info.)

Target upgrades

Powerpc and sh4 should now work fine with stock qemu 0.11.0.

The arm platform now has an armv4tl target. This is a little-endian armv4 eabi, for armv4 chips offering the "thumb" extension which eabi requires. (The plain armv4l target is still oabi. The oabi/eabi thing is different binary ABIs, a bit like Linux vs BSD binaries or aout vs elf. The eabi standard is the new one, but some of the older arm hardware can't support it. This takes it down as far as it can go.)

Infrastructure

Behind the scenes, some extensive work is going on to simplify the build scripts. (The documentation's fallen a bit behind reality here, but should be back up to date next release.)

The uClibc miniconfigs are now automatically generated, with the bulk of the config living in sources/baseconfig-uClibc and the few target specific lines being appended from the UCLIBC_CONFIG variable defined in the target's settings file.

The other big change is factoring out lots of common code, moving it into the sources/sections directory, so things like the uClibc, busybox, and toolchain builds now live in one place and are called from the various other scripts that need them. (This may eventually allow alternate build scripts for these packages, potentially swapping out glibc for uClibc, or llvm/clang for gcc. In the meantime, it eliminates a lot of redundant code.)

This required lots of small cleanups, such as making the scripts now consistently use the $STAGE_DIR variable for their output directory. Some other cleanups were primarily cosmetic, such as moving the extracted tarball cache directory from build/sources to build/packages. (Tarballs live in packages, extracted tarballs live in build/packages.)

The root filesystem now contains simple /etc/passwd and /etc/group files, because dropbear won't work without them (not even to ssh out).

Several small bugfixes (ccwrap no longer segfaults if $PATH isn't exported, /etc/resolv.conf doesn't get overwritten unnecessarily by init.sh, etc).


August 20, 2009

Version 0.9.7 is out, a little over a month and a half late. It's a larger update than usual, so the release notes are a bit long.

This release is based on mercurial version 807 of the build scripts, and includes Linux 2.6.30.4 and BusyBox 1.14.3. No new uClibc release is out since last time.

Lots of little bugs got fixed. For example, this release should build on Fedora 11 (and any other platform that hasn't got "which" installed by default), and now supports distcc and ccache on the host (and will use them automatically if they are installed).

Changes to build stages: new root-filesystem, cross-static, and native-compiler tarballs, upgraded buildall.sh and smoketest-all.sh.

The old "mini-native" stage has been renamed "root-filesystem" (the same way package-mini-native became system-image last release). The script names (and resulting tarballs) now indicate what each stage makes. So build.sh calls host-tools.sh, cross-compiler.sh, root-filesystem.sh, and system-image.sh, in that order.

The old build-static-toolchains.sh script is gone, with its functionality integrated into buildall.sh (which now builds i686 hosted static toolchains for each target by default). This is implemented by two new environment variables triggering optional behavior: STATIC_CROSS_COMPILER_HOST and BUILD_STATIC_NATIVE_COMPILER.

Setting STATIC_CROSS_COMPILER_HOST to an architecture name creates a cross compiler statically linked to run on that host (via a technique known as canadian cross compile). For example, the invocation:

BUILD_CROSS_COMPILER_HOST=i686 ./build.sh armv4l

Would create a tarball called "cross-static-armv4l.tar.bz2" which was built to run on an i686 host, and was statically linked (against uClibc) on that host to be distribution independent.

Setting BUILD_STATIC_NATIVE_COMPILER to any non-empty value creates a native compiler for the target (packaged as native-compiler-$ARCH.tar.bz2), which is statically linked against uClibc so it can be extracted and run on just about any system of the appropriate type. (So if you can get a Linux shell prompt and have enough storage and memory, you should be able to compile stuff.)

The upgraded smoketest-all.sh script runs smoketest.sh on each target built. (The smoketest.sh script attempts to compile and run "hello world" natively within a system image under qemu, and indicates whether or not it worked via "pass" or "fail".) Note that smoketest-all.sh now autodetects which targets have been built (have system images in the build directory), and smoketest.sh times out after 60 seconds so targets that hang will return failure eventually.

Squashfs 4.0 now the default system image type

The root filesystem image type is now squashfs by default. (To change it set SYSIMAGE_TYPE to ext2 or initramfs, either in your environment or the file "configure".)

This involved upgrading the userspace squashfs support package to the version 4.0 supported by the upstream kernel, and numerous small fixes to work with a read-only root filesystem (with writeable spaced mounted on /home and /tmp).

Target upgrades

ARM upgrades

Lots of work on the arm targets, with the addition of a new armv6l target and conversion of the armv5l target to ARM EABI (which used to be documented here, but ARM screwed up its website and now you have to go here). The armv4l target still provides an ARM v4 little endian soft float OABI target (which is "the i386 of ARM", meaning it should run on just about any arm hardware, if a bit slowly). A new armv4be target is the same as armv4l except big endian.

A new hw-tct-hammer target supports the Tin Can Tools hammer/nail board.

Updated sparc support, almost works now. (uClibc for sparc still sucks.)

The sparc target has also been upgraded to the point where it now gives a shell prompt. (And immediately dies with a bus error or a hang if you try to do anything, because uClibc for sparc still doesn't work. The "upgrade" consisted of statically linking everything because uClibc's dynamic loader doesn't work for sparc.)

PowerPC remains fiddly due to QEMU issues, new powerpc-440fp target

PowerPC support in QEMU is officially unstable, so although FWL's powerpc target hasn't significantly changed from last release, the version of QEMU in source control no longer runs it (nor the binary powerpc system images released with FWL 0.9.6) and QEMU's developers do not consider this a regression. While we search for a new Linux kernel .config that matches QEMU's current expectations (and await a release considered "stable"), note that setting HOST_BUILD_EXTRA=1 before running host-tools.sh will build a version of qemu that does run powerpc and put it in build/host where run-from-build.sh can find it. (Note also that building qemu from source still takes significantly longer than the rest of the host-tools.sh stage combined.)

A new powerpc-440fp target produces code for the powerpc 440 with hardware floating point. (Theoretically QEMU's "bamboo" board emulation should run this, but QEMU doesn't emulate a 440 processor yet. We compromise by running a ppc440 filesystem under qemu's power mac emulation, to take advantage of the ppc440 instruction set being almost a subset of full powerpc. This gets us a shell prompt, but throws illegal instruction errors if you try to do anything fancy.)

(Note that powerpc-440fp is not a hw-target for powerpc, it's a different processor instruction set like i586 vs i686. Alas, one that qemu doesn't properly emulate yet.)

Now using last GPLv2 release of GCC, and future plans

The upgrade to gcc 4.2.1 gives us the last GPLv2 licensed version of gcc (thus avoiding various problems). Make 3.81 was already the last GPLv2 release of that project.

Note that bash 2.05b is still intentionally ancient (to avoid the bloat of 3.x), and may be replaced with busybox ash as the default FWL shell in a future release. Similarly, although binutils 2.17 was the last GPLv2 version from the FSF, an upgrade to the last GPLv2 release of Red Hat's binutils fork, 2.17.5.0.17 is under consideration for a future release. But mostly the GNU compiler toolchain has been upgraded as far as possible for the foreseeable future, either because new versions provide no significant benefits or because they're not available under a license compatible with the Linux kernel's.

Future upgrades to these packages will either be via individual patches (maintaining GPLv2 license) or by investigating alternative non-FSF packages such as OpenBSD's PCC or Apple's LLVM. In preparation for support of alternate compilers, the build scripts have moved from using "gcc" to the more generic "cc" for the compiler name wherever possible, but currently no drop-in replacement compiler is available.

Implementation details

Each target configuration renamed "details" to "settings", and added a "description". (This makes Gentoo From Scratch's kconfig easier.)

The download.sh stage now populates a "packages" directory instead of "sources/packages", meaning that everything in the sources directory now comes from the FWL source tarball. To do the equivalent of "make clean" you can "rm -rf build", and to do a "make distclean" you can "rm -rf packages build".

The build no longer deletes the alt-* tarballs when UNSTABLE isn't set, you must do that by hand if you want an updated alt- version of a package. The packages/MANIFEST file creation can now use the extended tarball information to detect git version numbers.

The === bars now show the current stage name and architecture each package is being built for. (Pipe the build output to grep "^===" to see just the start of each new package build.) The same information is sent to the title bar of your xterm, export NO_TITLE_BAR=1 if you don't want the build to update the title bar.

The BUILD_STATIC option now applies to all packages built in root-filesystem.sh, and can be set from a target configuration's settings file (as sparc is now doing).

The BUILD_VERBOSE option passes V=1 to the linux, busybox, and uClibc builds that otherwise sanitize their output.

You can now set NATIVE_TOOLCHAIN=only to build just the compiler and not the other parts (busybox, make, toybox, bash...) This is used by the canadian cross steps in build.sh to create static cross and native compilers.

This release broke up scripts/include.sh so that script just sets lots of environment variables, and now includes scripts/functions.sh to define shell functions. The architecture setup (reading architecture files and creating temporary directories and such) is now done via the "read_arch_dir $ARCH" shell function, so merely including sources/include.sh should no longer have significant side effects. Similarly, setupfor now has an EXTRACT_ONLY environment variable instead of depending on a blank $ARCH (which host-tools.sh has, despite wanting to snapshot source so it can build stuff).

Various fixes to make parallel builds work better. (So it neither leaves background processes running nor kills the parent shell and closes your xterm when you hit ctrl-c.) The new "killtree" shell function, which kills a process and all its children and grandchildren recursively with pgrep. We also build our own sources/toys/mysetsid.c program with the host compiler (because the normal command line "setsid" doesn't also do a tcsetpgrp on stdin so signal handling isn't forwarded to the new session). This should more reliably allow ctrl-c to exit smoketest.sh and system-image.sh and such, and let FORK=1 ./buildall.sh stop all its background processes when interrupted. (Note that smoketest-all.sh also honors FORK=1.)

The logs from buildall.sh now live in build/logs.

The new config option ROOT_NODIRS avoids creating the normal set of empty directories in the new root filesystem. (The old /tools support went away, but a BUILD_STATIC ROOT_NODIRS build can be extracted into /tools and used from there. Then either symlink "/lib" to "/tools/lib" or "export UCLIBC_DYNAMIC_LINKER=/tools/lib/ld-uClibc.so.0" to add more stuff to /tools natively.)

The compiler wrapper (sources/toys/ccwrap.c) can now use architecture-specific WRAPPER_TOPDIR values. (I.E. armv4l-cc could "export armv4l_WRAPPER_TOPDIR=/path/to/directory".) This makes canadian cross compiles a lot easier.


April 2, 2009

Version 0.9.6 includes Linux 2.6.29 and uClibc 0.9.30.1, now with support for sh4 and upgraded powerpc support.

The big news is the Gentoo From Scratch project, which extends a FWL system image into a Gentoo Stage 1 environment, building natively inside qemu.

The documentation has been extensively rewritten (but needs more). The source now contains a README and each image has a usr/src/MANIFEST file listing the packages it built from.

The cross and native toolchains now include ldd, readelf, and ldconfig. Added support for hardware targets (an unfinished example of which is hw-wrt610n) that use the same cross compiler and system image as a qemu platform, but build/package a different kernel to boot on actual hardware.

Several new build options:

The run-emulator.sh scripts of system images have several new options, do "./run-emulator.sh --help" for a list. Takes several environment variables such as $QEMU_EXTRA and $KERNEL_EXTRA. Root filesystems can now run in read only mode with "KERNEL_EXTRA=ro".

Lots of build infrastructure upgrades. The build stages have been reorganized somewhat (most noticeably package-mini-native.sh is now called system-image.sh). The "sources/packages" directory has moved up a directory (to separate repository files from stuff downloaded at runtime). Upgrades to the cc wrapper and the USE_UNSTABLE infrastructure. Better stripping of the cross compilers and system images, so everything's a bit smaller. The mirror list now lives in download.sh. New buildall.sh is a wrapper around build.sh. New build-static-toolchains.sh builds them under qemu; set FORK=1 to build them in parallel just like buildall.sh. The ext2 system images are now built at a small size with genext2fs and then expanded with resize2fs (because genext2fs slows down exponentially as size increases; this means e2fsprogs is built in host-tools.sh now). Distcc has been upgraded, and should work more reliably now.

Too many little bug fixes to list.


December 16, 2008

The mailing list moved to firmware at impactlinux.com. The link in the nav bar on the left has changed, but the old archives are still there for the moment.

December 3, 2008

After several false starts (I.E. last minute bugs, yes I test this thing before shipping it), Version 0.9.5 is out, inching towards a 1.0 release. Lots of new work this time.

The biggest news is uClibc 0.9.30 and uClibc++ 0.2.2. The native toolchain now supports C++, hooked up to distcc and everything.

The documentation has been extensively rewritten. (There's always more to do, but most of it isn't horribly out of date anymore.)

The packaging stage now uses genext2fs instead of User Mode Linux, so the most brittle part of the build has been removed. FWL can now fully rebuild itself under itself, all the way through the packaging stage.

My personal website's getting a touch overwhelmed, so a higher bandwidth site is now available at impactlinux.com. This should become the new primary website going forward.

BusyBox has also been upgraded (to 1.13.0), and the new USE_UNSTABLE infrastructure allows test versions of any package to be downloaded and built without forking the build scripts, so future package upgrades can be dealt with as side projects without blocking other important work.

The packaging stage now uses genext2fs instead of User Mode Linux, so the most brittle part of the build has been removed. FWL can now fully rebuild itself under itself, all the way through the packaging stage.

The old forkbomb script has been replaced with sources/build-all-targets.sh. with an argument --fork to build all targets in parallel, and "--fork 3" to limit the parallelism. This script can run from a cron job, producing a "buildall" directory with tarballs, compressed logs, and a README file listing the version of each package. The environment variable USE_STATIC_HOST specifies an $ARCH to create statically linked cross compilers for. (The build will then try to use them for the later stages, and won't get far if you specify an arch that can't run on the host.)

The include.sh script has been broken up, with a new sources/functions.sh containing shell functions (like download, setupfor, and cleanup) and a new "config" setting the user-editable environment variables that control the build. Several of these variables have been renamed and their number has grown. You can now specify the colors for each build stage using the variables at the end of config.

Building mini-native in a Linux From Scratch style /tools directory is no longer the default. Instead set NATIVE_TOOLSDIR=1 if you want that kind of layout instead of a more conventional layout. (Similarly, NATIVE_TOOLCHAIN=1 now enables the native toolchain, but that variable defaults to enabled in config.)

See the docs for details.


August 6, 2008

Version 0.9.0 is out (changeset 378) and can rebuild itself under itself. (The packaging step still requires User Mode Linux to create ext2 images, which only works on x86 and x86-64 hosts. The next release should replace that with something more portable.) The root filesystem is now based on BusyBox 1.11.1 (plus a few bug fixes).

The prebuilt binary images have been moved into their own binaries directory. The prebuilt binary cross compilers are statically linked against uClibc for better portability. Updated the README to better describe the contents of the system-image tarballs.

The forkbomb.sh script can now take a number of parallel targets to launch, for example "./forkbomb.sh --fork 3". (The resulting display is kind of horrible but the out-$ARCH.txt files are still legible.) Leaving off the number still builds all targets in parallel, with the original display.

The include.sh script was reorganized so that interesting environment variables the user might want to export for themselves are set near the top of the script. Currently this includes:

Updated documentation is in progress...


June 6, 2008

Version 0.4.0 is out (changeset 345), with kernel 2.6.25.4 and the "distcc trick" working out of the box.

The distcc trick accelerates a native build by calling out to the cross compiler. To use it, download the appropriate cross-compiler-$ARCH and system-image-$ARCH tarballs, run the "run-with-distcc.sh" from the system image directory with the path to the cross compiler directory as its first argument. (You need to have distcc installed on the host system.)

After building from source, the cross-compiler and system-image directories remain in the build directory. The "emulator-build.sh" runs these with distcc acceration set up. The script "smoketest.sh" is a wrapper around emulator-build.sh which feeds a script into qemu's stdin to compile and run a multi-threaded "hello world" under the emulator, using distcc and the cross compiler.

If smoketest.sh works for a platform, it means a lot of stuff worked to get that far. The cross compiler worked to build the system, and the emulated system booted so QEMU, the uClibc config, and the kernel config agreed. The emulated system has a working virtual hard drive and serial port, and for distcc to work it has a working virtual network connection. Running "make" would also require a working realtime clock.

The seven commands missing in order to run the FWL build under itself are bzip2, sort, diff, wget, install, od, and find. (The busybox versions, where available, had a bug, and they're not in toybox yet.) I'm trying to get those in for the next release in September.

Sorry I missed a release back in April. Shouldn't happen again.


January 29, 2008

Version 0.3.1 is out (changeset 275), with kernel 2.6.24. The images are now tarballs each containing the ext2, zImage, and run script files. The run scripts now run qemu-setup.sh by default so /proc, /sys, /dev and the virtual network are initialized, and the default shell is busybox ash (so cursor keys and history work).

Sparc is still broken (tracked it down to a dynamic linking bug, I think, haven't fixed it yet) and m68k is failing to build with an internal compiler error (next release I should upgrade the compiler). All the other targets have a working virtual network, and all but powerpc exit qemu when you exit the PID 1 command shell.

Building on an x86-64 host< should no longer be insanely slow (the gcc-min-heapsize arguments that allow 32-bit hosts to build in 128 megs of ram do very impolite things to 64-bit hosts), and the resulting x86-64 cross compiler should now be properly relocatable.

Better build dependency tracking: the host-tools script is now populating a single directory with all the commands needed by the build, and restricting $PATH to that directory (so it can't accidentally call any commands that aren't explicitly listed). It's building toybox and busybox to provide most of these commands (the remaining ones are listed towards the end of include.sh), and creating symlinks to the remaining ones out of the original $PATH. This allows it to build on a wider range of host distributions.


November 2, 2007

Version 0.3.0 is out (changeset 242), with kernel 2.6.23. PowerPC finally works under qemu (network and everything), and it's building a m68k target (although I have no emulator to run that). That means there are ten targets, of which eight run under qemu. (Sparc is still broken, let me know if anybody actually cares.)

The new "./forkbomb.sh --fork" script builds all the targets in parallel, or use --nofork if you haven't got the memory for that (and --watch to watch and existing build if you ctrl-c out of the display before it's done).

The build is now smarter about extracting only one copy of the source code (in build/sources) and re-using it for multiple builds, building out of tree (via cp -rs to create a tree full of symlinks so the package doesn't need native out-of-tree support).


September 2, 2007

Added a porting guide. I'm working on m68k, ppc, big endian arm, and maybe blackfin targets for the next release, and thought other people might be interested in the process. (The vast majority of it is fiddly debugging. These are the other steps.)


June 27, 2007

Version 0.2.2 is out (changeset 185). Using uClibc 0.9.29 release, several small bugfixes, new mips big-endian config, actually work when /bin/sh->dash (mostly by explicitly saying /bin/bash everywhere), automatically detect the number of CPUs in the machine to feed -j to make, and better diagnostic output.


April 9, 2007

Version 0.2.1 is out (changeset 156). Significant upgrades to armv4l: the native compiler works now, and it's using soft-float. Added armv5l and i586 targets to show how to do that kind of variant. Added preliminary powerpc support (although qemu can't run it yet). Reorganized the download directory and rewrote the README. Added squashfs back (the kernel supports it, but it's not using it yet). Moved uClibc to miniconfig. (Yes, the patch for this is huge, and yes I need to push it all upstream into the Linux kernel. On the bright side, this one builds on a host that doesn't have curses installed.) Stopped trying to build qemu (at least until they get the gcc 3.x dependencies cleaned out), now just warn if it's not there (so far the build only uses it to sanity-test the cross compiler, which is now skipped automatically if it's not there).

I note that some versions of Ubuntu (Edgy and up) are pushing a political agenda. They install bash, but point /bin/sh to dash instead. (So why install bash if you're not going to use it?) This breaks all sorts of things, and if somebody can tell me how to force make to run "/bin/bash" instead of "/bin/sh", I'll consider trying to deal with the mess. Until then, I suggest "rm /bin/sh; ln -s bash /bin/sh" to get a working system. (I'm no fan of bash, but dash is just broken.)

(Those of you who use vi and want to be able to use the cursor keys while in insert mode might want to do something similar with vi->vim. Ubuntu seems to be trying to train users to change their habits. Dog biscuits would be a more honest approach.)

Update: I've been pointed at this as evidence Ubuntu isn't pushing a political agenda. Ok, maybe it's just really bad technical judgement. Are they honestly saying that people write shell scripts for the speed? And how do you make a size argument and then install bash by default in ADDITION to the other shell? Boggle. In 1991 after Linus Torvalds taught his term program to understand the minix filesystem so he could upload and download without rebooting, he taught it to handle the system calls of bash so he could rm/mv/mkdir without rebooting. That's where Linux 0.0.1 came from. Bash was the default Linux shell _before_ 0.0.1. Be very careful messing with that kind of a de-facto standard, guys. Dash is garbage. Background a process with & and then hit ctrl-c: it'll kill it. It doesn't understand blah/{blah,blah} file list syntax. It can't do "source file.sh" (just ". file.sh", which is unreadable). I repeat: I'm not a fan of bash (yeah, the bloat's getting crazy), but dash is broken.

And that still doesn't explain why "vi" can't use the cursor keys in insert mode, but "vim" can...


March 6, 2007

Version 0.2.0 is out. This one builds ext2 images which can boot up to a shell prompt under qemu. This works on i686, armv4l, x86_64, and mipsel. (Sparc's broken. Anyone who cares about sparc is welcome to debug it.)

There are now prebuilt ext2 image files in the download directory. The kernels are in the mini-native tarballs. (Yeah, I should repackage that next release.) The system emulator invocation for each platform is the appropriate sources/configs file, and run-mini-native.sh invokes 'em.

The mailing list is in the usual place if you have any questions.


January 14, 2007

There's a new irc channel for the project, #firmware on freenode.

Changeset 88 builds i686, x86_64, armv4l, mips, and sparc. I've made a release tarball of that (firmware 0.1.1, "It works for me"), and updated the prebuilt cross-compiler tarballs in the downloads directory. I've added a big README and prebuilt mini-native tarballs for each platform (although packing them up into something qemu can boot is currently left as an exercise for the reader).

Speaking of which, I'm currently working on adding ext2 packaging (via toybox) to the build scripts, so qemu system emulation can boot the result and then run the next stage automatically.


December 28, 2006

Changeset 68 builds a native build environment with a working toolchain. And in celebration, I've revamped the website with an actual navigation bar and content and such.

I should put out a release soon.


December 5, 2006

Changeset 27 builds a relocatable armv4l cross-compile toolchain! Download the tarball, run ./download.sh, then run ./build.sh, then grab the "build/cross-compiler" directory and use "bin/armv4l-unknown-linux-gnu-gcc" out of that (which I admit is a bit of a mouthful, for which I blame the FSF).


August 6, 2006

Mecurial repository created. Nothing to see yet, move along...