toybox

About Download Development

Q: Do you capitalize toybox?

A: Only at the start of a sentence. The command name is all lower case so it seems silly to capitalize the project name, but not capitalizing the start of sentences is awkward, so... compromise. (It is _not_ "ToyBox".)

Q: "Why is there toybox? What was wrong with busybox?"

A: Toybox started back in 2006 when I handed off BusyBox maintainership and started over from scratch on a new codebase after a protracted licensing argument took all the fun out of working on BusyBox.

Toybox was just a personal project until it got relaunched in November 2011 with a new goal to make Android self-hosting. This involved me relicensing my own code, which made people who had never used or participated in the project loudly angry. The switch came after a lot of thinking about licenses and the transition to smartphones, which led to a 2013 talk laying out a strategy to make Android self-hosting using toybox. This helped bring it to Android's attention, and they merged it into Android M.

The answer to the second question is "licensing". BusyBox predates Android by almost a decade but Android still doesn't ship with it because GPLv3 came out around the same time Android did and caused many people to throw out the GPLv2 baby with the GPLv3 bathwater. Android explicitly discourages use of GPL and LGPL licenses in its products, and has gradually reimplemented historical GPL components such as its bluetooth stack under the Apache license. Similarly, Apple froze xcode at the last GPLv2 releases (GCC 4.2.1 with binutils 2.17) for over 5 years while it sponsored the development of new projects (clang/llvm/lld) to replace them, implemented its SMB server from scratch to replace samba, and so on. Toybox itself exists because somebody with in a legacy position just wouldn't shut up about GPLv3, otherwise I would probably still happily be maintaining BusyBox. (For more on how I wound up working on busybox in the first place, see here.)

Q: Why a 7 year support horizon?

A: Our longstanding rule of thumb is to try to run and build on hardware and distributions released up to 7 years ago, and feel ok dropping support for stuff older than that. (This is a little longer than Ubuntu's Long Term Support, but not by much.)

If a kernel or libc feature is less than 7 years old, I try to have a build-time configure test for it and let the functionality cleanly drop out. I also keep old Ubuntu images around in VMs and perform the occasional defconfig build there to see what breaks. (I'm not perfect about this, but I accept bug reports.)

My original theory was "4 to 5 18-month cycles of moore's law should cover the vast majority of the installed base of PC hardware", loosely based on some research I did back in 2003 and updated in 2006 which said that low end systems were 2 iterations of moore's law below the high end systems, and that another 2-3 iterations should cover the useful lifetime of most systems no longer being sold but still in use and potentially being upgraded to new software releases.

It turns out I missed industry changes in the 1990's that stretched the gap from low end to high end from 2 cycles to 4 cycles, and _that_ analysis ignored the switch from PC to smartphone cutting off the R&D air supply of the laptop market. Meanwhile the Moore's Law s-curve started bending down in 2000 and these days is pretty flat because the drive for faster clock speeds stumbled then died, and the subsequent drive to go wide maxed out around 4x SMP with ~2 megabyte caches for most applications. These days the switch from exponential to linear growth in hardware capabilities is common knowledge.

But the 7 year rule of thumb stuck around anyway: if a kernel or libc feature is less than 7 years old, I try to have a build-time configure test for it and let the functionality cleanly drop out. I also keep old Ubuntu images around in VMs and perform the occasional defconfig build there to see what breaks.

Where do I start understanding the source code?

Toybox is written in C. There are longer writeups of the design ideas and a code walkthrough, and the about page summarizes what we're trying to accomplish, but here's a quick start:

Toybox uses the standard three stage configure/make/install build, in this case "make defconfig; make; make install". Type "make help" to see available make targets.

The configure stage is copied from the Linux kernel (in the "kconfig" directory), and saves your selections in the file ".config" at the top level. The "defconfig" target selects the maximum sane configuration (enabling all the commands and features that aren't unfinished, only intended as examples, debug code, etc) and is probably what you want. You can use "make menuconfig" to manually select specific commands to include, through an interactive menu (cursor up and down, enter to descend into a sub-menu, space to select an entry, ? to see an entry's help text, esc to exit). The menuconfig help text is the same as the command's --help output.

The "make" stage creates a toybox binary (which is stripped, look in generated/unstripped for the debug versions), and "install" adds a bunch of symlinks to toybox under the various command names. Toybox determines which command to run based on the filename, or you can use the "toybox" name in which case the first argument is the command to run (ala "toybox ls -l"). You can also build individual commands as standalone executables, ala "make sed cat ls".

The main() function is in main.c at the top level, along with setup plumbing and selecting which command to run this time. The function toybox_main() implements the "toybox" multiplexer command.

The individual command implementations are under "toys", and are grouped into categories (mostly based on which standard they come from, posix, lsb, android...) The "pending" directory contains unfinished commands, and the "examples" directory contains examples. Commands in those two directories are _not_ selected by defconfig. (These days pending directory is mostly third party submissions that have not yet undergone proper code review.)

Common infrastructure shared between commands is under "lib". Most commands call lib/args.c to parse their command line arguments before calling the command's own main() function, which uses the option string in the command's NEWTOY() macro. This is similar to the libc function getopt(), but more powerful, and is documented at the top of lib/args.c.

Most of the actual build/install infrastructure is shell scripts under "scripts". These populate the "generated" directory with headers created from other files, which are described in the code walkthrough. All the build's temporary files live under generated, including the .o files built from the .c files (in generated/obj). The "make clean" target deletes that directory. ("make distclean" also deletes your .config and deletes the kconfig binaries that process .config.)

Each command's file contains all the information for that command, so adding a command to toybox means adding a single file under "toys". Usually you start a new command by copying an existing command file to a new filename (toys/examples/hello.c, toys/examples/skeleton.c, toys/posix/cat.c, and toys/posix/true.c have all been used for this purpose) and then replacing all instances of its old name with the new name (which should match the new filename), and modifying the help text, argument string, and what the code does. You might have to "make distclean" before you new command shows up in defconfig or menuconfig.

The toybox test suite lives in the "tests" directory. From the top level you can "make tests" to test everything, or "make test_sed" test a single command's standalone version (which should behave identically) but that's why we test.