From 0c51af01cc88b71eaad5fad3b9ac7c37245749e2 Mon Sep 17 00:00:00 2001
From: Rob Landley
Toybox's main goal is to make Android +
Toybox was trying to make Android self-hosting by improving Android's command line utilities so it can build an installable Android Open Source Project image @@ -16,7 +16,15 @@ Embedded Linux Conference explaining this plan (outline, video), Google merged toybox into AOSP and -began shipping toybox in Android Marshmallow in 2015.
+began shipping toybox in Android Marshmallow in 2015. Google then went +really weird between 2019 +and 2023, and has +continued alarmingly along that +trajectory, +but the fundamental problem of +phones and tablets being restricted devices rather than general purpose +computers (unlike the mainframes, minicomputers, and microcomputers that +came before) remains.Toybox aims to provide one quarter of a theoretical "minimal native development environment", which is the simplest Linux system capable of @@ -59,7 +67,7 @@ at ELC was devoted to this question, and has the following sections:
-A more recent talk from 2019 compares +
A talk from 2019 compares BusyBox vs toybox and explains the design decisions behind both. (A 2015 toybox talk was part of the channel @@ -67,6 +75,10 @@ and explains the design decisions behind both. but the outline is still available.)
+A 2026 talk describes how toybox's mkroot/mkroot.sh is intended to build the +simplest possible +Linux system.
+The toybox maintainer's previous minimal self-hosting system project, @@ -191,9 +203,9 @@ needs to add it after 1.0 to eliminate another gpl build prerequite from AOSP.)
[2] -The dividing line is +The dividing line was "Is there an acceptably licensed version Android can ship, or do we have -to write one?" Since android is not "GNU/Linux" in any way, we need to +to write one?" Since android was not "GNU/Linux" in any way, we needed to clean out all traces of gnu software from its build to get a clean self-hosting system.
diff --git a/www/code.html b/www/code.html index 58b3803d..7b338236 100644 --- a/www/code.html +++ b/www/code.html @@ -141,7 +141,7 @@ NOSPACE=1 to allow tests to pass with diff -bTo add a new command to toybox, add a C file implementing that command to -one of the subdirectories under the toys directory. No other files need to +one of the subdirectories under the toys directory. No other files need to be modified; the build extracts all the information it needs (such as command line arguments) from specially formatted comments and macros in the C file. (See the description of the "generated" directory @@ -195,7 +195,7 @@ link to other documentation or standards as appropriate.)
Update the USE_YOURCOMMAND(NEWTOY(yourcommand,"blah",0)) line. The NEWTOY macro fills out this command's toy_list -structure. The arguments to the NEWTOY macro are:
+structure. The arguments to the NEWTOY macro are:the name used to run your command
Change the kconfig data (from "config YOURCOMMAND" to the end of the comment block) to supply your command's configuration and help -information. The uppper case config symbols are used by menuconfig, and are -also what the CFG_ and USE_() macros are generated from (see [TODO]). The +information. The uppper case config symbols are used by menuconfig, and are +also what the CFG_ and USE_() macros are generated from (see [TODO]). The help information here is used by menuconfig, and also by the "help" command to -describe your new command. (See [TODO] for details.) By convention, +describe your new command. (See [TODO] for details.) By convention, unfinished commands default to "n" and finished commands default to "y", -so "make defconfig" selects all finished commands. (Note, "finished" means +so "make defconfig" selects all finished commands. (Note, "finished" means "ready to be used", not that it'll never change again.)
Each help block should start with a "usage: yourcommand" line explaining -any command line arguments added by this config option. The "help" command +any command line arguments added by this config option. The "help" command outputs this text, and scripts/config2help.c in the build infrastructure collates these usage lines for commands with multiple configuration options when producing generated/help.h.
@@ -249,7 +249,7 @@ command name, you'll need to "#define TT this.filename" yourself before #including toys.h if you want to use TT globalsRename hello_main() to yourcommand_main(). This is the main() function +
Rename hello_main() to yourcommand_main(). This is the main() function where execution of your command starts. Your command line options are already sorted into this.optflags, this.optargs, this.optc, and the GLOBALS() as appropriate by the time this function is called. (See @@ -308,14 +308,14 @@ specific to this command.
This file sucks in most of the commonly used standard #includes, so individual files can just #include "toys.h" and not have to worry about -stdargs.h and so on. Individual commands still need to #include +stdargs.h and so on. Individual commands still need to #include special-purpose headers that may not be present on all systems (and thus would prevent toybox from building that command on such a system with that command -enabled). Examples include regex support, any "linux/" or "asm/" headers, mtab +enabled). Examples include regex support, any "linux/" or "asm/" headers, mtab support (mntent.h and sys/mount.h), and so on.
The toys.h header also defines structures for most of the global variables -provided to each command by toybox_main(). These are described in +provided to each command by toybox_main(). These are described in detail in the description for main.c, where they are initialized.
The global variables are grouped into structures (and a union) for space @@ -327,13 +327,13 @@ local variables.
Contains the main() function where execution starts, plus common infrastructure to initialize global variables and select which command -to run. The "toybox" multiplexer command also lives here. (This is the +to run. The "toybox" multiplexer command also lives here. (This is the only command defined outside of the toys directory.)
Execution starts in main() which trims any path off of the first command name and calls toybox_main(), which calls toy_exec(), which calls toy_find() and toy_init() before calling the appropriate command's function from -toy_list[] (via toys.which->toy_main()). +toy_list[] (via toys.which->toy_main()). If the command is "toybox", execution recurses into toybox_main(), otherwise the call goes to the appropriate commandname_main() from a C file in the toys directory.
@@ -342,7 +342,7 @@ directory.struct toy_list toy_list[] - array describing all the -commands currently configured into toybox. The first entry (toy_list[0]) is +commands currently configured into toybox. The first entry (toy_list[0]) is for the "toybox" multiplexer command, which runs all the other built-in commands without symlinks by using its first argument as the name of the command to run and the rest as that command's argument list (ala "./toybox echo hello"). @@ -359,9 +359,9 @@ defining macros and #including generated/newtoys.h.
command.char *options - command line option string (used by get_optflags() in lib/args.c to intialize toys.optflags, toys.optargs, and -entries in the toy's GLOBALS struct). When this is NULL, no option +entries in the toy's GLOBALS struct). When this is NULL, no option parsing is done before calling toy_main().
int flags - Behavior flags for this command. The following flags are currently understood:
+int flags - Behavior flags for this command. The following flags are currently understood:
These flags are combined with | (or). For example, to install a command +
These flags are combined with | (or). For example, to install a command in /usr/bin, or together TOYFLAG_USR|TOYFLAG_BIN.
struct toy_list *which - a pointer to this command's toy_list -structure. Mostly used to grab the name of the running command -(toys->which.name).
+structure. Mostly used to grab the name of the running command +(toys->which.name).int exitval - Exit value of this command. Defaults to zero. The +
int exitval - Exit value of this command. Defaults to zero. The error_exit() functions will return 1 if this is zero, otherwise they'll return this value.
char **argv - "raw" command line options, I.E. the original -unmodified string array passed in to main(). Note that modifying this changes -"ps" output, and is not recommended. This array is null terminated; a NULL +unmodified string array passed in to main(). Note that modifying this changes +"ps" output, and is not recommended. This array is null terminated; a NULL entry indicates the end of the array.
Most commands don't use this field, instead the use optargs, optflags, and the fields in the GLOBALS struct initialized by get_optflags().
unsigned optflags - Command line option flags, set by -get_optflags(). Indicates which of the command line options listed in -toys->which.options occurred this time.
+get_optflags(). Indicates which of the command line options listed in +toys->which.options occurred this time. -The rightmost command line argument listed in toys->which.options sets bit -1, the next one sets bit 2, and so on. This means the bits are set in the same -order the binary digits would be listed if typed out as a string. For example, +
The rightmost command line argument listed in toys->which.options sets bit +1, the next one sets bit 2, and so on. This means the bits are set in the same +order the binary digits would be listed if typed out as a string. For example, the option string "abcd" would parse the command line "-c" to set optflags to 2, "-a" would set optflags to 8, and "-bd" would set optflags to 6 (4|2).
-Only letters are relevant to optflags. In the string "a*b:c#d", d=1, c=2, -b=4, a=8. Punctuation after a letter initializes global variables at the +
Only letters are relevant to optflags. In the string "a*b:c#d", d=1, c=2, +b=4, a=8. Punctuation after a letter initializes global variables at the start of the GLOBALS() block (see union toy_union this for details).
The build infrastructure creates FLAG_ macros for each option letter, -corresponding to the bit position, so you can check (toys.optflags & FLAG_x) +corresponding to the bit position, so you can check (toys.optflags & FLAG_x) to see if a flag was specified. (The correct set of FLAG_ macros is selected by defining FOR_mycommand before #including toys.h. The macros live in toys/globals.h which is generated by scripts/make.sh.)
@@ -422,8 +422,8 @@ toys/globals.h which is generated by scripts/make.sh.)char **optargs - Null terminated array of arguments left over -after get_optflags() removed all the ones it understood. Note: optarg[0] is -the first argument, not the command name. Use toys.which->name for the command +after get_optflags() removed all the ones it understood. Note: optarg[0] is +the first argument, not the command name. Use toys.which->name for the command name.
int optc - Optarg count, equivalent to argc but for optargs[].
Toybox handles this by encapsulating each command's global variables in a structure, and declaring a union of those structures with a single global -instance (called "this"). The GLOBALS() macro contains the global -variables that should go in the current command's global structure. Each +instance (called "this"). The GLOBALS() macro contains the global +variables that should go in the current command's global structure. Each variable can then be accessed as "this.commandname.varname". If you #defined FOR_commandname before including toys.h, the macro TT is #defined to this.commandname so the variable can then be accessed as -"TT.variable". See toys/hello.c for an example.
+"TT.variable". See toys/hello.c for an example.A command that needs global variables should declare a structure to -contain them all, and add that structure to this union. A command should never +contain them all, and add that structure to this union. A command should never declare global variables outside of this, because such global variables would allocate memory when running other commands that don't use those global variables.
The first few fields of this structure can be intialized by get_optargs(), -as specified by the options field off this command's toy_list entry. See +as specified by the options field off this command's toy_list entry. See the get_optargs() description in lib/args.c for details.
@@ -482,17 +482,17 @@ the global toys structure, calling get_optargs() if necessary.void toy_exec(char *argv[]) - Run a built-in command with arguments.
Calls toy_find() on argv[0] (which must be just a command name -without path). Returns if it can't find this command, otherwise calls -toy_init(), toys->which.toy_main(), and exit() instead of returning.
+without path). Returns if it can't find this command, otherwise calls +toy_init(), toys->which.toy_main(), and exit() instead of returning.Use the library function xexec() to fall back to external executables -in $PATH if toy_exec() can't find a built-in command. Note that toy_exec() +in $PATH if toy_exec() can't find a built-in command. Note that toy_exec() does not strip paths before searching for a command, so "./command" will never match an internal command.
void toybox_main(void) - the main function for the multiplexer -command (I.E. "toybox"). Given a command name as its first argument, calls -toy_exec() on its arguments. With no arguments, it lists available commands. +command (I.E. "toybox"). Given a command name as its first argument, calls +toy_exec() on its arguments. With no arguments, it lists available commands. If the first argument starts with "-" it lists each command with its default install path prepended.
Top level configuration file in a stylized variant of -kconfig format. Includes generated/Config.in.
+kconfig format. Includes generated/Config.in.These files are directly used by "make menuconfig" to select which commands to build into toybox (thus generating a .config file), and by @@ -513,7 +513,7 @@ scripts/config2help.py to create generated/help.h.
There is one temporary file in the top level source directory:
.config - Configuration file generated by kconfig, indicating -which commands (and options to commands) are currently enabled. Used +which commands (and options to commands) are currently enabled. Used to make generated/config.h and determine which toys/*/*.c files to build.
You can create a human readable "miniconfig" version of this file using @@ -535,7 +535,7 @@ help.h.
Each command has a configuration entry with an upper case version of the command name. Options to commands start with the command name followed by an underscore and the option name. Global options are attached -to the "toybox" command, and thus use the prefix "TOYBOX_". This organization +to the "toybox" command, and thus use the prefix "TOYBOX_". This organization is used by scripts/cfg2files to select which toys/*/*.c files to compile for a given .config.
@@ -778,7 +778,7 @@ operating systems, etc).
A macro like SWAP_LE32(x) means "The value in x is stored as a little endian 32 bit value, so perform the translation to/from whatever the native -32-bit format is". You do the swap once on the way in, and once on the way +32-bit format is". You do the swap once on the way in, and once on the way out. If your target is already little endian, the macro is a NOP.
The SWAP macros come in BE and LE each with 16, 32, and 64 bit versions. @@ -837,8 +837,8 @@ iterate through a list calling a function on each node.
struct double_list *dlist_add(struct double_list **llist, char *data) - append an entry to a circular linked list. This function allocates a new struct double_list wrapper and returns the -pointer to the new entry (which you can usually ignore since it's llist->prev, -but if llist was NULL you need it). The argument is the ->data field for the +pointer to the new entry (which you can usually ignore since it's llist->prev, +but if llist was NULL you need it). The argument is the ->data field for the new node.
void dlist_add_nomalloc(struct double_list **llist, struct double_list *new) - append existing struct double_list to @@ -858,8 +858,8 @@ a typecast.
Why do the names ->str, ->arg, and ->data differ? - To force -you to keep track of which one you're using, calling free(node->str) would -be bad, and _failing_ to free(node->arg) leaks memory.
Why does llist_pop() take a void * instead of void **? - because the stupid compiler complains about "type punned pointers" when @@ -867,12 +867,12 @@ you typecast and dereference on the same line, due to insane FSF developers hardwiring limitations of their optimizer into gcc's warning system. Since C automatically typecasts any other pointer type to and from void *, the current code works fine. It's sad that it -won't warn you if you forget the &, but the code crashes pretty quickly in +won't warn you if you forget the &, but the code crashes pretty quickly in that case.
How do I assemble a singly-linked-list in order? - use a double_list, dlist_add() your entries, and then call dlist_terminate(list) -to break the circle when done (turning the last ->next and the first ->prev +to break the circle when done (turning the last ->next and the first ->prev into NULLs).
Toybox's option parsing logic is controlled by an "optflags" string, using @@ -927,9 +927,9 @@ concisely described by a large comment at the top of lib/args.c.
other actions the option parsing logic should take.For example, suppose the command line command -b fruit -d walrus -a 42 -is parsed using the optflags string "a#b:c:d". (I.E. -toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d", -"walrus", "-a", "42"]). When get_optflags() returns, the following data is +is parsed using the optflags string "a#b:c:d". (I.E. +toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d", +"walrus", "-a", "42"]). When get_optflags() returns, the following data is available to command_main():
-GLOBALS( - char *c; - char *b; - long a; + char *c; + char *b; + long a; )
That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember, +
That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember, each entry that receives an argument must be a long or pointer, to line up -with the array position. Right to left in the optflags string corresponds to +with the array position. Right to left in the optflags string corresponds to top to bottom in GLOBALS().
Put globals not filled out by the option parsing logic at the end of the @@ -973,25 +973,25 @@ make the ordering explicit, first to last in globals corresponds to right to left in the option string), then leave a blank line before any non-option globals.
-long toys.optflags
+unsigned long long toys.optflags
Each option in the optflags string corresponds to a bit position in -toys.optflags, with the same value as a corresponding binary digit. The -rightmost argument is (1<<0), the next to last is (1<<1) and so on. If -the option isn't encountered while parsing argv[], its bit remains 0.
+toys.optflags, with the same value as a corresponding binary digit. The +rightmost argument is (1<<0), the next to last is (1<<1) and so on. +If the option isn't encountered while parsing argv[], its bit remains 0.Each option -x has a FLAG_x macro for the command letter. Bare --longopts
with no corresponding short option have a FLAG_longopt macro for the long
optionname. Commands enable these macros by #defining FOR_commandname before
-#including
Options disabled in the current configuration (wrapped in a USE_BLAH() macro for a CONFIG_BLAH that's switched off) have their corresponding FLAG macro set to zero, so code checking them ala -if (toys.optargs & FLAG_x) gets optimized out via dead code elimination. +if (toys.optargs & FLAG_x) gets optimized out via dead code elimination. #defining FORCE_FLAGS when switching flag context disables this behavior: the flag is never zero even if the config is disabled. This allows code shared between multiple commands to use the same flag @@ -1002,19 +1002,13 @@ strings.
the optflags string "abcd" would parse the command line argument "-c" to set optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to 6 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8). To check if -c -was encountered, code could test: if (toys.optflags & FLAG_c) printf("yup"); +was encountered, code could test: if (toys.optflags & FLAG_c) printf("yup"); (See the toys/examples directory for more.)Only letters are relevant to optflags, punctuation is skipped: in the string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter usually indicate that the option takes an argument.
-Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is -the amount a long would have on 32-bit platforms anyway; 64 bit code on -32 bit platforms is too expensive to require in common code used by almost -all commands.) Bit positions beyond the 1<<31 aren't recorded, but -parsing higher options can still set global variables.
-Automatically setting global variables from arguments (union this)
The following punctuation characters may be appended to an optflags @@ -1038,14 +1032,14 @@ argument letter, indicating the option takes an additional argument:
Options which have an argument fill in the corresponding slot in the global union "this" (see generated/globals.h), treating it as an array of longs -with the rightmost saved in this[0]. As described above, using "a*b:c#d", +with the rightmost saved in this[0]. As described above, using "a*b:c#d", "-c 42" would set this[0] = 42; and "-b 42" would set this[1] = "42"; each slot is left NULL if the corresponding argument is not encountered.
This behavior is useful because the LP64 standard ensures long and pointer are the same size. C99 guarantees structure members will occur in memory in the same order they're declared, and that padding won't be inserted between -consecutive variables of register size. Thus the first few entries can +consecutive variables of register size. Thus the first few entries can be longs or pointers corresponding to the saved arguments.
The main downside is that numeric arguments ("#" and "-" format) @@ -1070,11 +1064,11 @@ The order of entries is preserved, and as with argv[] this new array is also terminated by a NULL entry.
Option parsing can require a minimum or maximum number of optargs left -over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the +over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the start of the optflags string.
The special argument "--" terminates option parsing, storing all remaining -arguments in optargs. The "--" itself is consumed.
+arguments in optargs. The "--" itself is consumed.Other optflags control characters
@@ -1085,7 +1079,7 @@ optflags string, before any options that would set a bit in toys.optflags:The following may be appended to a float or double:
@@ -1108,13 +1102,13 @@ optflag, but letters are never control characters.)Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT +
Option parsing only understands "<>=" after "." when CFG_TOYBOX_FLOAT is enabled. (Otherwise the code to determine where floating point constants -end drops out. When disabled, it can reserve a global data slot for the +end drops out. When disabled, it can reserve a global data slot for the argument so offsets won't change, but will never fill it out.) You can handle this by using the USE_BLAH() macros with C string concatenation, ala:
-"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"+
"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"
--longopts
@@ -1201,7 +1195,7 @@ in /proc/self/limits is generally 1024.)There are two main ways to use dirtree: 1) assemble a tree of nodes representing a snapshot of directory state and traverse them using the -->next and ->child pointers, or 2) traverse the tree calling a callback +->next and ->child pointers, or 2) traverse the tree calling a callback function on each entry, and freeing its node afterwards. (You can also combine the two, using the callback as a filter to determine which nodes to keep.)
@@ -1220,7 +1214,7 @@ which discards "." and ".." entries and returns DIRTREE_SAVE|DIRTREE_RECURSE for everything else. Used directly, this assembles a snapshot tree of the contents of this directory and its subdirectories to be processed after dirtree_read() returns (by traversing the -struct dirtree's ->next and ->child pointers from the returned root node). +struct dirtree's ->next and ->child pointers from the returned root node).dirtree_path(struct dirtree *node, int *plen) - malloc() a string containing the path from the root of this tree to this node. If @@ -1261,7 +1255,7 @@ DIRTREE_COMEAGAIN on a directory, made after all children have been processed).
Users of this code may put anything they like into the long extra field. For example, "cp" and "mv" use this to store a dirfd for the destination directory (and use DIRTREE_COMEAGAIN to get the second callback so they can -close(node->extra) to avoid running out of filehandles). +close(node->extra) to avoid running out of filehandles). This field is not directly used by the dirtree code, and thanks to LP64 it's large enough to store a typecast pointer to an arbitrary struct.
@@ -1282,13 +1276,13 @@ non-directory entries. The remaining flags only take effect when recursing into the children of a directory.DIRTREE_COMEAGAIN - Call the callback on this node a second time after examining all directory contents, allowing depth-first traversal. -On the second call, dirtree->again is nonzero.
DIRTREE_SYMFOLLOW - follow symlinks when populating children's struct stat st (by feeding a nonzero value to the symfollow argument of dirtree_add_node()), which means DIRTREE_RECURSE treats symlinks to directories as directories. (Avoiding infinite recursion is the callback's -problem: the non-NULL dirtree->symlink can still distinguish between -them. The "find" command follows ->parent up the tree to the root node +problem: the non-NULL dirtree->symlink can still distinguish between +them. The "find" command follows ->parent up the tree to the root node each time, checking to make sure that stat's dev and inode pair don't match any ancestors.)
This directory contains command implementations. Each command is a single -self-contained file. Adding a new command involves adding a single +
This directory contains command implementations. Each command is usually +a single self-contained file. Adding a new command involves adding a single file, and removing a command involves removing that file. Commands use shared infrastructure from the lib/ and generated/ directories.
@@ -1351,10 +1345,23 @@ Note that the commands exist within a single namespace at runtime, so you can't have the same command in multiple subdirectories. (The build tries to fail informatively when you do that.) -There is one more sub-menus in "make menuconfig" containing global +
Each directory has a README file. The first line of that README is +used as the description for the menuconfig menu (via scripts/genconfig.sh +writing generated/Config.in). You can add your own menu to menuconfig +by adding a directory with a README, and placing your own source files +in it.
+ +There is one more sub-menu in "make menuconfig" containing global configuration options for toybox. This menu is defined in the top level Config.in.
+The toys/* directories are mostly equivalent, except that +the "example" and "pending" directories are treated specially +by scripts/genconfig.sh when creating the "make list*" targets, and +scripts/make.sh prints a warning if any commands from "pending" are used. +Nothing in "example" or "pending" having "default y" in its config is merely +a maintainer convention, it is not enforced by the build system.
+See adding a new command for details on the layout of a command file.
@@ -1364,22 +1371,17 @@ layout of a command file.Build infrastructure. The makefile calls scripts/make.sh for "make" and scripts/install.sh for "make install".
-There's also a test suite, "make test" calls make/test.sh, which runs all -the tests in make/test/*. You can run individual tests via +
There's also a test suite, "make tests" calls scripts/test.sh, which by +default runs all the tests/*.test files with the executable bit set (roughly +analogus to "default y" in the command's config: it means all these tests +are expected to pass). You can run individual tests via "scripts/test.sh command", or "TEST_HOST=1 scripts/test.sh command" to run that test against the host implementation instead of the toybox one.
-Run .config through this filter to get a list of enabled commands, which -is turned into a list of files in toys via a sed invocation in the top level -Makefile. -
-Menuconfig infrastructure copied from the Linux kernel a long time ago -(version 2.6.16). See the +(version 2.6.16). See the Linux kernel's Documentation/kbuild/kconfig-language.txt