toybox

About Download Development

Code style

The primary goal of toybox is _simple_ code. Keeping the code small is second, with speed and lots of features coming in somewhere after that. (For more on that, see the design page.)

A simple implementation usually takes up fewer lines of source code, meaning more code can fit on the screen at once, meaning the programmer can see more of it on the screen and thus keep more if in their head at once. This helps code auditing and thus reduces bugs. That said, sometimes being more explicit is preferable to being clever enough to outsmart yourself: don't be so terse your code is unreadable.

Toybox has an actual coding style guide over on the design page, but in general we just want the code to be consistent.

Building Toybox

Toybox is configured using the Kconfig language pioneered by the Linux kernel, and adopted by many other projects (uClibc, OpenEmbedded, etc). This generates a ".config" file containing the selected options, which controls which features are included when compiling toybox.

Each configuration option has a default value. The defaults indicate the "maximum sane configuration", I.E. if the feature defaults to "n" then it either isn't complete or is a special-purpose option (such as debugging code) that isn't intended for general purpose use.

The standard build invocation is:

  • make defconfig #(or menuconfig)
  • make
  • make install

Type "make help" to see all available build options.

The file "configure" contains a number of environment variable definitions which influence the build, such as specifying which compiler to use or where to install the resulting binaries. This file is included by the build, but accepts existing definitions of the environment variables, so it may be sourced or modified by the developer before building and the definitions exported to the environment will take precedence.

(To clarify: "configure" describes the build and installation environment, ".config" lists the features selected by defconfig/menuconfig.)

Running a command

main

The toybox main() function is at the end of main.c at the top level. It has two possible codepaths, only one of which is configured into any given build of toybox.

If CONFIG_SINGLE is selected, toybox is configured to contain only a single command, so most of the normal setup can be skipped. In this case the multiplexer isn't used, instead main() calls toy_singleinit() (also in main.c) to set up global state and parse command line arguments, calls the command's main function out of toy_list (in the CONFIG_SINGLE case the array has a single entry, no need to search), and if the function returns instead of exiting it flushes stdout (detecting error) and returns toys.exitval.

When CONFIG_SINGLE is not selected, main() uses basename() to find the name it was run as, shifts its argument list one to the right so it lines up with where the multiplexer function expects it, and calls toybox_main(). This leverages the multiplexer command's infrastructure to find and run the appropriate command. (A command name starting with "toybox" will recursively call toybox_main(); you can go "./toybox toybox toybox toybox ls" if you want to...)

toybox_main

The toybox_main() function is also in main,c. It handles a possible --help option ("toybox --help ls"), prints the list of available commands if no arguments were provided to the multiplexer (or with full path names if any other option is provided before a command name, ala "toybox --list"). Otherwise it calls toy_exec() on its argument list.

Note that the multiplexer is the first entry in toy_list (the rest of the list is sorted alphabetically to allow binary search), so toybox_main can cheat and just grab the first entry to quickly set up its context without searching. Since all command names go through the multiplexer at least once in the non-TOYBOX_SINGLE case, this avoids a redundant search of the list.

The toy_exec() function is also in main.c. It performs toy_find() to perform a binary search on the toy_list array to look up the command's entry by name and saves it in the global variable which, calls toy_init() to parse command line arguments and set up global state (using which->options), and calls the appropriate command's main() function (which->toy_main). On return it flushes all pending ansi FILE * I/O, detects if stdout had an error, and then calls xexit() (which uses toys.exitval).

Infrastructure

The toybox source code is in following directories:

  • The top level directory contains the file main.c (were execution starts), the header file toys.h (included by every command), and other global infrastructure.
  • The lib directory contains common functions shared by multiple commands:
  • The toys directory contains the C files implementating each command. Currently it contains five subdirectories categorizing the commands: posix, lsb, other, example, and pending.
  • The scripts directory contains the build and test infrastructure.
  • The kconfig directory contains the configuration infrastructure implementing menuconfig (copied from the Linux kernel).
  • The generated directory contains intermediate files generated from other parts of the source code.

Adding a new command

To add a new command to toybox, add a C file implementing that command under the toys directory. No other files need to be modified; the build extracts all the information it needs (such as command line arguments) from specially formatted comments and macros in the C file. (See the description of the "generated" directory for details.)

Currently there are five subdirectories under "toys", one for commands defined by the POSIX standard, one for commands defined by the Linux Standard Base, an "other" directory for commands not covered by an obvious standard, a directory of example commands (templates to use when starting new commands), and a "pending" directory of commands that need further review/cleanup before moving to one of the other directories (run these at your own risk, cleanup patches welcome). These directories are just for developer convenience sorting the commands, the directories are otherwise functionally identical. To add a new category, create the appropriate directory with a README file in it whose first line is the description menuconfig should use for the directory.)

An easy way to start a new command is copy the file "toys/example/hello.c" to the name of the new command, and modify this copy to implement the new command (more or less by turning every instance of "hello" into the name of your command, updating the command line arguments, globals, and help data, and then filling out its "main" function with code that does something interesting).

You could also start with "toys/example/skeleton.c", which provides a lot more example code (showing several variants of command line option parsing, how to implement multiple commands in the same file, and so on). But usually it's just more stuff to delete.

Here's a checklist of steps to turn hello.c into another command:

  • First "cp toys/example/hello.c toys/other/yourcommand.c" and open the new file in your preferred text editor.

    • Note that the name of the new file is significant: it's the name of the new command you're adding to toybox. The build includes all *.c files under toys/*/ whose names are a case insensitive match for an enabled config symbol. So toys/posix/cat.c only gets included if you have "CAT=y" in ".config".

  • Change the one line comment at the top of the file (currently "hello.c - A hello world program") to describe your new file.

  • Change the copyright notice to your name, email, and the current year.

  • Give a URL to the relevant standards document, where applicable. (Sample links to SUSv4 and LSB are provided, feel free to link to other documentation or standards as appropriate.)

  • Update the USE_YOURCOMMAND(NEWTOY(yourcommand,"blah",0)) line. The NEWTOY macro fills out this command's toy_list structure. The arguments to the NEWTOY macro are:

    1. the name used to run your command

    2. the command line argument option parsing string (0 if none)

    3. a bitfield of TOYFLAG values (defined in toys.h) providing additional information such as where your command should be installed on a running system, whether to blank umask before running, whether or not the command must run as root (and thus should retain root access if installed SUID), and so on.

  • Change the kconfig data (from "config YOURCOMMAND" to the end of the comment block) to supply your command's configuration and help information. The uppper case config symbols are used by menuconfig, and are also what the CFG_ and USE_() macros are generated from (see [TODO]). The help information here is used by menuconfig, and also by the "help" command to describe your new command. (See [TODO] for details.) By convention, unfinished commands default to "n" and finished commands default to "y", so "make defconfig" selects all finished commands. (Note, "finished" means "ready to be used", not that it'll never change again.)

    Each help block should start with a "usage: yourcommand" line explaining any command line arguments added by this config option. The "help" command outputs this text, and scripts/config2help.c in the build infrastructure collates these usage lines for commands with multiple configuration options when producing generated/help.h.

  • Change the "#define FOR_hello" line to "#define FOR_yourcommand" right before the "#include ". (This selects the appropriate FLAG_ macros and does a "#define TT this.yourcommand" so you can access the global variables out of the space-saving union of structures. If you aren't using any command flag bits and aren't defining a GLOBAL block, you can delete this line.)

  • Update the GLOBALS() macro to contain your command's global variables. If your command has no global variables, delete this macro.

    Variables in the GLOBALS() block are are stored in a space saving union of structures format, which may be accessed using the TT macro as if TT were a global structure (so TT.membername). If you specified two-character command line arguments in NEWTOY(), the first few global variables will be initialized by the automatic argument parsing logic, and the type and order of these variables must correspond to the arguments specified in NEWTOY(). (See lib/args.c for details.)

  • Rename hello_main() to yourcommand_main(). This is the main() function where execution of your command starts. Your command line options are already sorted into this.optflags, this.optargs, this.optc, and the GLOBALS() as appropriate by the time this function is called. (See get_optflags() for details.)

Headers.

Commands generally don't have their own headers. If it's common code it can live in lib/, if it isn't put it in the command's .c file. (The line between implementing multiple commands in a C file via OLDTOY() to share infrastructure and moving that shared infrastructure to lib/ is a judgement call. Try to figure out which is simplest.)

The top level toys.h should #include all the standard (posix) headers that any command uses. (Partly this is friendly to ccache and partly this makes the command implementations shorter.) Individual commands should only need to include nonstandard headers that might prevent that command from building in some context we'd care about (and thus requiring that command to be disabled to avoid a build break).

Target-specific stuff (differences between compiler versions, libc versions, or operating systems) should be confined to lib/portability.h and lib/portability.c. (There's even some minimal compile-time environment probing that writes data to generated/portability.h, see scripts/genconfig.sh.)

Only include linux/*.h headers from individual commands (not from other headers), and only if you really need to. Data that varies per architecture is a good reason to include a header. If you just need a couple constants that haven't changed since the 1990's, it's ok to #define them yourself or just use the constant inline with a comment explaining what it is. (A #define that's only used once isn't really helping.)

Top level directory.

This directory contains global infrastructure.

toys.h

Each command #includes "toys.h" as part of its standard prolog. It may "#define FOR_commandname" before doing so to get some extra entries specific to this command.

This file sucks in most of the commonly used standard #includes, so individual files can just #include "toys.h" and not have to worry about stdargs.h and so on. Individual commands still need to #include special-purpose headers that may not be present on all systems (and thus would prevent toybox from building that command on such a system with that command enabled). Examples include regex support, any "linux/" or "asm/" headers, mtab support (mntent.h and sys/mount.h), and so on.

The toys.h header also defines structures for most of the global variables provided to each command by toybox_main(). These are described in detail in the description for main.c, where they are initialized.

The global variables are grouped into structures (and a union) for space savings, to more easily track the amount of memory consumed by them, so that they may be automatically cleared/initialized as needed, and so that access to global variables is more easily distinguished from access to local variables.

main.c

Contains the main() function where execution starts, plus common infrastructure to initialize global variables and select which command to run. The "toybox" multiplexer command also lives here. (This is the only command defined outside of the toys directory.)

Execution starts in main() which trims any path off of the first command name and calls toybox_main(), which calls toy_exec(), which calls toy_find() and toy_init() before calling the appropriate command's function from toy_list[] (via toys.which->toy_main()). If the command is "toybox", execution recurses into toybox_main(), otherwise the call goes to the appropriate commandname_main() from a C file in the toys directory.

The following global variables are defined in main.c:

The following functions are defined in main.c:

  • struct toy_list *toy_find(char *name) - Return the toy_list structure for this command name, or NULL if not found.

  • void toy_init(struct toy_list *which, char *argv[]) - fill out the global toys structure, calling get_optargs() if necessary.

  • void toy_exec(char *argv[]) - Run a built-in command with arguments.

    Calls toy_find() on argv[0] (which must be just a command name without path). Returns if it can't find this command, otherwise calls toy_init(), toys->which.toy_main(), and exit() instead of returning.

    Use the library function xexec() to fall back to external executables in $PATH if toy_exec() can't find a built-in command. Note that toy_exec() does not strip paths before searching for a command, so "./command" will never match an internal command.

  • void toybox_main(void) - the main function for the multiplexer command (I.E. "toybox"). Given a command name as its first argument, calls toy_exec() on its arguments. With no arguments, it lists available commands. If the first argument starts with "-" it lists each command with its default install path prepended.

Config.in

Top level configuration file in a stylized variant of kconfig format. Includes generated/Config.in.

These files are directly used by "make menuconfig" to select which commands to build into toybox (thus generating a .config file), and by scripts/config2help.py to create generated/help.h.

Temporary files:

There is one temporary file in the top level source directory:

  • .config - Configuration file generated by kconfig, indicating which commands (and options to commands) are currently enabled. Used to make generated/config.h and determine which toys/*/*.c files to build.

    You can create a human readable "miniconfig" version of this file using these instructions.

Directory generated/

The remaining temporary files live in the "generated/" directory, which is for files generated at build time from other source files.

  • generated/Config.in - Included from the top level Config.in, contains one or more configuration entries for each command.

    Each command has a configuration entry with an upper case version of the command name. Options to commands start with the command name followed by an underscore and the option name. Global options are attached to the "toybox" command, and thus use the prefix "TOYBOX_". This organization is used by scripts/cfg2files to select which toys/*/*.c files to compile for a given .config.

    A command with multiple names (or multiple similar commands implemented in the same .c file) should have config symbols prefixed with the name of their C file. I.E. config symbol prefixes are NEWTOY() names. If OLDTOY() names have config symbols they must be options (symbols with an underscore and suffix) to the NEWTOY() name. (See generated/toylist.h)

  • generated/config.h - list of CFG_SYMBOL and USE_SYMBOL() macros, generated from .config by a sed invocation in the top level Makefile.

    CFG_SYMBOL is a comple time constant set to 1 for enabled symbols and 0 for disabled symbols. This allows the use of normal if() statements to remove code at compile time via the optimizer's dead code elimination (which removes from the binary any code that cannot be reached). This saves space without cluttering the code with #ifdefs or leading to configuration dependent build breaks. (See the 1992 Usenix paper #ifdef Considered Harmful for more information.)

    USE_SYMBOL(code) evaluates to the code in parentheses when the symbol is enabled, and nothing when the symbol is disabled. This can be used for things like varargs or variable declarations which can't always be eliminated by a simple test on CFG_SYMBOL. Note that (unlike CFG_SYMBOL) this is really just a variant of #ifdef, and can still result in configuration dependent build breaks. Use with caution.

  • generated/flags.h - FLAG_? macros indicating which command line options were seen. The option parsing in lib/args.c sets bits in toys.optflags, which can be tested by anding with the appropriate FLAG_ macro. (Bare longopts, which have no corresponding short option, will have the longopt name after FLAG_. All others use the single letter short option.)

    To get the appropriate macros for your command, #define FOR_commandname before #including toys.h. To switch macro sets (because you have an OLDTOY() with different options in the same .c file), #define CLEANUP_oldcommand and also #define FOR_newcommand, then #include "generated/flags.h" to switch.

  • generated/globals.h - Declares structures to hold the contents of each command's GLOBALS(), and combines them into "global_union this". (Yes, the name was chosen to piss off C++ developers who think that C is merely a subset of C++, not a language in its own right.)

    The union reuses the same memory for each command's global struct: since only one command's globals are in use at any given time, collapsing them together saves space. The headers #define TT to the appropriate "this.commandname", so you can refer to the current command's global variables out of "this" as TT.variablename.

    The globals start zeroed, and the first few are filled out by the lib/args.c argument parsing code called from main.c.

  • toys/help.h - #defines two help text strings for each command: a single line command_help and an additinal command_help_long. This is used by help_main() in toys/help.c to display help for commands.

    This file is created by scripts/make.sh, which compiles scripts/config2help.c into the binary generated/config2help, and then runs it against the top level .config and Config.in files to extract the help text from each config entry and collate together dependent options.

    This file contains help text for all commands, regardless of current configuration, but only the ones currently enabled in the .config file wind up in the help_data[] array, and only the enabled dependent options have their help text added to the command they depend on.

  • generated/newtoys.h - All the NEWTOY() and OLDTOY() macros in alphabetical order, each of which should be inside the appropriate USE_ macro. (Ok, not _quite_ alphabetical orer: the "toybox" multiplexer is always the first entry.)

    By #definining NEWTOY() to various things before #including this file, it may be used to create function prototypes (in toys.h), initialize the toy_list array (in main.c, the alphabetical order lets toy_find() do a binary search), initialize the help_data array (in lib/help.c), and so on. (It's even used to initialize the NEED_OPTIONS macro, which is has a 1 or 0 for each command using command line option parsing, ORed together. This allows compile-time dead code elimination to remove the whole of lib/args.c if nothing currently enabled is using it.)

    Each NEWTOY and OLDTOY macro contains the command name, command line option string (telling lib/args.c how to parse command line options for this command), recommended install location, and miscelaneous data such as whether this command should retain root permissions if installed suid.

  • toys/oldtoys.h - Macros with the command line option parsing string for each NEWTOY. This allows an OLDTOY that's just an alias for an existing command to refer to the existing option string instead of having to repeat it.

Directory lib/

TODO: document lots more here.

lib: getmountlist(), error_msg/error_exit, xmalloc(), strlcpy(), xexec(), xopen()/xread(), xgetcwd(), xabspath(), find_in_path(), itoa().

lib/portability.h

This file is automatically included from the top of toys.h, and smooths over differences between platforms (hardware targets, compilers, C libraries, operating systems, etc).

This file provides SWAP macros (SWAP_BE16(x) and SWAP_LE32(x) and so on).

A macro like SWAP_LE32(x) means "The value in x is stored as a little endian 32 bit value, so perform the translation to/from whatever the native 32-bit format is". You do the swap once on the way in, and once on the way out. If your target is already little endian, the macro is a NOP.

The SWAP macros come in BE and LE each with 16, 32, and 64 bit versions. In each case, the name of the macro refers to the _external_ representation, and converts to/from whatever your native representation happens to be (which can vary depending on what you're currently compiling for).

lib/llist.c

Some generic single and doubly linked list functions, which take advantage of a couple properties of C:

  • Structure elements are laid out in memory in the order listed, and the first element has no padding. This means you can always treat (typecast) a pointer to a structure as a pointer to the first element of the structure, even if you don't know anything about the data following it.

  • An array of length zero at the end of a structure adds no space to the sizeof() the structure, but if you calculate how much extra space you want when you malloc() the structure it will be available at the end. Since C has no bounds checking, this means each struct can have one variable length array.

Toybox's list structures always have their next pointer as the first entry of each struct, and singly linked lists end with a NULL pointer. This allows generic code to traverse such lists without knowing anything else about the specific structs composing them: if your pointer isn't NULL typecast it to void ** and dereference once to get the next entry.

lib/lib.h defines three structure types:

  • struct string_list - stores a single string (char str[0]), memory for which is allocated as part of the node. (I.E. llist_traverse(list, free); can clean up after this type of list.)

  • struct arg_list - stores a pointer to a single string (char *arg) which is stored in a separate chunk of memory.

  • struct double_list - has a second pointer (struct double_list *prev along with a char *data for payload.

List Functions
  • void *llist_pop(void **list) - advances through a list ala node = llist_pop(&list); This doesn't modify the list contents, but does advance the pointer you feed it (which is why you pass the _address_ of that pointer, not the pointer itself).

  • void llist_traverse(void *list, void (*using)(void *data)) - iterate through a list calling a function on each node.

  • struct double_list *dlist_add(struct double_list **llist, char *data) - append an entry to a circular linked list. This function allocates a new struct double_list wrapper and returns the pointer to the new entry (which you can usually ignore since it's llist->prev, but if llist was NULL you need it). The argument is the ->data field for the new node.

    • void dlist_add_nomalloc(struct double_list **llist, struct double_list *new) - append existing struct double_list to list, does not allocate anything.

List code trivia questions:
  • Why do arg_list and double_list contain a char * payload instead of a void *? - Because you always have to typecast a void * to use it, and typecasting a char * does no harm. Since strings are the most common payload, and doing math on the pointer ala "(type *)(ptr+sizeof(thing)+sizeof(otherthing))" requires ptr to be char * anyway (at least according to the C standard), defaulting to char * saves a typecast.

  • Why do the names ->str, ->arg, and ->data differ? - To force you to keep track of which one you're using, calling free(node->str) would be bad, and _failing_ to free(node->arg) leaks memory.

  • Why does llist_pop() take a void * instead of void **? - because the stupid compiler complains about "type punned pointers" when you typecast and dereference on the same line, due to insane FSF developers hardwiring limitations of their optimizer into gcc's warning system. Since C automatically typecasts any other pointer type to and from void *, the current code works fine. It's sad that it won't warn you if you forget the &, but the code crashes pretty quickly in that case.

  • How do I assemble a singly-linked-list in order? - use a double_list, dlist_add() your entries, and then break the circle with list->prev->next = NULL; when done.

lib/args.c

Toybox's main.c automatically parses command line options before calling the command's main function. Option parsing starts in get_optflags(), which stores results in the global structures "toys" (optflags and optargs) and "this".

The option parsing infrastructure stores a bitfield in toys.optflags to indicate which options the current command line contained. Arguments attached to those options are saved into the command's global structure ("this"). Any remaining command line arguments are collected together into the null-terminated array toys.optargs, with the length in toys.optc. (Note that toys.optargs does not contain the current command name at position zero, use "toys.which->name" for that.) The raw command line arguments get_optflags() parsed are retained unmodified in toys.argv[].

Toybox's option parsing logic is controlled by an "optflags" string, using a format reminiscent of getopt's optargs but has several important differences. Toybox does not use the getopt() function out of the C library, get_optflags() is an independent implementation which doesn't permute the original arguments (and thus doesn't change how the command is displayed in ps and top), and has many features not present in libc optargs() (such as the ability to describe long options in the same string as normal options).

Each command's NEWTOY() macro has an optflags string as its middle argument, which sets toy_list.options for that command to tell get_optflags() what command line arguments to look for, and what to do with them. If a command has no option definition string (I.E. the argument is NULL), option parsing is skipped for that command, which must look at the raw data in toys.argv to parse its own arguments. (If no currently enabled command uses option parsing, get_optflags() is optimized out of the resulting binary by the compiler's --gc-sections option.)

You don't have to free the option strings, which point into the environment space (I.E. the string data is not copied). A TOYFLAG_NOFORK command that uses the linked list type "*" should free the list objects but not the data they point to, via "llist_free(TT.mylist, NULL);". (If it's not NOFORK, exit() will free all the malloced data anyway unless you want to implement a CONFIG_TOYBOX_FREE cleanup for it.)

Optflags format string

Note: the optflags option description string format is much more concisely described by a large comment at the top of lib/args.c.

The general theory is that letters set optflags, and punctuation describes other actions the option parsing logic should take.

For example, suppose the command line command -b fruit -d walrus -a 42 is parsed using the optflags string "a#b:c:d". (I.E. toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d", "walrus", "-a", "42"]). When get_optflags() returns, the following data is available to command_main():

  • In struct toys:

    • toys.optflags = 13; // -a = 8 | -b = 4 | -d = 1
    • toys.optargs[0] = "walrus"; // leftover argument
    • toys.optargs[1] = NULL; // end of list
    • toys.optc = 1; // there was 1 leftover argument
    • toys.argv[] = {"-b", "fruit", "-d", "walrus", "-a", "42"}; // The original command line arguments

  • In union this (treated as long this[]):

    • this[0] = NULL; // -c didn't get an argument this time, so get_optflags() didn't change it and toys_init() zeroed "this" during setup.)
    • this[1] = (long)"fruit"; // argument to -b
    • this[2] = 42; // argument to -a

If the command's globals are:

GLOBALS(
	char *c;
	char *b;
	long a;
)

That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember, each entry that receives an argument must be a long or pointer, to line up with the array position. Right to left in the optflags string corresponds to top to bottom in GLOBALS().

Put globals not filled out by the option parsing logic at the end of the GLOBALS block. Common practice is to list the options one per line (to make the ordering explicit, first to last in globals corresponds to right to left in the option string), then leave a blank line before any non-option globals.

long toys.optflags

Each option in the optflags string corresponds to a bit position in toys.optflags, with the same value as a corresponding binary digit. The rightmost argument is (1<<0), the next to last is (1<<1) and so on. If the option isn't encountered while parsing argv[], its bit remains 0.

For example, the optflags string "abcd" would parse the command line argument "-c" to set optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to 6 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8).

Only letters are relevant to optflags, punctuation is skipped: in the string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter usually indicate that the option takes an argument.

Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is the amount a long would have on 32-bit platforms anyway; 64 bit code on 32 bit platforms is too expensive to require in common code used by almost all commands.) Bit positions beyond the 1<<31 aren't recorded, but parsing higher options can still set global variables.

Automatically setting global variables from arguments (union this)

The following punctuation characters may be appended to an optflags argument letter, indicating the option takes an additional argument:

  • : - plus a string argument, keep most recent if more than one.
  • * - plus a string argument, appended to a linked list.
  • @ - plus an occurrence counter (stored in a long)
  • # - plus a signed long argument.
  • - - plus a signed long argument defaulting to negative (start argument with + to force a positive value).
  • . - plus a floating point argument (if CFG_TOYBOX_FLOAT).
    • The following can be appended to a float or double:
    • <123 - error if argument is less than this
    • >123 - error if argument is greater than this
    • =123 - default value if argument not supplied

A note about "." and CFG_TOYBOX_FLOAT: option parsing only understands <>= after . when CFG_TOYBOX_FLOAT is enabled. (Otherwise the code to determine where floating point constants end drops out; it requires floating point). When disabled, it can reserve a global data slot for the argument (so offsets won't change in your GLOBALS[] block), but will never fill it out. You can handle this by using the USE_BLAH() macros with C string concatenation, ala: "abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"

GLOBALS

Options which have an argument fill in the corresponding slot in the global union "this" (see generated/globals.h), treating it as an array of longs with the rightmost saved in this[0]. As described above, using "a*b:c#d", "-c 42" would set this[0] = 42; and "-b 42" would set this[1] = "42"; each slot is left NULL if the corresponding argument is not encountered.

This behavior is useful because the LP64 standard ensures long and pointer are the same size. C99 guarantees structure members will occur in memory in the same order they're declared, and that padding won't be inserted between consecutive variables of register size. Thus the first few entries can be longs or pointers corresponding to the saved arguments.

See toys/other/hello.c for a longer example of parsing options into the GLOBALS block.

char *toys.optargs[]

Command line arguments in argv[] which are not consumed by option parsing (I.E. not recognized either as -flags or arguments to -flags) will be copied to toys.optargs[], with the length of that array in toys.optc. (When toys.optc is 0, no unrecognized command line arguments remain.) The order of entries is preserved, and as with argv[] this new array is also terminated by a NULL entry.

Option parsing can require a minimum or maximum number of optargs left over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the start of the optflags string.

The special argument "--" terminates option parsing, storing all remaining arguments in optargs. The "--" itself is consumed.

Other optflags control characters

The following characters may occur at the start of each command's optflags string, before any options that would set a bit in toys.optflags:

  • ^ - stop at first nonoption argument (for nice, xargs...)
  • ? - allow unknown arguments (pass non-option arguments starting with - through to optargs instead of erroring out).
  • & - the first argument has imaginary dash (ala tar/ps. If given twice, all arguments have imaginary dash.)
  • < - must be followed by a decimal digit indicating at least this many leftover arguments are needed in optargs (default 0)
  • > - must be followed by a decimal digit indicating at most this many leftover arguments allowed (default MAX_INT)

The following characters may be appended to an option character, but do not by themselves indicate an extra argument should be saved in this[]. (Technically any character not recognized as a control character sets an optflag, but letters are never control characters.)

  • ^ - stop parsing options after encountering this option, everything else goes into optargs.
  • | - this option is required. If more than one marked, only one is required.

The following may be appended to a float or double:

  • <123 - error if argument is less than this
  • >123 - error if argument is greater than this
  • =123 - default value if argument not supplied

Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT is enabled. (Otherwise the code to determine where floating point constants end drops out. When disabled, it can reserve a global data slot for the argument so offsets won't change, but will never fill it out.). You can handle this by using the USE_BLAH() macros with C string concatenation, ala:

"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"

--longopts

The optflags string can contain long options, which are enclosed in parentheses. They may be appended to an existing option character, in which case the --longopt is a synonym for that option, ala "a:(--fred)" which understands "-a blah" or "--fred blah" as synonyms.

Longopts may also appear before any other options in the optflags string, in which case they have no corresponding short argument, but instead set their own bit based on position. So for "(walrus)#(blah)xy:z" "command --walrus 42" would set toys.optflags = 16 (-z = 1, -y = 2, -x = 4, --blah = 8) and would assign this[1] = 42;

A short option may have multiple longopt synonyms, "a(one)(two)", but each "bare longopt" (ala "(one)(two)abc" before any option characters) always sets its own bit (although you can group them with +X).

[groups]

At the end of the option string, square bracket groups can define relationships between existing options. (This only applies to short options, bare --longopts can't participate.)

The first character of the group defines the type, the remaining characters are options it applies to:

  • - - Exclusive, switch off all others in this group.
  • + - Inclusive, switch on all others in this group.
  • ! - Error, fail if more than one defined.

So "abc[-abc]" means -ab = -b, -ba = -a, -abc = -c. "abc[+abc]" means -ab=-abc, -c=-abc, and "abc[!abc] means -ab calls error_exit("no -b with -a"). Note that [-] groups clear the GLOBALS option slot of options they're switching back off, but [+] won't set options it didn't see (just the optflags).

whitespace

Arguments may occur with or without a space (I.E. "-a 42" or "-a42"). The command line argument "-abc" may be interepreted many different ways: the optflags string "cba" sets toys.optflags = 7, "c:ba" sets toys.optflags=4 and saves "ba" as the argument to -c, and "cb:a" sets optflags to 6 and saves "c" as the argument to -b.

Note that & changes whitespace handling, so that the command line "tar cvfCj outfile.tar.bz2 topdir filename" is parsed the same as "tar filename -c -v -j -f outfile.tar.bz2 -C topdir". Note that "tar -cvfCj one two three" would equal "tar -c -v -f Cj one two three". (This matches historical usage.)

Appending a space to the option in the option string ("a: b") makes it require a space, I.E. "-ab" is interpreted as "-a" "-b". That way "kill -stop" differs from "kill -s top".

Appending ; to a longopt in the option string makes its argument optional, and only settable with =, so in ls "(color):;" can accept "ls --color" and "ls --color=auto" without complaining that the first has no argument.

lib/dirtree.c

The directory tree traversal code should be sufficiently generic that commands never need to use readdir(), scandir(), or the fts.h family of functions.

These functions do not call chdir() or rely on PATH_MAX. Instead they use openat() and friends, using one filehandle per directory level to recurseinto subdirectories. (I.E. they can descend 1000 directories deep if setrlimit(RLIMIT_NOFILE) allows enough open filehandles, and the default in /proc/self/limits is generally 1024.)

The basic dirtree functions are:

  • dirtree_read(char *path, int (*callback)(struct dirtree node)) - recursively read directories, either applying callback() or returning a tree of struct dirtree if callback is NULL.

  • dirtree_path(struct dirtree *node, int *plen) - malloc() a string containing the path from the root of this tree to this node. If plen isn't NULL then *plen is how many extra bytes to malloc at the end of string.

  • dirtree_parentfd(struct dirtree *node) - return fd of containing directory, for use with openat() and such.

The dirtree_read() function takes two arguments, a starting path for the root of the tree, and a callback function. The callback takes a struct dirtree * (from lib/lib.h) as its argument. If the callback is NULL, the traversal uses a default callback (dirtree_notdotdot()) which recursively assembles a tree of struct dirtree nodes for all files under this directory and subdirectories (filtering out "." and ".." entries), after which dirtree_read() returns the pointer to the root node of this snapshot tree.

Otherwise the callback() is called on each entry in the directory, with struct dirtree * as its argument. This includes the initial node created by dirtree_read() at the top of the tree.

struct dirtree

Each struct dirtree node contains char name[] and struct stat st entries describing a file, plus a char *symlink which is NULL for non-symlinks.

During a callback function, the int data field of directory nodes contains a dirfd (for use with the openat() family of functions). This is generally used by calling dirtree_parentfd() on the callback's node argument. For symlinks, data contains the length of the symlink string. On the second callback from DIRTREE_COMEAGAIN (depth-first traversal) data = -1 for all nodes (that's how you can tell it's the second callback).

Users of this code may put anything they like into the long extra field. For example, "cp" and "mv" use this to store a dirfd for the destination directory (and use DIRTREE_COMEAGAIN to get the second callback so they can close(node->extra) to avoid running out of filehandles). This field is not directly used by the dirtree code, and thanks to LP64 it's large enough to store a typecast pointer to an arbitrary struct.

The return value of the callback combines flags (with boolean or) to tell the traversal infrastructure how to behave:

  • DIRTREE_SAVE - Save this node, assembling a tree. (Without this the struct dirtree is freed after the callback returns. Filtering out siblings is fine, but discarding a parent while keeping its child leaks memory.)

  • DIRTREE_ABORT - Do not examine any more entries in this directory. (Does not propagate up tree: to abort entire traversal, return DIRTREE_ABORT from parent callbacks too.)

  • DIRTREE_RECURSE - Examine directory contents. Ignored for non-directory entries. The remaining flags only take effect when recursing into the children of a directory.

  • DIRTREE_COMEAGAIN - Call the callback a second time after examining all directory contents, allowing depth-first traversal. On the second call, dirtree->data = -1.

  • DIRTREE_SYMFOLLOW - follow symlinks when populating children's struct stat st (by feeding a nonzero value to the symfollow argument of dirtree_add_node()), which means DIRTREE_RECURSE treats symlinks to directories as directories. (Avoiding infinite recursion is the callback's problem: the non-NULL dirtree->symlink can still distinguish between them.)

Each struct dirtree contains three pointers (next, parent, and child) to other struct dirtree.

The parent pointer indicates the directory containing this entry; even when not assembling a persistent tree of nodes the parent entries remain live up to the root of the tree while child nodes are active. At the top of the tree the parent pointer is NULL, meaning the node's name[] is either an absolute path or relative to cwd. The function dirtree_parentfd() gets the directory file descriptor for use with openat() and friends, returning AT_FDCWD at the top of tree.

The child pointer points to the first node of the list of contents of this directory. If the directory contains no files, or the entry isn't a directory, child is NULL.

The next pointer indicates sibling nodes in the same directory as this node, and since it's the first entry in the struct the llist.c traversal mechanisms work to iterate over sibling nodes. Each dirtree node is a single malloc() (even char *symlink points to memory at the end of the node), so llist_free() works but its callback must descend into child nodes (freeing a tree, not just a linked list), plus whatever the user stored in extra.

The dirtree_read() function is a simple wrapper, calling dirtree_add_node() to create a root node relative to the current directory, then calling handle_callback() on that node (which recurses as instructed by the callback return flags). Some commands (such as chgrp) bypass this wrapper, for example to control whether or not to follow symlinks to the root node; symlinks listed on the command line are often treated differently than symlinks encountered during recursive directory traversal).

The ls command not only bypasses the wrapper, but never returns DIRTREE_RECURSE from the callback, instead calling dirtree_recurse() manually from elsewhere in the program. This gives ls -lR manual control of traversal order, which is neither depth first nor breadth first but instead a sort of FIFO order requried by the ls standard.

Directory toys/

This directory contains command implementations. Each command is a single self-contained file. Adding a new command involves adding a single file, and removing a command involves removing that file. Commands use shared infrastructure from the lib/ and generated/ directories.

Currently there are three subdirectories under "toys/" containing commands described in POSIX-2008, the Linux Standard Base 4.1, or "other". The only difference this makes is which menu the command shows up in during "make menuconfig", the directories are otherwise identical. Note that they commands exist within a single namespace at runtime, so you can't have the same command in multiple subdirectories.

(There are actually four sub-menus in "make menuconfig", the fourth contains global configuration options for toybox, and lives in Config.in at the top level.)

See adding a new command for details on the layout of a command file.

Directory scripts/

Build infrastructure. The makefile calls scripts/make.sh for "make" and scripts/install.sh for "make install".

There's also a test suite, "make test" calls make/test.sh, which runs all the tests in make/test/*. You can run individual tests via "scripts/test.sh command", or "TEST_HOST=1 scripts/test.sh command" to run that test against the host implementation instead of the toybox one.

scripts/cfg2files.sh

Run .config through this filter to get a list of enabled commands, which is turned into a list of files in toys via a sed invocation in the top level Makefile.

Directory kconfig/

Menuconfig infrastructure copied from the Linux kernel. See the Linux kernel's Documentation/kbuild/kconfig-language.txt

Directory generated/

All the files in this directory except the README are generated by the build. (See scripts/make.sh)

Everything in this directory is a derivative file produced from something else. The entire directory is deleted by "make distclean".