Toybox source is formatted to be read with 4-space tab stops. Each file starts with a special comment telling vi to set the tab stop to 4. Note that one of the bugs in Ubuntu 7.10 broke vi's ability to parse these comments; you must either rebuild vim from source, or go ":ts=4" yourself each time you load the file.
Gotos are allowed for error handling, and for breaking out of nested loops. In general, a goto should only jump forward (not back), and should either jump to the end of an outer loop, or to error handling code at the end of the function. Goto labels are never indented: they override the block structure of the file. Putting them at the left edge makes them easy to spot as overrides to the normal flow of control, which they are.
The primary goal of toybox is _simple_ code. Small is second, speed and lots of features come in somewhere after that. Note that environmental dependencies are a type of complexity, so needing other packages to build or run is a big downside. For example, don't use curses when you can output ansi escape sequences instead.
The toybox source code is in following directories:
To add a new command to toybox, add a C file implementing that command to the toys directory. No other files need to be modified; the build extracts all the information it needs (such as command line arguments) from specially formatted comments and macros in the C file. (See the description of the generated directory for details.)
An easy way to start a new command is copy the file "hello.c" to the name of the new command, and modify this copy to implement the new command. This file is an example command meant to be used as a "skeleton" for new commands (more or less by turning every instance of "hello" into the name of your command, updating the command line arguments, globals, and help data, and then filling out its "main" function with code that does something interesting). It provides examples of all the build infrastructure (including optional elements like command line argument parsing and global variables that a "hello world" program doesn't strictly need).
Here's a checklist of steps to turn hello.c into another command:
First "cd toys" and "cp hello.c yourcommand.c". Note that the name of this file is significant, it's the name of the new command you're adding to toybox. Open your new file in your favorite editor.
Change the one line comment at the top of the file (currently "hello.c - A hello world program") to describe your new file.
Change the copyright notice to your name, email, and the current year.
Give a URL to the relevant standards document, or say "Not in SUSv4" if there is no relevant standard. (Currently both lines are there, delete whichever is inappropriate.) The existing link goes to the directory of SUSv4 command line utility standards on the Open Group's website, where there's often a relevant commandname.html file. Feel free to link to other documentation or standards as appropriate.
Update the USE_YOURCOMMAND(NEWTOY(yourcommand,"blah",0)) line. The NEWTOY macro fills out this command's toy_list structure. The arguments to the NEWTOY macro are:
the name used to run your command
the command line argument option parsing string (NULL if none)
a bitfield of TOYFLAG values (defined in toys.h) providing additional information such as where your command should be installed on a running system, whether to blank umask before running, whether or not the command must run as root (and thus should retain root access if installed SUID), and so on.
Change the kconfig data (from "config YOURCOMMAND" to the end of the comment block) to supply your command's configuration and help information. The uppper case config symbols are used by menuconfig, and are also what the CFG_ and USE_() macros are generated from (see [TODO]). The help information here is used by menuconfig, and also by the "help" command to describe your new command. (See [TODO] for details.) By convention, unfinished commands default to "n" and finished commands default to "y", so "make defconfig" selects all finished commands. (Note, "finished" means "ready to be used", not that it'll never change again.)
Each help block should start with a "usage: yourcommand" line explaining any command line arguments added by this config option. The "help" command outputs this text, and scripts/config2help.c in the build infrastructure collates these usage lines for commands with multiple configuration options when producing generated/help.h.
Update the DEFINE_GLOBALS() macro to contain your command's global variables, and also change the name "hello" in the #define TT line afterwards to the name of your command. If your command has no global variables, delete this macro (and the #define TT line afterwards). Note that if you specified two-character command line arguments in NEWTOY(), the first few global variables will be initialized by the automatic argument parsing logic, and the type and order of these variables must correspond to the arguments specified in NEWTOY(). See [TODO] for details.
If you didn't delete the DEFINE_GLOBALS macro, change the "#define TT this.hello" line to use your command name in place of the "hello". This is a shortcut to access your global variables as if they were members of the global struct "TT". (Access these members with a period ".", not a right arrow "->".)
Rename hello_main() to yourcommand_main(). This is the main() function where execution of your command starts. See [TODO] to figure out what happened to your command line arguments and how to access them.
This directory contains global infrastructure.
Each command #includes "toys.h" as part of its standard prolog.
This file sucks in most of the commonly used standard #includes, so individual files can just #include "toys.h" and not have to worry about stdargs.h and so on. Individual commands still need to #include special-purpose headers that may not be present on all systems (and thus would prevent toybox from building that command on such a system with that command enabled). Examples include regex support, any "linux/" or "asm/" headers, mtab support (mntent.h and sys/mount.h), and so on.
The toys.h header also defines structures for most of the global variables provided to each command by toybox_main(). These are described in detail in the description for main.c, where they are initialized.
The global variables are grouped into structures (and a union) for space savings, to more easily track the amount of memory consumed by them, so that they may be automatically cleared/initialized as needed, and so that access to global variables is more easily distinguished from access to local variables.
Contains the main() function where execution starts, plus common infrastructure to initialize global variables and select which command to run. The "toybox" multiplexer command also lives here. (This is the only command defined outside of the toys directory.)
Execution starts in main() which trims any path off of the first command name and calls toybox_main(), which calls toy_exec(), which calls toy_find() and toy_init() before calling the appropriate command's function from toy_list (via toys.which->toy_main()). If the command is "toybox", execution recurses into toybox_main(), otherwise the call goes to the appropriate commandname_main() from a C file in the toys directory.
The following global variables are defined in main.c:
struct toy_list toy_list - array describing all the commands currently configured into toybox. The first entry (toy_list) is for the "toybox" multiplexer command, which runs all the other built-in commands without symlinks by using its first argument as the name of the command to run and the rest as that command's argument list (ala "./toybox echo hello"). The remaining entries are the commands in alphabetical order (for efficient binary search).
This is a read-only array initialized at compile time by defining macros and #including generated/newtoys.h.
Members of struct toy_list (defined in "toys.h") include:
char *name - the name of this command.
void (*toy_main)(void) - function pointer to run this command.
char *options - command line option string (used by get_optflags() in lib/args.c to intialize toys.optflags, toys.optargs, and entries in the toy's DEFINE_GLOBALS struct). When this is NULL, no option parsing is done before calling toy_main().
int flags - Behavior flags for this command. The following flags are currently understood:
These flags are combined with | (or). For example, to install a command in /usr/bin, or together TOYFLAG_USR|TOYFLAG_BIN.
struct toy_context toys - global structure containing information common to all commands, initializd by toy_init() and defined in "toys.h". Members of this structure include:
struct toy_list *which - a pointer to this command's toy_list structure. Mostly used to grab the name of the running command (toys->which.name).
int exitval - Exit value of this command. Defaults to zero. The error_exit() functions will return 1 if this is zero, otherwise they'll return this value.
char **argv - "raw" command line options, I.E. the original unmodified string array passed in to main(). Note that modifying this changes "ps" output, and is not recommended. This array is null terminated; a NULL entry indicates the end of the array.
Most commands don't use this field, instead the use optargs, optflags, and the fields in the DEFINE_GLOBALS struct initialized by get_optflags().
unsigned optflags - Command line option flags, set by get_optflags(). Indicates which of the command line options listed in toys->which.options occurred this time.
The rightmost command line argument listed in toys->which.options sets bit 1, the next one sets bit 2, and so on. This means the bits are set in the same order the binary digits would be listed if typed out as a string. For example, the option string "abcd" would parse the command line "-c" to set optflags to 2, "-a" would set optflags to 8, and "-bd" would set optflags to 6 (4|2).
Only letters are relevant to optflags. In the string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter initializes global variables (see [TODO] DECLARE_GLOBALS() for details).
For more information on option parsing, see get_optflags().
char **optargs - Null terminated array of arguments left over after get_optflags() removed all the ones it understood. Note: optarg is the first argument, not the command name. Use toys.which->name for the command name.
int optc - Optarg count, equivalent to argc but for optargs.
int exithelp - Whether error_exit() should print a usage message via help_main() before exiting. (True during option parsing, defaults to false afterwards.)
union toy_union this - Union of structures containing each command's global variables.
Global variables are useful: they reduce the overhead of passing extra command line arguments between functions, they conveniently start prezeroed to save initialization costs, and the command line argument parsing infrastructure can also initialize global variables with its results.
But since each toybox process can only run one command at a time, allocating space for global variables belonging to other commands you aren't currently running would be wasteful.
Toybox handles this by encapsulating each command's global variables in a structure, and declaring a union of those structures with a single global instance (called "this"). The DEFINE_GLOBALS() macro contains the global variables that should go in the current command's global structure. Each variable can then be accessed as "this.commandname.varname". Generally, the macro TT is #defined to this.commandname so the variable can then be accessed as "TT.variable". See toys/hello.c for an example.
A command that needs global variables should declare a structure to contain them all, and add that structure to this union. A command should never declare global variables outside of this, because such global variables would allocate memory when running other commands that don't use those global variables.
The first few fields of this structure can be intialized by get_optargs(), as specified by the options field off this command's toy_list entry. See the get_optargs() description in lib/args.c for details.
The following functions are defined in main.c:
struct toy_list *toy_find(char *name) - Return the toy_list structure for this command name, or NULL if not found.
void toy_init(struct toy_list *which, char *argv) - fill out the global toys structure, calling get_optargs() if necessary.
void toy_exec(char *argv) - Run a built-in command with arguments.
Calls toy_find() on argv (which must be just a command name without path). Returns if it can't find this command, otherwise calls toy_init(), toys->which.toy_main(), and exit() instead of returning.
Use the library function xexec() to fall back to external executables in $PATH if toy_exec() can't find a built-in command. Note that toy_exec() does not strip paths before searching for a command, so "./command" will never match an internal command.
void toybox_main(void) - the main function for the multiplexer command (I.E. "toybox"). Given a command name as its first argument, calls toy_exec() on its arguments. With no arguments, it lists available commands. If the first argument starts with "-" it lists each command with its default install path prepended.
Top level configuration file in a stylized variant of kconfig format. Includes generated/Config.in.
These files are directly used by "make menuconfig" to select which commands to build into toybox (thus generating a .config file), and by scripts/config2help.py to create generated/help.h.
There is one temporary file in the top level source directory:
.config - Configuration file generated by kconfig, indicating which commands (and options to commands) are currently enabled. Used to make generated/config.h and determine which toys/*.c files to build.
You can create a human readable "miniconfig" version of this file using these instructions.
The "generated/" directory contains files generated from other source code in toybox. All of these files can be recreated by the build system, although some (such as generated/help.h) are shipped in release versions to reduce environmental dependencies (I.E. so you don't need python on your build system).
generated/config.h - list of CFG_SYMBOL and USE_SYMBOL() macros, generated from .config by a sed invocation in the top level Makefile.
CFG_SYMBOL is a comple time constant set to 1 for enabled symbols and 0 for disabled symbols. This allows the use of normal if() statements to remove code at compile time via the optimizer's dead code elimination (which removes from the binary any code that cannot be reached). This saves space without cluttering the code with #ifdefs or leading to configuration dependent build breaks. (See the 1992 Usenix paper #ifdef Considered Harmful for more information.)
USE_SYMBOL(code) evaluates to the code in parentheses when the symbol is enabled, and nothing when the symbol is disabled. This can be used for things like varargs or variable declarations which can't always be eliminated by a simple test on CFG_SYMBOL. Note that (unlike CFG_SYMBOL) this is really just a variant of #ifdef, and can still result in configuration dependent build breaks. Use with caution.
Included from the top level Config.in, contains one or more configuration entries for each command.
Each command has a configuration entry matching the command name (although configuration symbols are uppercase and command names are lower case). Options to commands start with the command name followed by an underscore and the option name. Global options are attached to the "toybox" command, and thus use the prefix "TOYBOX_". This organization is used by scripts/cfg2files to select which toys/*.c files to compile for a given .config.
A command with multiple names (or multiple similar commands implemented in the same .c file) should have config symbols prefixed with the name of their C file. I.E. config symbol prefixes are NEWTOY() names. If OLDTOY() names have config symbols they're options (symbols with an underscore and suffix) to the NEWTOY() name. (See toys/toylist.h)
The first half of this file prototypes all the structures to hold global variables for each command, and puts them in toy_union. These prototypes are only included if the macro NEWTOY isn't defined (in which case NEWTOY is defined to a default value that produces function prototypes).
The second half of this file lists all the commands in alphabetical order, along with their command line arguments and install location. Each command has an appropriate configuration guard so only the commands that are enabled wind up in the list.
The first time this header is #included, it defines structures and produces function prototypes for the commands in the toys directory.
The first time it's included, it defines structures and produces function prototypes. This is used to initialize toy_list in main.c, and later in that file to initialize NEED_OPTIONS (to figure out whether the command like parsing logic is needed), and to put the help entries in the right order in toys/help.c.
#defines two help text strings for each command: a single line command_help and an additinal command_help_long. This is used by help_main() in toys/help.c to display help for commands.
Although this file is generated from Config.in help entries by scripts/config2help.py, it's shipped in release tarballs so you don't need python on the build system. (If you check code out of source control, or modify Config.in, then you'll need python installed to rebuild it.)
This file contains help for all commands, regardless of current configuration, but only the currently enabled ones are entered into help_data in toys/help.c.
lib: llist, getmountlist(), error_msg/error_exit, xmalloc(), strlcpy(), xexec(), xopen()/xread(), xgetcwd(), xabspath(), find_in_path(), itoa().
Toybox's main.c automatically parses command line options before calling the command's main function. Option parsing starts in get_optflags(), which stores results in the global structures "toys" (optflags and optargs) and "this".
The option parsing infrastructure stores a bitfield in toys.optflags to indicate which options the current command line contained. Arguments attached to those options are saved into the command's global structure ("this"). Any remaining command line arguments are collected together into the null-terminated array toys.optargs, with the length in toys.optc. (Note that toys.optargs does not contain the current command name at position zero, use "toys.which->name" for that.) The raw command line arguments get_optflags() parsed are retained unmodified in toys.argv.
Toybox's option parsing logic is controlled by an "optflags" string, using a format reminiscent of getopt's optargs but has several important differences. Toybox does not use the getopt() function out of the C library, get_optflags() is an independent implementation which doesn't permute the original arguments (and thus doesn't change how the command is displayed in ps and top), and has many features not present in libc optargs() (such as the ability to describe long options in the same string as normal options).
Each command's NEWTOY() macro has an optflags string as its middle argument, which sets toy_list.options for that command to tell get_optflags() what command line arguments to look for, and what to do with them. If a command has no option definition string (I.E. the argument is NULL), option parsing is skipped for that command, which must look at the raw data in toys.argv to parse its own arguments. (If no currently enabled command uses option parsing, get_optflags() is optimized out of the resulting binary by the compiler's --gc-sections option.)
You don't have to free the option strings, which point into the environment space (I.E. the string data is not copied). A TOYFLAG_NOFORK command that uses the linked list type "*" should free the list objects but not the data they point to, via "llist_free(TT.mylist, NULL);". (If it's not NOFORK, exit() will free all the malloced data anyway unless you want to implement a CONFIG_TOYBOX_FREE cleanup for it.)
Note: the optflags option description string format is much more concisely described by a large comment at the top of lib/args.c.
The general theory is that letters set optflags, and punctuation describes other actions the option parsing logic should take.
For example, suppose the command line command -b fruit -d walrus -a 42 is parsed using the optflags string "a#b:c:d". (I.E. toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d", "walrus", "-a", "42"]). When get_optflags() returns, the following data is available to command_main():
In struct toys:
In union this (treated as long this):
If the command's globals are:
DECLARE_GLOBALS( char *c; char *b; long a; ) #define TT this.command
That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember, each entry that receives an argument must be a long or pointer, to line up with the array position. Right to left in the optflags string corresponds to top to bottom in DECLARE_GLOBALS().
Each option in the optflags string corresponds to a bit position in toys.optflags, with the same value as a corresponding binary digit. The rightmost argument is (1<<0), the next to last is (1<<1) and so on. If the option isn't encountered while parsing argv, its bit remains 0.
For example, the optflags string "abcd" would parse the command line argument "-c" to set optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to 6 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8).
Only letters are relevant to optflags, punctuation is skipped: in the string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter usually indicate that the option takes an argument.
Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is the amount a long would have on 32-bit platforms anyway; 64 bit code on 32 bit platforms is too expensive to require in common code used by almost all commands.) Bit positions beyond the 1<<31 aren't recorded, but parsing higher options can still set global variables.
Automatically setting global variables from arguments (union this)
The following punctuation characters may be appended to an optflags argument letter, indicating the option takes an additional argument:
Arguments may occur with or without a space (I.E. "-a 42" or "-a42"). The command line argument "-abc" may be interepreted many different ways: the optflags string "cba" sets toys.optflags = 7, "c:ba" sets toys.optflags=4 and saves "ba" as the argument to -c, and "cb:a" sets optflags to 6 and saves "c" as the argument to -b.
Options which have an argument fill in the corresponding slot in the global union "this" (see generated/globals.h), treating it as an array of longs with the rightmost saved in this. Again using "a*b:c#d", "-c 42" would set this=42; and "-b 42" would set this="42"; each slot is left NULL if the corresponding argument is not encountered.
This behavior is useful because the LP64 standard ensures long and pointer are the same size. C99 guarantees structure members will occur in memory in the same order they're declared, and that padding won't be inserted between consecutive variables of register size. Thus the first few entries can be longs or pointers corresponding to the saved arguments.
Command line arguments in argv which are not consumed by option parsing (I.E. not recognized either as -flags or arguments to -flags) will be copied to toys.optargs, with the length of that array in toys.optc. (When toys.optc is 0, no unrecognized command line arguments remain.) The order of entries is preserved, and as with argv this new array is also terminated by a NULL entry.
Option parsing can require a minimum or maximum number of optargs left over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the start of the optflags string.
The special argument "--" terminates option parsing, storing all remaining arguments in optargs. The "--" itself is consumed.
Other optflags control characters
The following characters may occur at the start of each command's optflags string, before any options that would set a bit in toys.optflags:
The following characters may be appended to an option character, but do not by themselves indicate an extra argument should be saved in this. (Technically any character not recognized as a control character sets an optflag, but letters are never control characters.)
The following may be appended to a float or double:
Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT is enabled. (Otherwise the code to determine where floating point constants end drops out. When disabled, it can reserve a global data slot for the argument so offsets won't change, but will never fill it out.). You can handle this by using the USE_BLAH() macros with C string concatenation, ala:
"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"
The optflags string can contain long options, which are enclosed in parentheses. They may be appended to an existing option character, in which case the --longopt is a synonym for that option, ala "a:(--fred)" which understands "-a blah" or "--fred blah" as synonyms.
Longopts may also appear before any other options in the optflags string, in which case they have no corresponding short argument, but instead set their own bit based on position. So for "(walrus)#(blah)xy:z" "command --walrus 42" would set toys.optflags = 16 (-z = 1, -y = 2, -x = 4, --blah = 8) and would assign this = 42;
A short option may have multiple longopt synonyms, "a(one)(two)", but each "bare longopt" (ala "(one)(two)abc" before any option characters) always sets its own bit (although you can group them with +X).
Run .config through this filter to get a list of enabled commands, which is turned into a list of files in toys via a sed invocation in the top level Makefile.
Menuconfig infrastructure copied from the Linux kernel. See the Linux kernel's Documentation/kbuild/kconfig-language.txt
All the files in this directory except the README are generated by the build. (See scripts/make.sh)
config.h - CFG_COMMAND and USE_COMMAND() macros set by menuconfig via .config.
Config.in - Kconfig entries for each command. Included by top level Config.in. The help text in here is used to generated help.h
help.h - Help text strings for use by "help" command. Building this file requires python on the host system, so the prebuilt file is shipped in the build tarball to avoid requiring python to build toybox.
newtoys.h - List of NEWTOY() or OLDTOY() macros for all available commands. Associates command_main() functions with command names, provides option string for command line parsing (see lib/args.c), specifies where to install each command and whether toysh should fork before calling it.
Everything in this directory is a derivative file produced from something else. The entire directory is deleted by "make distclean".