Directory lib/

TODO: document lots more here.

lib: getmountlist(), error_msg/error_exit, xmalloc(), strlcpy(), xexec(), xopen()/xread(), xgetcwd(), xabspath(), find_in_path(), itoa().

lib/xwrap.c

Functions prefixed with the letter x call perror_exit() when they hit errors, to eliminate common error checking. This prints an error message and the strerror() string for the errno encountered.

We replaced exit(), _exit(), and atexit() with xexit(), _xexit(), and sigatexit(). This gives _xexit() the option to siglongjmp(toys.rebound, 1) instead of exiting, lets xexit() report stdout flush failures to stderr and change the exit code to indicate error, lets our toys.exit function change happen for signal exit paths and lets us remove the functions after we've called them.

You can intercept our exit by assigning a sigsetjmp/siglongjmp buffer to toys.rebound (set it back to zero to restore the default behavior). If you do this, cleaning up resource leaks is your problem.

lib/lib.c

Eight gazillion common functions, see lib/lib.h for the moment:

lib/portability.h

This file is automatically included from the top of toys.h, and smooths over differences between platforms (hardware targets, compilers, C libraries, operating systems, etc).

This file provides SWAP macros (SWAP_BE16(x) and SWAP_LE32(x) and so on).

A macro like SWAP_LE32(x) means "The value in x is stored as a little endian 32 bit value, so perform the translation to/from whatever the native 32-bit format is". You do the swap once on the way in, and once on the way out. If your target is already little endian, the macro is a NOP.

The SWAP macros come in BE and LE each with 16, 32, and 64 bit versions. In each case, the name of the macro refers to the _external_ representation, and converts to/from whatever your native representation happens to be (which can vary depending on what you're currently compiling for).

lib/llist.c

Some generic single and doubly linked list functions, which take advantage of a couple properties of C:

• Structure elements are laid out in memory in the order listed, and the first element has no padding. This means you can always treat (typecast) a pointer to a structure as a pointer to the first element of the structure, even if you don't know anything about the data following it.

• An array of length zero at the end of a structure adds no space to the sizeof() the structure, but if you calculate how much extra space you want when you malloc() the structure it will be available at the end. Since C has no bounds checking, this means each struct can have one variable length array.

Toybox's list structures always have their next pointer as the first entry of each struct, and singly linked lists end with a NULL pointer. This allows generic code to traverse such lists without knowing anything else about the specific structs composing them: if your pointer isn't NULL typecast it to void ** and dereference once to get the next entry.

lib/lib.h defines three structure types:

• struct string_list - stores a single string (char str[0]), memory for which is allocated as part of the node. (I.E. llist_traverse(list, free); can clean up after this type of list.)

• struct arg_list - stores a pointer to a single string (char *arg) which is stored in a separate chunk of memory.

• struct double_list - has a second pointer (struct double_list *prev along with a char *data for payload.

• void *llist_pop(void **list) - advances through a list ala node = llist_pop(&list); This doesn't modify the list contents, but does advance the pointer you feed it (which is why you pass the _address_ of that pointer, not the pointer itself).

• void llist_traverse(void *list, void (*using)(void *data)) - iterate through a list calling a function on each node.

• struct double_list *dlist_add(struct double_list **llist, char *data) - append an entry to a circular linked list. This function allocates a new struct double_list wrapper and returns the pointer to the new entry (which you can usually ignore since it's llist->prev, but if llist was NULL you need it). The argument is the ->data field for the new node.

• void dlist_add_nomalloc(struct double_list **llist, struct double_list *new) - append existing struct double_list to list, does not allocate anything.

• Why do arg_list and double_list contain a char * payload instead of a void *? - Because you always have to typecast a void * to use it, and typecasting a char * does no harm. Since strings are the most common payload, and doing math on the pointer ala "(type *)(ptr+sizeof(thing)+sizeof(otherthing))" requires ptr to be char * anyway (at least according to the C standard), defaulting to char * saves a typecast.

• Why do the names ->str, ->arg, and ->data differ? - To force you to keep track of which one you're using, calling free(node->str) would be bad, and _failing_ to free(node->arg) leaks memory.

• Why does llist_pop() take a void * instead of void **? - because the stupid compiler complains about "type punned pointers" when you typecast and dereference on the same line, due to insane FSF developers hardwiring limitations of their optimizer into gcc's warning system. Since C automatically typecasts any other pointer type to and from void *, the current code works fine. It's sad that it won't warn you if you forget the &, but the code crashes pretty quickly in that case.

• How do I assemble a singly-linked-list in order? - use a double_list, dlist_add() your entries, and then call dlist_terminate(list) to break the circle when done (turning the last ->next and the first ->prev into NULLs).

lib/args.c

Toybox's main.c automatically parses command line options before calling the command's main function. Option parsing starts in get_optflags(), which stores results in the global structures "toys" (optflags and optargs) and "this".

The option parsing infrastructure stores a bitfield in toys.optflags to indicate which options the current command line contained, and defines FLAG macros code can use to check whether each argument's bit is set. Arguments attached to those options are saved into the command's global structure ("this"). Any remaining command line arguments are collected together into the null-terminated array toys.optargs, with the length in toys.optc. (Note that toys.optargs does not contain the current command name at position zero, use "toys.which->name" for that.) The raw command line arguments get_optflags() parsed are retained unmodified in toys.argv[].

Toybox's option parsing logic is controlled by an "optflags" string, using a format reminiscent of getopt's optargs but with several important differences. Toybox does not use the getopt() function out of the C library, get_optflags() is an independent implementation which doesn't permute the original arguments (and thus doesn't change how the command is displayed in ps and top), and has many features not present in libc optargs() (such as the ability to describe long options in the same string as normal options).

Each command's NEWTOY() macro has an optflags string as its middle argument, which sets toy_list.options for that command to tell get_optflags() what command line arguments to look for, and what to do with them. If a command has no option definition string (I.E. the argument is NULL), option parsing is skipped for that command, which must look at the raw data in toys.argv to parse its own arguments. (If no currently enabled command uses option parsing, get_optflags() is optimized out of the resulting binary by the compiler's --gc-sections option.)

You don't have to free the option strings, which point into the environment space (I.E. the string data is not copied). A TOYFLAG_NOFORK command that uses the linked list type "*" should free the list objects but not the data they point to, via "llist_free(TT.mylist, NULL);". (If it's not NOFORK, exit() will free all the malloced data anyway unless you want to implement a CONFIG_TOYBOX_FREE cleanup for it.)

Optflags format string

Note: the optflags option description string format is much more concisely described by a large comment at the top of lib/args.c.

The general theory is that letters set optflags, and punctuation describes other actions the option parsing logic should take.

For example, suppose the command line command -b fruit -d walrus -a 42 is parsed using the optflags string "a#b:c:d". (I.E. toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d", "walrus", "-a", "42"]). When get_optflags() returns, the following data is available to command_main():

• In struct toys:

  • toys.optflags = 13; // FLAG_a = 8 | FLAG_b = 4 | FLAG_d = 1
  • toys.optargs[0] = "walrus"; // leftover argument
  • toys.optargs[1] = NULL; // end of list
  • toys.optc = 1; // there was 1 leftover argument
  • toys.argv[] = {"-b", "fruit", "-d", "walrus", "-a", "42"}; // The original command line arguments

• In union this (treated as long this[]):

  • this[0] = NULL; // -c didn't get an argument this time, so get_optflags() didn't change it and toys_init() zeroed "this" during setup.)
  • this[1] = (long)"fruit"; // argument to -b
  • this[2] = 42; // argument to -a

If the command's globals are:

GLOBALS(
	char *c;
	char *b;
	long a;
)

That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember, each entry that receives an argument must be a long or pointer, to line up with the array position. Right to left in the optflags string corresponds to top to bottom in GLOBALS().

Put globals not filled out by the option parsing logic at the end of the GLOBALS block. Common practice is to list the options one per line (to make the ordering explicit, first to last in globals corresponds to right to left in the option string), then leave a blank line before any non-option globals.

long toys.optflags

Each option in the optflags string corresponds to a bit position in toys.optflags, with the same value as a corresponding binary digit. The rightmost argument is (1<<0), the next to last is (1<<1) and so on. If the option isn't encountered while parsing argv[], its bit remains 0.

Each option -x has a FLAG_x macro for the command letter. Bare --longopts with no corresponding short option have a FLAG_longopt macro for the long optionname. Commands enable these macros by #defining FOR_commandname before #including . When multiple commands are implemented in the same source file, you can switch flag contexts later in the file by #defining CLEANUP_oldcommand and #defining FOR_newcommand, then #including .

Options disabled in the current configuration (wrapped in a USE_BLAH() macro for a CONFIG_BLAH that's switched off) have their corresponding FLAG macro set to zero, so code checking them ala if (toys.optargs & FLAG_x) gets optimized out via dead code elimination. #defining FORCE_FLAGS when switching flag context disables this behavior: the flag is never zero even if the config is disabled. This allows code shared between multiple commands to use the same flag values, as long as the common flags match up right to left in both option strings.

For example, the optflags string "abcd" would parse the command line argument "-c" to set optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to 6 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8). To check if -c was encountered, code could test: if (toys.optflags & FLAG_c) printf("yup"); (See the toys/examples directory for more.)

Only letters are relevant to optflags, punctuation is skipped: in the string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter usually indicate that the option takes an argument.

Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is the amount a long would have on 32-bit platforms anyway; 64 bit code on 32 bit platforms is too expensive to require in common code used by almost all commands.) Bit positions beyond the 1<<31 aren't recorded, but parsing higher options can still set global variables.

Automatically setting global variables from arguments (union this)

The following punctuation characters may be appended to an optflags argument letter, indicating the option takes an additional argument:

• : - plus a string argument, keep most recent if more than one.
• * - plus a string argument, appended to a linked list.
• @ - plus an occurrence counter (stored in a long)
• # - plus a signed long argument.
• - - plus a signed long argument defaulting to negative (start argument with + to force a positive value).
• . - plus a floating point argument (if CFG_TOYBOX_FLOAT).
The following can be appended to a float or double:
• <123 - error if argument is less than this
• >123 - error if argument is greater than this
• =123 - default value if argument not supplied

GLOBALS

Options which have an argument fill in the corresponding slot in the global union "this" (see generated/globals.h), treating it as an array of longs with the rightmost saved in this[0]. As described above, using "a*b:c#d", "-c 42" would set this[0] = 42; and "-b 42" would set this[1] = "42"; each slot is left NULL if the corresponding argument is not encountered.

This behavior is useful because the LP64 standard ensures long and pointer are the same size. C99 guarantees structure members will occur in memory in the same order they're declared, and that padding won't be inserted between consecutive variables of register size. Thus the first few entries can be longs or pointers corresponding to the saved arguments.

The main downside is that numeric arguments ("#" and "-" format) are limited to +- 2 billion on 32 bit platforms (the "truncate -s 8G" problem), because long is only 64 bits on 64 bit hosts, so the capabilities of some tools differ when built in 32 bit vs 64 bit mode. Fixing this kind of ugly and even embedded designs are slowly moving to 64 bits, so our current plan is to document the problem and wait it out. (If "x32 mode" and similar becomes popular enough, we may revisit this decision.)

See toys/example/*.c for longer examples of parsing options into the GLOBALS block.

char *toys.optargs[]

Command line arguments in argv[] which are not consumed by option parsing (I.E. not recognized either as -flags or arguments to -flags) will be copied to toys.optargs[], with the length of that array in toys.optc. (When toys.optc is 0, no unrecognized command line arguments remain.) The order of entries is preserved, and as with argv[] this new array is also terminated by a NULL entry.

Option parsing can require a minimum or maximum number of optargs left over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the start of the optflags string.

The special argument "--" terminates option parsing, storing all remaining arguments in optargs. The "--" itself is consumed.

Other optflags control characters

The following characters may occur at the start of each command's optflags string, before any options that would set a bit in toys.optflags:

• ^ - stop at first nonoption argument (for nice, xargs...)
• ? - allow unknown arguments (pass non-option arguments starting with - through to optargs instead of erroring out).
• & - the first argument has imaginary dash (ala tar/ps. If given twice, all arguments have imaginary dash.)
• < - must be followed by a decimal digit indicating at least this many leftover arguments are needed in optargs (default 0)
• > - must be followed by a decimal digit indicating at most this many leftover arguments allowed (default MAX_INT)

The following characters may be appended to an option character, but do not by themselves indicate an extra argument should be saved in this[]. (Technically any character not recognized as a control character sets an optflag, but letters are never control characters.)

• ^ - stop parsing options after encountering this option, everything else goes into optargs.
• | - this option is required. If more than one marked, only one is required.

The following may be appended to a float or double:

• <123 - error if argument is less than this
• >123 - error if argument is greater than this
• =123 - default value if argument not supplied

Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT is enabled. (Otherwise the code to determine where floating point constants end drops out. When disabled, it can reserve a global data slot for the argument so offsets won't change, but will never fill it out.) You can handle this by using the USE_BLAH() macros with C string concatenation, ala:

"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"

--longopts

The optflags string can contain long options, which are enclosed in parentheses. They may be appended to an existing option character, in which case the --longopt is a synonym for that option, ala "a:(--fred)" which understands "-a blah" or "--fred blah" as synonyms.

Longopts may also appear before any other options in the optflags string, in which case they have no corresponding short argument, but instead set their own bit based on position. So for "(walrus)#(blah)xy:z", "command --walrus 42" would set toys.optflags = 16 (-z = 1, -y = 2, -x = 4, --blah = 8) and would assign this[1] = 42;

A short option may have multiple longopt synonyms, "a(one)(two)", but each "bare longopt" (ala "(one)(two)abc" before any option characters) always sets its own bit (although you can group them with +X).

Only bare longopts have a FLAG_ macro with the longopt name (ala --fred would #define FLAG_fred). Other longopts use the short option's FLAG macro to test the toys.optflags bit.

Options with a semicolon ";" after their data type can only set their corresponding GLOBALS() entry via "--longopt=value". For example, option string "x(boing): y" would set TT.x if it saw "--boing=value", but would treat "--boing value" as setting FLAG_x in toys.optargs, leaving TT.x NULL, and keeping "value" in toys.optargs[]. (This lets "ls --color" and "ls --color=auto" both work.)

[groups]

At the end of the option string, square bracket groups can define relationships between existing options. (This only applies to short options, bare --longopts can't participate.)

The first character of the group defines the type, the remaining characters are options it applies to:

• - - Exclusive, switch off all others in this group.
• + - Inclusive, switch on all others in this group.
• ! - Error, fail if more than one defined.

So "abc[-abc]" means -ab = -b, -ba = -a, -abc = -c. "abc[+abc]" means -ab=-abc, -c=-abc, and "abc[!abc] means -ab calls error_exit("no -b with -a"). Note that [-] groups clear the GLOBALS option slot of options they're switching back off, but [+] won't set options it didn't see (just the optflags).

whitespace

Arguments may occur with or without a space (I.E. "-a 42" or "-a42"). The command line argument "-abc" may be interepreted many different ways: the optflags string "cba" sets toys.optflags = 7, "c:ba" sets toys.optflags=4 and saves "ba" as the argument to -c, and "cb:a" sets optflags to 6 and saves "c" as the argument to -b.

Note that & changes whitespace handling, so that the command line "tar cvfCj outfile.tar.bz2 topdir filename" is parsed the same as "tar filename -c -v -j -f outfile.tar.bz2 -C topdir". Note that "tar -cvfCj one two three" would equal "tar -c -v -f Cj one two three". (This matches historical usage.)

Appending a space to the option in the option string ("a: b") makes it require a space, I.E. "-ab" is interpreted as "-a" "-b". That way "kill -stop" differs from "kill -s top".

Appending ; to a longopt in the option string makes its argument optional, and only settable with =, so in ls "(color):;" can accept "ls --color" and "ls --color=auto" without complaining that the first has no argument.

lib/dirtree.c

The directory tree traversal code should be sufficiently generic that commands never need to use readdir(), scandir(), or the fts.h family of functions.

These functions do not call chdir() or rely on PATH_MAX. Instead they use openat() and friends, using one filehandle per directory level to recurse into subdirectories. (I.E. they can descend 1000 directories deep if setrlimit(RLIMIT_NOFILE) allows enough open filehandles, and the default in /proc/self/limits is generally 1024.)

There are two main ways to use dirtree: 1) assemble a tree of nodes representing a snapshot of directory state and traverse them using the ->next and ->child pointers, or 2) traverse the tree calling a callback function on each entry, and freeing its node afterwards. (You can also combine the two, using the callback as a filter to determine which nodes to keep.)

The basic dirtree functions are:

• struct dirtree *dirtree_read(char *path, int (*callback)(struct dirtree node)) - recursively read files and directories, calling callback() on each, and returning a tree of saved nodes (if any). If path doesn't exist, returns DIRTREE_ABORTVAL. If callback is NULL, returns a single node at that path.

• dirtree_notdotdot(struct dirtree *new) - standard callback which discards "." and ".." entries and returns DIRTREE_SAVE|DIRTREE_RECURSE for everything else. Used directly, this assembles a snapshot tree of the contents of this directory and its subdirectories to be processed after dirtree_read() returns (by traversing the struct dirtree's ->next and ->child pointers from the returned root node).

• dirtree_path(struct dirtree *node, int *plen) - malloc() a string containing the path from the root of this tree to this node. If plen isn't NULL then *plen is how many extra bytes to malloc at the end of string.

• dirtree_parentfd(struct dirtree *node) - return fd of directory containing this node, for use with openat() and such.

The dirtree_read() function is the standard way to start directory traversal. It takes two arguments: a starting path for the root of the tree, and a callback function. The callback() is called on each directory entry, its argument is a fully populated struct dirtree * (from lib/lib.h) describing the node, and its return value tells the dirtree infrastructure what to do next.

(There's also a three argument version, dirtree_flagread(char *path, int flags, int (*callback)(struct dirtree node)), which lets you apply flags like DIRTREE_SYMFOLLOW and DIRTREE_SHUTUP to reading the top node, but this only affects the top node. Child nodes use the flags returned by callback().

struct dirtree

Each struct dirtree node contains char name[] and struct stat st entries describing a file, plus a char *symlink which is NULL for non-symlinks.

During a callback function, the int dirfd field of directory nodes contains a directory file descriptor (for use with the openat() family of functions). This isn't usually used directly, intstead call dirtree_parentfd() on the callback's node argument. The char again field is 0 for the first callback on a node, and 1 on the second callback (triggered by returning DIRTREE_COMEAGAIN on a directory, made after all children have been processed).

Users of this code may put anything they like into the long extra field. For example, "cp" and "mv" use this to store a dirfd for the destination directory (and use DIRTREE_COMEAGAIN to get the second callback so they can close(node->extra) to avoid running out of filehandles). This field is not directly used by the dirtree code, and thanks to LP64 it's large enough to store a typecast pointer to an arbitrary struct.

The return value of the callback combines flags (with boolean or) to tell the traversal infrastructure how to behave:

• DIRTREE_SAVE - Save this node, assembling a tree. (Without this the struct dirtree is freed after the callback returns. Filtering out siblings is fine, but discarding a parent while keeping its child leaks memory.)

• DIRTREE_ABORT - Do not examine any more entries in this directory. (Does not propagate up tree: to abort entire traversal, return DIRTREE_ABORT from parent callbacks too.)

• DIRTREE_RECURSE - Examine directory contents. Ignored for non-directory entries. The remaining flags only take effect when recursing into the children of a directory.

• DIRTREE_COMEAGAIN - Call the callback on this node a second time after examining all directory contents, allowing depth-first traversal. On the second call, dirtree->again is nonzero.

• DIRTREE_SYMFOLLOW - follow symlinks when populating children's struct stat st (by feeding a nonzero value to the symfollow argument of dirtree_add_node()), which means DIRTREE_RECURSE treats symlinks to directories as directories. (Avoiding infinite recursion is the callback's problem: the non-NULL dirtree->symlink can still distinguish between them. The "find" command follows ->parent up the tree to the root node each time, checking to make sure that stat's dev and inode pair don't match any ancestors.)

Each struct dirtree contains three pointers (next, parent, and child) to other struct dirtree.

The parent pointer indicates the directory containing this entry; even when not assembling a persistent tree of nodes the parent entries remain live up to the root of the tree while child nodes are active. At the top of the tree the parent pointer is NULL, meaning the node's name[] is either an absolute path or relative to cwd. The function dirtree_parentfd() gets the directory file descriptor for use with openat() and friends, returning AT_FDCWD at the top of tree.

The child pointer points to the first node of the list of contents of this directory. If the directory contains no files, or the entry isn't a directory, child is NULL.

The next pointer indicates sibling nodes in the same directory as this node, and since it's the first entry in the struct the llist.c traversal mechanisms work to iterate over sibling nodes. Each dirtree node is a single malloc() (even char *symlink points to memory at the end of the node), so llist_free() works but its callback must descend into child nodes (freeing a tree, not just a linked list), plus whatever the user stored in extra.

The dirtree_flagread() function is a simple wrapper, calling dirtree_add_node() to create a root node relative to the current directory, then calling dirtree_handle_callback() on that node (which recurses as instructed by the callback return flags). The flags argument primarily lets you control whether or not to follow symlinks to the root node; symlinks listed on the command line are often treated differently than symlinks encountered during recursive directory traversal.

The ls command not only bypasses this wrapper, but never returns DIRTREE_RECURSE from the callback, instead calling dirtree_recurse() manually from elsewhere in the program. This gives ls -lR manual control of traversal order, which is neither depth first nor breadth first but instead a sort of FIFO order requried by the ls standard.

Directory scripts/

Build infrastructure. The makefile calls scripts/make.sh for "make" and scripts/install.sh for "make install".

There's also a test suite, "make test" calls make/test.sh, which runs all the tests in make/test/*. You can run individual tests via "scripts/test.sh command", or "TEST_HOST=1 scripts/test.sh command" to run that test against the host implementation instead of the toybox one.

scripts/cfg2files.sh

Run .config through this filter to get a list of enabled commands, which is turned into a list of files in toys via a sed invocation in the top level Makefile.

Directory kconfig/

Menuconfig infrastructure copied from the Linux kernel a long time ago (version 2.6.16). See the Linux kernel's Documentation/kbuild/kconfig-language.txt