CELF 2015 Toybox talk - http://landley.net/talks/celf-2015.txt Video of this talk: https://www.youtube.com/watch?v=04XwAbtPmAg Three previous talks I shouldn't repeat here: 1) "Why Toybox?" 2013 talk here at CELF video: http://youtu.be/SGmtP5Lg_t0 outline: http://landley.net/talks/celf-2013.txt linked from http://landley.net/toybox/ in nav bar on left as "Why is it?" - march 21, 2013 entry has section links. 2) "Why Public Domain?" The rise and fall of copyleft, Ohio LinuxFest 2013 audio: https://archive.org/download/OhioLinuxfest2013/24-Rob_Landley-The_Rise_and_Fall_of_Copyleft.mp3 outline: http://landley.net/talks/ohio-2013.txt 3) Why did I do Aboriginal Linux (which led me here) A 260 slide presentation about why: https://speakerdeck.com/landley/developing-for-non-x86-targets-using-qemu project page: http://landley.net/aboriginal/about.html How and why to make android self-hosting: http://landley.net/aboriginal/about.html#selfhost Random other thing I did the embedded world might care about: initmpfs http://thread.gmane.org/gmane.linux.kernel.mm/103438 initramfs can now be tmpfs, which is cool for $REASONS (listed in link) What is toybox? Summary paragraph at top of http://landley.net/toybox Toybox combines common Linux command line utilities together into a single BSD-licensed executable that's simple, small, fast, reasonably standards-compliant, and powerful enough to turn Android into a development environment. See the links on the left for details. Let's rebut Wikipedia[citation needed]'s opinion: http://en.wikipedia.org/wiki/Toybox - Existing linux implementations weighed about equally with Posix and LSB. - android in there too. And I'm at least vaguely curious what BSD does. - my main regression test is building Linux From Scratch (6.8) under it - They say 2006, file log on LICENSE in the repository says November 2011 - Tim's bentobox project proposed new code, I suggested reviving toybox. Who is using it? - toybox integrated into Android M, most of toolbox already moved over. - Elliott Hughes, the android core maintainer (bionic and toolbox) doing it. - I'm slowly poking at AOSP, but not up to speed yet - All the documentation in the world for java app development. - Low-level system stuff? Not so much. - tizen interested, has repo adding "smack", working out logistics - older version in meta-openembedded - packaged for gentoo, arch. Pending in buildroot (ball's in my court there). Development: Links in http://landley.net/toybox nav bar on left: Mailing list: development discussion, where I take patches Also freenode (IRC) channel #toybox http://landley.net/toybox/roadmap.html describes list of commands planned for 1.0 release and the reasons for each one. Not set in stone, but current goals http://landley.net/toybox/status.html automatically generated by scripts/mkstatus.py not the friendliest script I've ever written, but usage is easy: 1) make clean && make defconfig && make 2) www/mkstatus.py Reads www/roadmap.html and www/status.html - need to edit those to update commands in categories - mkstatus.py will list uncategorized commands runs ./toybox (cryptic python stack dump if not there, sorry) produces www/status.gen: in status.html Back in 0.4.3 defconfig had: basename bzcat cal cat catv chgrp chmod chown chroot chvt cksum clear cmp comm count cp cut date df dirname dmesg dos2unix du echo env expand false free head help hostname id insmod kill killall link ln login logname losetup ls lsmod md5sum mkdir mkfifo mknod mkswap mktemp modinfo mountpoint nc netcat nice nohup od oneit passwd patch pidof printenv pwd readlink realpath rev rm rmdir rmmod seq setsid sha1sum sleep sort swapoff swapon switch_root sync tac tail taskset tee touch true truncate tty uname uniq unix2dos unlink unshare uptime usleep vconfig vmstat w wc which whoami xargs yes In 0.4.4 through 0.5.2, 55 new commands promoted acpi base64 blkid blockdev chattr cpio egrep eject factor fallocate fgrep find freeramdisk fsfreeze fstype grep groups halt ifconfig inotifyd install killall5 lsattr lspci lsusb makedevs mix mkpasswd mount mv nbd-client nl partprobe paste pivot_root pmap poweroff printf pwdx readahead reboot renice rfkill sed split stat strings su sysctl time timeout umount uudecode uuencode who hard ones were: cpio, find, grep, ifconfig, mount, stat, printf, sed - "shrinking C code" panel later today covers ifconfig cleanup - See http://landley.net/toybox/cleanup.html - describes each of the 27 commits to ifconfig cleaning it up from old total: 1504 lines (44268 bytes) in 38 functions new total: 521 lines (15963 bytes) in 4 functions - new one 1/3 of the size, easier to read, did more. - BUT that's 6 months spent polishing a command that already worked. - I regression smoketest under glibc, musl, and enh does on bionic. - My Aboriginal Linux has a hacked up uClibc version based on the last-ever release from 2012. When I migrate that to musl, I stop testing uClibc. - my big regression test each release: it must build aboriginal and LFS 6.8 Remaining busybox commands used by aboriginal linux 1.4.0 with toybox 0.5.2 http://landley.net/hg/aboriginal/file/tip/sources/baseconfig-busybox - up through "toybox" comment bunzip2 dd ftpd gunzip pgrep route test wget diff ftpget gzip ping sh tr xzcat bzcat expr ftpput less pkill sha512sum, unxz, zcat awk fdisk getent man ps tar vi Some other big pending things on todo list towards 1.0: nommu support - actually mostly done, still converting xfork() to xpopen*() - if you can't share text segments, "make change" - Jeff Dionne, cofounder of uclinux, is my boss at se-instruments.com he was at the monday dinner, where he registered nommu.org so we can start a new condensation nuclei for the nommu community. - reviving nommu support for linux is bigger than just toybox. - nommu musl! nommu buildroot! collate/document elf2flt/fdpic! - working on it... toybox shell (toys/pending/toysh is a stub, doesn't even $VARIABLE yet). - quoting and variable expansion, VAR="$(echo "it's ${RECURSIVE:-fun}")" - flow control (if/for/while), signal handling, job control - reasonable bash replacement, ala path/{curly,bracket}, <(command) - command line editing and history you can't spell "evil" without "vi": - only editor in posix - infrastructure should accept other keybindings, behavior - real infrastructure need is "poor man's curses": - less, more, watch, command line editing, vi. - insert in middle, screen resize, serial, utf8, different escape codes... - easy to do, hard to do _right_. init (oneit/lunchd) - extending oneit with signal handling and completion pipe - write a "lunchd" a bit like macos launchd without xml config files awk - I learned sed by writing a sed implementation. Sigh. data compression - have finished bzcat and zcat, and xzcat in pending - working on deflate (zip/gzip/zlib) - may skip compression-side other two - I wrote a bzip2 compressor once for busybox, it's sad - layers of string sort fallback - semi-obsolete now? (gzip survives as streaming compressor) - as long as we have _a_ compressor, and can deal with incoming data - feel free to tell me why I'm wrong about that. Test suite and documentation: I need to spend so much more time on both. - tests/*.test - "scripts/test.sh command" vs "TEST_HOST=1 scripts/test.sh command" - also VERBOSE=1 and VERBOSE=fail - plumbing in scripts/runtest.sh testing "name" "command" "result" "infile" "stdin" optional CONFIG_SYMBOL id -u test for root (see chgrp.test) - Need to make aboriginal linux build control image to run tests - blkid has test data files, todo items is a standard tarball Shut up and show me the code: The main repository is mercurial, but there's a (read-only) git mirror. - submissions are mostly patches to the list. I can also clone git or hg, but then I usually move them over by hand. toys/* - menus are developer convenience, command namespace is flat - pending commands need cleanup ("shrinking C code", ballroom III) - they default to "n" in defconfig. - "make defconfig" selects maximum sane config Let's make a new command - self-contained, one file - help text: --help, "help command". - magic collating in scripts FLAGS_ design.html - design goals, portability issues, coding style.... code.html: - how to build it (make defconfig, LDFLAGS=--static make, make install) - "make help" and "cat configure" - build is parallel now, set CPUS=1 to force single build also generated/build.sh - build is incremental now. (Helps "make change" go much faster.) scripts/single.sh builds a single command standalone - disables multiplexer, this breaks some commands, working on it. "make change" builds all commands standalone - including pending!!! be careful. - flow of control from main() to your command_main() scripts/single.sh builds standalone - adding a new command - source code walkthrough (attempt to document the code in detail) - lib/args.c - command line argument parsing - lib/dirtree.c - recursive directory descent code - lib/llist.c - linked list code - lib/lib.c,xwrap.c - general shared infrastructure xfuncs() (xwrap.c) error_exit() instead of returning errors. toys/$MENU/*.c - individual commands - I regression test under glibc, musl, and enh does on bionic. - My Aboriginal Linux has a hacked up uClibc version based on the last-ever