I have a cold. I hate having a cold.
Over the past few days I've surveyed various filesystems to use in the firmware. I want an "archiver" type filesystem, where instead of making a block device of a given length, loopback mounting it, and filling it up with files, I instead point a command at a directory and have it make some kind of archive that can be mounted as a filesystem.
The first of these is romfs, but it doesn't compress, doesn't do the full range of ownership and file permissions, and uses a simple linked list approach that doesn't seem designed to scale to large deeply nested filesystems.
The compressed rom fs is "cramfs", which would be ideal except that it's old. The biggest file it can handle is 16 megabytes, and the maximum starting offset of a file in the filesystem is 256 megabytes. (Plus it has 8 bit gids and 16 bit uids, and various other limitations.) I should offer it as an option, but it's not something I could use to repackage knoppix, for example.
Last time I did something like this, I used the transparent compression cd-rom extensions (zisofs). That still works, and scales up nicely, but it's not that efficient. A quick prototype converting the knoppix cloop image to zisofs bloated it from 725 megabytes to almost a gigabyte.
Knoppix uses cloop, which has a number of problems. The performance sucks (a design issue more than an implementation issue), you have to load it as a module because module arguments tell it which filename to operate on (ouch), and not only am I building a static kernel but I don't know if that would be compatible with losetup -o anyway. Plus it's not integrated into 2.6.9, and apparently the reason it's not integrated is the code is really ugly.
Still, in terms of absolute size cloop is smallest. It's the old tar/gzip vs zip file issue again: compressing each individual file is less efficient than compressing a group of files all at once. That's also where the performance problems come in, of course. (And why you can't loopback mount a gzipped tarball, although you theoretically could do so with a zip file. Not that I have time to write my own filesystem driver to be able to mount a zip file; I've hoped somebody would for years, but no. And no, I'm not playing with userspace filesystem drivers either.)
JFFS2 has compression support, but it's not an archiver style filesystem: I'd have to guess at what size block device I need. I could also track down the old ext2 compression extensions, but again: I want something that's already in the kernel and works like an archiver...
So it looks like zisofs does what I want. I should see if cramfs produces noticeably smaller archives for small filesystems, and give the option to use that if so. In actual embedded devices with a few megabytes of firmware, cramfs would be nice. But for boot-from-cd (or live alongside windows) style desktop distributions, it could be very limiting...
New job definitely impacting my open source hacking time. Oh well, if I can do this for a few years I can retire...
Merged build.sh and make-chroot.sh so that the second is now a "here document" in the first. It was also kind of ugly that I was using a modified busybox tarball, so I upgraded to a new/clean busybox cvs snapshot and am now applying my "sort" patch against that. (Everything else, up to and including the loop.c fix, is checked into CVS by now...)
The gcc build is dumping out lots of errors about unknown usage of busybox "tail". Yet another to-do item...
But first: Whip up a knoppix image using the combined kernel/root image trick, both as a proof of concept and something to run on my laptop. (My laptop is still running a knoppix image I pried off the CD and nailed to the hard drive. Knoppix is more or less the minimal distro that has both a build environment I can compile stuff with and all the the desktop goodies I want: web browser, email, openoffice, xmms, and so on. Eventually I'd like to get Firmware Linux to replace it, but it doesn't even have X11 yet.)
Puttered around making a sources.txt file listing all the websites each package comes from, both to document where they all come from out on the web and so I can make a script that will download all the sources a user needs so I don't have to put up big tarballs.
Right now, every small tweak that changes one file in the source collection means I put up a new 100 megabyte tarball, and keeping a half-dozen old versions around would eat up a disproportionate amount of space. Especially considering that 95% of each tarball would be the same stuff. I wouldn't mind having a CGI on my website that creates each tarball on the fly, but A) I've got to move the site off of Eric Raymond's DSL line to my cable modem before I really try to attract too much attention to the thing, B) it would be nice if people know where to go to get package upgrades anyway.
Took out perl, which I'm not building at present. Aso moved groff, man, and man-pages out to the todo pile, hopefully to be replaced with doclifter and a tiny man shell script that calls lynx. (Gotta get python working first to run doclifter, though. At least as a build tool.)
I suppose I should come up with some way to put the rest of the development tools in /tools, so you can have a finished system with just the bare essentials. New to-do item, as if I had a shortage...
Speaking of which, finally put this log up on the firmware linux web page. Go me.
Added a new feature to lilo, a length option for kernel images to tell it the length (in bytes) of a file. (Goes with the other options like append=, feed it a decimal number.) That way, I can use an initramfs, append a zisofs image to the end of the kernel, and viola: whole os in one file.
Send the patch off to the lilo maintainer, who said he'd "take it under advisement". He's not sure how much demand there is for it, but oh well. It was three lines...
Now I need to get initramfs working (which means port cpio creation to busybox), get a zisofs creation step going, actually whip up some init scripts, work out some kind of install mechanism... And of course get X11 building so it's got a desktop.
Upgraded the version on the website to the new no-coreutils version. Made a note at the top of the firmware linux main page, "I upgraded busybox sort so I could finally dump the last of coreutils. Go me." Decided I should do a more regular log. (Shamelessly using vi, like I did on my old flash.net webpage back in 1997, before the word "blog" was invented.)
Still not integrated yet. Back on the 3rd I asked Eric Andersen if I could check in the new sort and he said he wants to convert the busybox cvs repository to subversion and fork off a new development branch first, which could take a while. Oh well.
Actually built firmware linux with the new sort. (There was a thinko, of course. All my tests supplied input from stdin, actually feeding it a file led to an infinite loop. Fixed, with patch sent to the busybox list...)
Along the way did more work on sort.c. I cleaned up the -n support so it has an integer implementation if you don't declare SORT_BIG. I also took the seperate GNU extensions selector away and just made it all part of SORT_BIG, and moved a lot of other stuff under SORT_BIG so that the tiny version doesn't support -o and such. Actually added a menuconfig entry for it too, and posted it all to the list.
But the point is, coreutils is _toast_. And there was much rejoicing. Yes, I'll need to implement comm when I want to build perl. I want a busybox diff too, so I can dump diffutils. (diff -u, anyway.) Sort has some "read files into memory, line by line" functionality that I should generecize to put into libbb, eventually.
Posted my new sort.c to the busybox list today. I started that months and months ago, before Linucon started eating up my time and put firmware linux on hold. This one should be SuSv3 compliant, with a few GNU extensions. (I need to make more tests before I can really say whether or not it IS SuSv3 compliant. And the real test, of course, would be building gcc and binutils and the rest of firmware linux with it...)
Entries before this are mostly in my old livejournal, although back before that I dredged up a few old entries from 2002.