Rob's Blog (rss feed) (mastodon)

2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 2004 2002


February 3, 2023

Darn it, fseek() is underspecified. If I lseek() on a file descriptor I know what happens, and what error conditions to check for if the fd isn't seekable. But if I fseek() back a few bytes, is it doing an lseek() on the underlying file descriptor or just adjusting the buffer in the FILE * object? If I fseek() on something that isn't seekable does it cause a problem for future reads?

I just fixed head.c, but toysh's read builtin also needs to put back extra data it read for the corresponding test to work right, and lseek(fileno(FILE)) would leave the FILE * readahead buffer with leftover trash in it, so in THEORY I want to do fseek() but in practice I dunno how much I can trust it? (More debris from the C specification people pretending file descriptors don't exist so they don't need to interact with them, and posix refusing to go far enough specifying the interaction.) Honestly, "fseek() shall fail... IF the call to fseek() causes an underlying lseek() and [error happens]" because calling fseek() is by no means guaranteed to cause an actual lseek() to update system status. (Grr, do an fseek() AND lseek(fileno(FILE)) maybe? I'm not convinced this is BETTER than just doing single byte reads of the input so we never get ahead...

Sigh, time to read multiple libc implementations...

Ok, from musl and bionic it LOOKS like fseek() is generally implemented as a wrapper around lseek that flushes and drops the FILE * internal buffer data when the seek works, and the ambivalence about whether not it actualy does that is because fmemopen() and friends exist, so some FILE * objects AREN'T a wrapper around a file descriptor. And those are weird, but I don't have to care about them here.

Ha! If I feed the O_DIRECT flag to pipe(2) then in THEORY that prevents multiple writes from being collated in the pipe buffer, meaning "while true; echo $((++x)); done | while read i; echo $i; done" shouldn't skip any numbers even if it creates and destroys a separate FILE * each time through. (Which it still shouldn't for stdin/out/err, but I need to throw in whatever the read equivalent of a fflush() is each time we redirect stdin.)

Hmmm. There's a gratuitous artificial limitation on fcntl(F_GETFD/F_SETFD) which ONLY lets it change FD_CLOEXEC and NOTHING ELSE. Why even have the API then?

Wow, glibc is truly craptacular. If I go over to my freebsd-13 image and include unistd.h and fcntl.h and do pipe2(fds, O_DIRECT); it works fine. And it works fine built with musl-libc too. In bionic, they have O_DIRECT but not pipe2 because their unistd.h has an inexplicable #ifdef IA_IA_STALLMAN_FTAGH around the prototype. (And I still haven't figured out how to #ifdef for the presence of a function prototype.) But if I do that on glibc it complains about pipe2 _and_ O_DIRECT both failing to be exported from the header files I included without #defining about how RMS sleeps in ryleh. Guys: pipe2() was introduced in 2008 and O_DIRECT has been in Linux for more than 20 years (and grew its pipe2 meaning in Linux 3.4 released May 2012), it is a Linux system call, not a gnu thing.

Linux is not and never has been part of the gnu project, and RMS explicitly objected to the existence of Linux before he switched to trying to take credit for it, and yes his explanation at that link is a big lie because Linux forked off minix not gnu, which is why the early development was all done on comp.os.minix and he had a famous design argument with Minix' creator (when said professor returned from summer break) who kicked him off minix's usenet newsgroup and made him start his own mailing list. I collected some interesting posts from the first couple years on my history mirror: note the COMPLETE lack of Stallman or FSF participation in any of it, and if you boot 0.0.1 under an emulator, the userspace ain't gnu either. Stallman was 100% talking out of his ass: Linux was inspired by (and developed under) Minix with the help of printed SunOS manuals in Torvalds' university library, and it incorporated a bunch of the BSD work going on at the time. The gnu project was one of MANY unix clones happening in the wake of the 1983 Apple vs Franklin decision extending copyright to cover binaries and inspiring AT&T to try to close and commercialize Unix after 15 years of de facto open source development (and the FIRST full Unix clone shipped in 1980) By the time Linux happened, the GNU "project" had been spinning its wheels for eight years. When Linus's 1991 announcement said it WOULDN'T be like gnu, he was MOCKING WIDELY KNOWN VAPORWARE, like a game developer referencing Duke Nukem Forever or Diakatana.

Anyway, the point is the glibc developers have had PLENTY OF TIME to get these symbols into the darn userspace headers, and the only reason they haven't is the same reason Stallman tries to take credit for Linux, which has led to bad blood in both directions. (Stallman also tries to take credit for the existence of FreeBSD, but they just point and laugh at him. He had nothing to do with Wikipedia or project gutenberg either. The term "Freeware" was invented by Andrew Fluegelman years before Stallman's GNU announcement. Magazines like Compute's Gazette had BASIC listings in the back every month dating back to the 1970s. Dude can shut up and sit down aleady, that sexist privileged white male Boomer has Elon Musk levels of taking credit for other people's work going on, and needs to just stop.)

Aha! There's a SECOND fcntl(F_GETFL/F_SETFL) API which CAN toggle O_DIRECT. That's just _sad_, but sure. Assuming I can reliably beat a definition of O_DIRECT out of the headers, which I can't really #ifdef/#define myself because it varies by architecture. But I can get that from everything except glibc, and maybe I just don't care about it working with glibc? There's only so persistently stupid you get to be before I leave you behind. Define it to zero when glibc's broken headers did not provide, and let the call drop out, you get unreliable behavior due to a libc bug. I will not, ever, define stallman because my code is not part of the gnu project. One of its many goals is to provide an antidote to gnu.

Huh, it's surprisingly easy to get derailed into half an hour of closing tabs. Something like a hundred accumulated open terminal windows in desktop 7 (email) which are mostly just "type exit, hit enter" in each one because it's some man page I was looking at or command line tests I an confirm I finished with (or "pulldown->move to another workspace" and send off to desktop 2 (toybox) or 6 (linux/qemu/mkroot, and my kvm instance running freebsd hangs out there too), a bunch of "last thing here was pushing to git" or git show $HASH, or running some simple command like "pkill -f renderer" or df /mnt (shows me what if anything is currently mounted on it) or doing math with $((123*456)), or grepping for a symbol in /usr/include or the output of something like "aptitude search thingy" (an apt-get wrapper with better syntax) where I recognize and can discard the results but switched away from that window once I had my answer. When vi is editing a file exiting out and doing a git diff shows me whether I was browsing or actually made changes.

And lots and LOTS of "vi was editing a file and then got killed" because when you fire up vim on a file that's already being edited, it tells you the PID of the old vim instance but doesn't have an obvious way to just kill the old one and let you inherit the editing session. Instead you have to "kill PID" manually if it's still running (or search around to try to find the tab but good luck with that), then :recover and if the file's changed write it out under a new name to see if the changes are interesting, then rm the temp file and the .file.swp and THEN you can go back and edit it normally. Wheee...) If I'm feeling posh I can even go collate windows that got moved to the proper desktops (you can not only drag and reorganize tabs within a window, on xfce you can drag and drop then between terminal windows. If you haven't got a tab, open a new tab to force the tab bar to show up, then exit the new tab when it's the last one in the window.)

Heh, here's the directory where I was re-ripping some CDs (usb DVD drive still works, cdparanoia still works, most of the CDs are still in the right cases) and hitting them with flac to scp up to my website so I could download them to my phone. (Long ago I had big youtube music playlists, but youtube became 100% useless without paying. Not just two ads between each song, but interrupting longer songs in the middle to play ads. Digging out old CDs and mp3 collections it is...) Pretty sure I can rm *.wav in there, I could zap the .flac files too but eh, I'm not short of space just now. (2 terabyte ssd covers a multitude of sins. Or at least allows them to quietly accumulate.)

Here's the window I download and filed my twitter archives in (both for my original account, which I then deleted, and the backup account Fade made me during all those years i refused to give @jack my phone number, which I still have but haven't posted to even once since making that archive because downloading a fresh archive wants to do 2FA through Fade's phone in Minneapolis which is just not worth it. (I check a couple individual feeds there about as often as I remember to check Charles Stross' blog or Seanan Mcguire's Tumblr. I don't have an account on either site...)

That's the EASY part of tidying one's desktop, of course. Browser tabs have gone beyond the timesink event horizon. Chrome remebering them between restarts is both a blessing and a curse, but at least "pkill -f renderer" keeps the memory usage down to a dull roar. It would be nice if it could save each inactive tab to a zip file under .chrome somewhere so that tab didn't have to reload from the website as if freshly opened whenever I get back to it, but hey. I've learned I basically never look at bookmarks again, and I _do_ periodically revisit and finish/cull old browser tabs. Not as fast as they accumulate, but still...


February 2, 2023

The ice storm has REALLY screwed up my sleep schedule. Woozy. (Couldn't work, couldn't go out, the lights were off all day, and it was stressful.) My internal clock is flashing 12, doing the whole "Too tired to focus but lying down does not result in sleep" thing...

It's hard for me to get worked up about "yoda conditions" when it's THE SAME COMPARISON. 1 == x and x == 1 are equivalent, but the one on the left can't be mistaken for/typoed into an assignment. "Correcting" everything to the one on the right because it's not "mentally comfortable" is something I'm having trouble sympathizing with? (My mental arithmetic apparently does not have "handedness". This is a thing the language has always allowed you to do, and there is a reason to do it and zero reason to do the other one. Arguing "but it's not a _strong_ reason to do it" vs having literally zero reason other than aesthetic preference... Sigh.)

Darn it, my clever "while read" combo hack in toysh has a problem.

So getline is a glacial CPU-eating slog without cacheing, and FILE * is the cacheing layer ANSI C decided to provide back in the day and (eventually) implement getline() on top of, and if you're just reading from stdin then the "read" builtin can use the stdin global constant (as get_next_line() is curently doing), and my THEORY was that for anything else (either read -u or read < source) I could fdopen() a FILE * object and cache it in the struct sh_blockstack instance for the enclosing loop (adding a field to the control flow data), and thus not lose cached readahead buffer data by destroying and recreating the FILE * wrapper each time the read command ran and exited.

BUT: read -u $VARIABLE is not guaranteed to be the SAME filehandle each time through the loop. I guess I can call fileno() on the FILE * and compare the fd we're trying to operate on, and tear down the old one and replace it when they change it?

while read i -u 37; do for in {1..10}; do read j k l -u 37; do echo $i $j $k $l; done; done; done

I can come up with a bunch of test cases I don't care about OPTIMIZING, but I'd prefer they didn't actively break. (But why would anyone do that? "for i in a b c d; do read a b c < $i; do_stuff; done" could happen. Hmmm, but then it's doing an open/close on the file object in the read context, so cacheing the FILE * object in the flow control would be wrong. Grrr. Lifetime rules!)

Hmmm... alright, there are two cases here: read from a tty and read from a file. In the tty case, the input (should) come in chunked so the block reads are short and shouldn't readahead much anyway. (If you've ever typed stuff before a shell was ready and the input got lost... that. Password prompts are notorious for it, but it happens elsewhere.)

The other case is "while read... < file.txt" where it will very much read all the way ahead, and if you ever discard extra buffer you deterministically lose bits of the file. Which says (oh goddess) I need a reference counted cache of FILE * wrappers for file descriptors >2 (stdin, stdout, stderr have persistent globals) but bump the reference increment/decrement to the enclosing loop block object (if any), which STILL won't work with "while read x; do command that also reads input $x; done < file.txt" because the FILE * will read ahead and then pass the filehandle to the command which starts reading after whatever the FILE * ate.

$ while read i; do echo =$i; head -n 1; done <<< $'one\ntwo\nthree\nfour\nfive'
=one
two
=three
four
=five

How. HOW? Is it doing single byte reads from input?

$ echo -e 'one\ntwo\nthree\nfour\nfive' | while read i; do echo =$i; head -n 1; done
=one
two

Ah. It gets it right when the input is seekable. Of course.

$ while read i; do echo =$i; toybox head -n 1; done <<< $'one\ntwo\nthree\nfour\nfive'
=one
two

And it's at least partly "head" doing extra work, and toybox is getting it wrong. (New test!)

RIGHT.

And this says that FILE * is generically borked in the presence of fork/exec _anyway_, because the inheritor of our fd 0 won't see the data read ahead into the FILE *stdin buffer. I'm more familiar with this problem as it relates to stdout flushing, because glibc's gotten that very wrong before, and that was just trying to make flush on exit() reliable, let alone exec without exit.

The two big problems in computer science REMAIN naming things, cache invalidation, and off by one errors.


February 1, 2023

For my birthday, an ice storm knocked out the power from before I woke up in the morning until sometime after 10pm. I had some battery in my laptop, but didn't fire it up because if it drains all the way I lose all my open windows, and with more freezing rain predicted tonight I didn't know if power would be restored before thursday. (Plus our geriatric cat's heating pad was off, so sat on me for hours instead.)

Luckily it got just cold enough to sleet instead of more freezing rain. None of the trees that could have collapsed on my house did, although two on the block dropped some quite big chunks, and one such trees has drooped significantly and is resting half its branches on our roof, but in a bend-not-break sort of way. (One around the corner has bent basically in half and is resting its branches on the _ground_, which I find impressive. Pecans are survivors.)

So yeah, not a productive day, but way better than it could have been. No flood damage, no hurricane scouring the paint off a corner of the house...

Sigh. The very nice glasses I got in Japan shortly before the pandemic are finally wearing out. The lenses were outright scratchproof for a good three years, but the coating's weathered enough they're starting to scratch. They've been WAY more durable than anything I got from Zenni, and I dunno whether they're still functional at all with that whole "outsource to china" strategy meets china's covid lockdowns, the container pileup, and now wolf warrior diplomacy and reshoring? (I didn't get my prescription checked in Japan and instead handed them an old pair of glasses to copy the prescription from, and I've passed them off as "reading glasses" ever since. That was intentional: I'm not driving so I care more about reading up close for long periods, and glasses that focus more naturally at that length cause less eyestrain.

I _have_ newer/stronger glasses somewhere, but about 5 years ago I worked out that my eyes are adjusting to my normal usage patterns (staring at up-close things for hours at a time), and the whole reason my vision sucks is years of a correct-and-adapt cycle I probably could have just avoided if I hadn't been reading comic books all morning before the school eye test back on Kwaj. I'd never needed glasses before, but the roofline was a touch blurry... because my eyes took a couple hours to swing back to looking at far away stuff. I'm a lot older so it takes my eyes a lot longer to move their overton window, but even today it still happens: if I stop wearing glasses for 8 hours or so far away things are WAY sharper when I finally do put them back on. I just... hardly ever do that? No phone, no lights, no motorcars, not a single luxury... Sometimes I take them off on long walks to the table while listening to podcasts, but that's about it.)


January 31, 2023

Honestly, WHY does qemu keep gratuitously changing its user interfaces? Once again the old one was simple and straightforward, the new one is insane, and removing the old simple API serves no obvious purpose. They broke tcp forwarding, they broke -hda, they broke -bootp... Stoppit.


January 30, 2023

It occurs to me I can test the lib/passwd.c rewrite under a debootstrap chroot instead of waiting for mkroot, because it's just twiddling files rather than poking at syscalls or /proc the way route and insmod do to actually change the host kernel's system state.

In theory, it's "debootstrap beowulf beowulf" (for devuan anyway) and then when that's finished copy a stripped down version of mkroot's "init" script in there and sudo env -i USER=root TERM=linux SHELL=/bin/bash LANG=$LANG PATH=/bin:/sbin:/usr/bin:/usr/sbin unshare -Cimnpuf chroot beowulf /init and... in PRACTICE it's being stroppy. I dealt with this for Jeff some months back, but apparently didn't blog about it enough, and can't find my notes? Hmmm... I remember tracking down a weird bug involving accidentally running the Defective Annoying SHell instead of bash, hence the SHELL= export there, and that's the kind of thing I WOULD have blogged about, but no?

I might have tweeted about it, in which case it's lost to history because of the muskrat's midlife crisis. (For his quarter life crisis he bought a company that makes shiny red sports cars. The bald Amazon billionaire bought the a newspaper, the south african emerald brat tried to pretend he wasn't copying him by instead buying the latest iteration of aol/livejournal/myspace. Because SpaceX clearly isn't in a dick measuring contest with Blue Origin. A company named after the X-prize, which he lost -- Paul Allen sponsored Burt Rutan to win -- is clearly NOT about competition and ego, it's an entirely original thing that emerged fully formed from his very large brain, which is no way a cry for help.)


January 29, 2023

Alright, FIX WHAT'S THERE in dirtree. BREADTH traversal means dirtree_recurse() needs to iterate through the child list of stored entries (if any), which calls handle_callback() which frees the node when the callback didn't return DIRTREE_SAVE. The problem is, we're recursing through that list and free(node) doesn't remove it from the list. We're only told AFTERWARDS whether or not it saved it (did handle_callback return a pointer or NULL). So I need to fetch the next entry _before_ calling handle_callback so we can iterate without read-after-free list traversal, but I need to update and advance the saved-node pointer _after_ calling handle_callback, making sure it always points to valid memory.

Dear C++ developers who have hijacked gcc development:

In file included from ./toys.h:69,
                 from lib/dirtree.c:6:
lib/dirtree.c: In function 'dirtree_recurse':
./lib/lib.h:71:35: error: label 'done' used but not defined
 #define DIRTREE_ABORTVAL ((struct dirtree *)1)
                                   ^~~~~~~
lib/dirtree.c:174:21: note: in expansion of macro 'DIRTREE_ABORTVAL'
     else if (new == DIRTREE_ABORTVAL) goto done;
                     ^~~~~~~~~~~~~~~~
lib/dirtree.c:154:18: warning: unused variable 'entry' [-Wunused-variable]
   struct dirent *entry;

Bravo on the warning and error message generation. Exactly what I would expect from people who think C++ is a good idea. (And yes, that is a single processor build with no output interleaving. I double-checked. And yes, those were the first output messages before it had a chance to get itself good and confused, which it did and complained just as uselessly for quite a while after that. For the record, I had an extra } on line 177, a few lines AFTER all that nonsense. The compiler was no help whatsoever in finding it.)

Ok, got sort checked in. It uses -s as its short option which is a bit questionable (as far as I can tell the gnu/dammit one has -s produce the behavior it was already _doing_ for extract and throws an error if you try to use it with create: bravo guys), and my --sort can take an optional =thingy argument for compatibility but only implements sort by name. (Again, there's no "rm -r --switch-off-r" so --sort=none seems useless, and --sort=inode is a micro-optimization for 1980s vax systems without disk cache? It claims a performancce improvement but extract ain't gonna care (it's not USING the old inodes) and create has to read all the directory entries in order and then do a second pass to open them when it sorts ANYTHING, and then using inode number as a proxy for disk layout is optimizing seek time on uncached spinning disks which is also assuming they're regularly defragmented in a way that doesn't get the file locations out of sync with the inodes AND which assumes the disk was basically empty when all the files were created so the on-disk file locations correspond to the inode numbers, AND assumes a filesystem that's allocating inodes sequentially instead of using them as hash values... seriously, this was a marginal idea in 1989, trying to do it on a VM using virtfs to talk to a host storing data in btrfs is just NONSENSE.

The request was just for generating stable tarballs. I'm a little "eh" about mine vs gnu/dammit producing different output because I'm using strcmp() and the FSF loons are probably listening to the locale information and doing the same "upper case sorts mixed in with lowercase" nonsense that forces everybody to go LC_ALL=c before calling 'sort' out of the host path, but I can't control that and "stable produced with the same tool" is presumably the goal here.

Yes, the test I added for --sort is not using "professional" names. No, I'm not cleaning it up to look presentable. Possibly I should have left sed as it was and let the culture catch back up...


January 28, 2023

Grrr, the design of dirtree.c isn't right. And I've known it isn't right, but it's hard to GET right. There are FOUR interlocking functions (dirtree_add_node(), dirtree_recurse(), dirtree_handle_callback()), plus a fourth wrapper function dirtree_read() you generally start out by calling, and that's way too complicated.

The job of dirtree_add_node() is to stat a directory entry and populate a struct dirtree instance from it, which is fine. That's good granularity. That's the only one of the lot that ISN'T crazy, although possibly that assumption is what needs to change to let me fix everything...

When each dirtree instance gets created a callback function can happen, with behavior that happens in response to that callback's return code. That's what dirtree_handle_callback() does: you feed it a dirtree instance and the callback function, and it calls one on the other and responds to its return code. Possibly dirtree_add_node() could just take the callback as another argument... except what I was trying to avoid was recursing into subdirectories causing the function to recurse too. I don't want NOMMU systems with tiny unexpandable stacks to have unnecessarily limited directory traversal depth. Although I don't think I've got that right NOW either, so...

The dirtree_recurse() function handles recursion into subdirectories. Badly. Right now it opens a filehandle at each level to use the openat() family of functions, meaning directory traversal depth is limited by number of filehandles a process can open simultaneously. Instead I need to traverse ".." from the directory I'm in to get back to the parent directory, and then compare the saved dev/ino pair in the cached stat structure to see if that's the same node, and if not traverse back down from the top again. (And if THAT doesn't work, prune the traversal. That's "mv a subdir while archiving" levels of Don't Do That. SECDED memory falls back to DETECTING an error it can't correct, quite possibly this is xexit() time.)

The linked list of dirtree structures is less of a problem than the recursion stack depth because a linked list doesn't have to be contiguous, you can fragment that allocation all you want.

Sigh, the real outlier here is ls.c. Everything else just calls dirtree_flagread() and gets callbacks, but ls micromanages the traversal because it had weird sequencing requirements. So I need to refamiliarize myself with the ls weirdness to make sure a new cleaner dirtree implemenation could provide the callbacks it needs (quite possibly it _is_ the new DIRTREE_BREADTH semantics) so I can stop exporting dirtree_recurse().

Grrr, but Elliott pinged me about a new android code freeze and I wanna get him --sort before that goes in. I should debug what's THERE instead of redesigning it, but it's REALLY hard to get the object lifetimes right with multiple functions passing stuff off between them in a loop like it is now.

I think I need two functions: dirtree_add_node() and dirtree_read() that does all the callback handling by non-recursively traversing the tree (adding/removing nodes as it goes if/when the callback says to). Hmmm, but what would the arguments be? There isn't a global "tree" object that can hold things like "flags", and I want to be able to traverse on a path _or_ under an existing struct dirtree *node... Maybe dirtree_read(char *path, int flags, function *callback) which is a wrapper for dirtree_traverse(dirtree_add_node(char *name, int flags), int flags, function *callback)... except the reason dirtree_add_node() needs the parent pointer is for parentfd due to the openat() stuff, that's why the caller can't just set it after it returns. Right...

Fiddly. Hmmm...

When I'm done all this plumbing SHOULD look so simple that it's all obvious and trivial and seems like I didn't do anything. Getting there is usually a flaming pain, and a lot of the times I DON'T and have to ship something overcomplicated, which says to ME that I'm not very good at this. Alas, the reason I _don't_ have impostor syndrome is the rest of the industry turns out, on average, to be even worse at it than me.


January 27, 2023

Trying to debug tar --sort and it's being stroppy. I'm not sure I've got the design right, which is sad for something so seemingly simple?

Sort of regretting having implemented --no-ignore-case. It's the default, just don't specify it when you don't mean it? I didn't have sort check it, and am going "eh...". (The extra code to check it is bad. Having it and NOT checking it here is bad. Grrr. NOT PICKING AT IT. I haven't figured out how to make lib/args.c gracefully handle this category and I'm trying NOT to go down a rathole of spending 3 days on the design of something relatively unimportant. Not a fan of ---longopts at the best of times, having extra options to put the behavior BACK to the default... rm -r does not have a turn-off-r-again option because it DOES NOT NEED TO.

The gnu/dammit clowns are BAD AT UNIX, Stallman only cloned unix after ITS died because his community had collapsed under him and he wanted to hijack an existing userbase, he hated and fought unix until he was forced by circumstance to join, and was an outsider who never properly understood WHY it worked.

The old history writeup I did on this years ago didn't even MENTION Digital Equipment Corporation's Project Jupiter which was the proposed successor to their 6-bit mainframes (the PDP-6 and PDP-10). The Jupiter prototype system was used to render part of the graphics in the 1982 disney movie Tron, but DEC pulled the plug on development in April 1983, and THAT's what caused Stallman to give up on ITS and start over cloning Unix. He'd backed the wrong horse, the hardware platform he'd inherited (after everybody else who worked on it graduated and moved on with their lives, he stuck around as a perpetual college student) died out from under it, and NOBODY ELSE CARED. He was forced to move because the univesity was going to unplug the old hardware and throw it away. This wasn't a decision, this was a forced REACTION. RMS was always a conservative reactionary working to prevent change, who took the smallest steps possible each time the legacy position he defended became untenable. As with all ultra-conservatives, he mistakes this for "visionary thinking" and talks himself up, but it's the same "looking back to a largely imaginary golden age" you see so much of from any other privileged old fogey complaining about kids these days.

Stallman couldn't even predict the obvious near future: 6 bit systems inevitably lost to 8 bit systems as memory got cheaper because the whole POINT had been that you could fit 25% more text a given amount of memory using 6 bits per symbol instead of 8... with glaringly obvious limitations. With only 64 combinations you just couldn't fit everything: 26 upper chase characters, 26 lower case characters, and 10 digits left only TWO symbols for space and newline -- you couldn't even end sentences with a period. If you wanted ANY puncutation, you had to sacrifice digits, or make everything uppercase, and different compromises meant incompatible encodings.

The first 7 bit ASCII standard was published in 1964. With twice as many symbols there was no need to compromise -- after upper, lower, and digits half the space was still available for punctuation and control characters -- so every 8-bit system could use a compatible encoding for all documents. Gordon Moore's article describing Moore's Law was published in 1965, predicting exponential increases in memory availability for the forseeable future. Clinging to a 6-bit system almost 20 years later (after all his classmates had already abandoned it) was head-in-the-sand levels of stubbornness on Stallman's part.

DEC had introduced its first system with 8-bit bytes (the 16-bit PDP-11) in 1970, 13 years before canceling Jupiter, and its 32-bit successor the VAX came out in 1977. In DEC's entire history it only ever sold about 700 of its 36-bit PDP-10 mainframe systems. DEC sold almost a _thousand_ times as many PDP-11, and DEC shipped a dual-processor VAX the year before canceling Jupiter.

Stallman is the exact opposite of "visionary". He's just another classically educated white male with decades of practice retroactively justifying what he's already decided to do by constructing a convincing shell of logic around his emotional motivations, and it is just as exhausting dealing with his fanboys as it is dealing with the fanboys of muskrat or jordache peterman or the ex-Resident or any of the others.

Jeff's flying back to Japan. I am jealous. But Fade made a flight reservation for me to visit her from Feb 10 to 22, so that's nice. (Her dorm apartment thingy still has the second room empty and locked, so it doesn't both anybody if I stay more than a couple days.)


January 26, 2023

Last year I ordered a cthulamp for the desk in the bedroom (one of them "five positionable metal tentacles with a lampshade at the end of each" deals), but couldn't figure out how to assemble it properly and then wound up flying off to Fade's and finishing the contract from there. Took another stab at assembling it today and figured out what I got wrong this time (the little plastic not-washer thing with the raised inner bit was both on the wrong side of the shade AND rotated 180 degrees, so it fit perfectly but then the light bulb didn't), and WOW that desk is a nicer workspace with 5 more LED bulbs right next to it.

Finished and checked in --wildcards. Needs more tests in the test suite, but it didn't cause obvious regressions and should be enough to unblock the android kernel guys?

Implementing tar --sort next.

I tried Chloe Ting's "5 minute warmup" video.

Made it to the end this time.

Everything hurts.

(It wasn't even one of her proper EXERCISE videos. I did the WARMUP and am still in pain an hour later. It turns out slowly walking 4 miles a night 3 or 4 times a week not exercise a wide variety of muscle groups.)


January 25, 2023

Elliott emailed me asking for a bug report if I could reproduce the adb compatibility issue, because he says the policy is the developer kit should be backwards compatible all the way back to kit kat, including ADB working. I apologized and acknowledged it's been a while since I've tried the distro version of ADB. (For file transfer I scp files to my webserver so my phone can download them, and attach stuff to myself in slack going the other way. I installed an ssh app on my phone but haven't bothered to use it in forever.

Back when I was running Devuan Ascii, _many_ things out of the repo didn't work (llvm was too old for the packages I was trying to build, ninja was too old, I finally upgraded to Beowulf because building qemu from source demanded a newer vesion of python 3...) The adb in Ascii having been broken probably wasn't surprising. I got in the habit of downloading a new version of the android tools rather than trying the distro version, and haven't checked if I still NEED to in a while...

My current phone's a Pixel 3a that end-of-lifed on Android 12 (the system->update menu has a big "regular updates have ended for this device" banner, with the last one 10 months ago), so isn't exactly a moving target anymore anyway. (Yeah, I should upgrade my laptop to Devuan Chimaera, but nothing major's broken yet that I've noticed?)

At a guess, debian breaking adb is like debian breaking qemu: I always build that from source because debian's distro version never works. Even when the theoretically exact same release built from source via "./configure; make; make install" works fine.

Alright, where did I leave off with wildcards: --wildcards{-no,}{-match-slash,} --{no-,}anchored --{no-,}ignore-case and this is why I got so distracted by trying to automate no- prefix in the plumbing. Right, just explicitly spell out all 8 flags for now and clean it up later. What are the USERS: Inclusion vs exclusion, creation vs extraction, command line arguments vs recursively encountered arguments: that's 8 combinations. No, 16 with and without case sensitivity. (This is assuming extract and test behave the same.) Each of those can have wildcards default to enabled or disabled: case sensitivity is the global default, exclusion defaults to wildcards no-anchored match-slash. Not everything can be enabled in every position, for example --wildcards does not affect command line arguments when creating an archive. (That's one of the tests I wrote back in October.)

I'm also annoyed at --show-transformed-names and --show-stored-names because it should just pick one. I'm also reminded that --verbtim-files-from exists and I think that's what I'm doing already? (Need to compare with busybox...)

Sigh, it's so easy to find -K and -N and go "I could implement that" but nobody's ASKED for it and if you go down that road even ignoring crap like -n (not implementing multiple codepaths to do the same thing, thanks) and --sparse-version there's gratuitous complication like --owner-map (not the same as --group-map) and the $TAPE environment variable and twelve --exclude variants that really could be done via "find" ("find -print0 | xargs -0" covers a multitude of sins, fairly portably) and then just nuts stuff like --hard-dereference that... what's the alternative? Linux doesn't let you hardlink directories, and a file with more than one hardlink is A FILE. Would --ignore-command-error apply to the compressor or just programmatic output streams?

Busybox NOT implementing stuff for a long time is a useful data point: they got a couple decades of people poking them and going "I need this". If it didn't happen (strongly enough for them to react), that's informative.

Except I got asked (on github somewhere) to support appending: -r and -u and maybe -A? (Which is append with existing archive which you don't need tar for...? I mean, it cuts off the trailing NUL blocks I guess. There's an -i option which... I don't know why that always being on would be a bad thing? Probably some historical reason...)

The existencce of "lzip", "lzop", and "lzma" makes me tired. None of which are "xz". (It's like being back in the days of arj and zoo.)

Ahem: ok, back up to the motivating use case: tar --directory={intermediates_dir} --wildcards --xform='s#^.+/##x' -xf {base_modules_archive} '*.ko'

Oh yes, and with gnu/dammit tar --wildcards affects patterns AFTER it but not before it in the command line. Sequencing! Right.

Ok, wildcards can be switched on for extract but NOT for create because creation isn't doing a directory search but is opening (and recursing into) specific command line thingies so there's no comparison being done: there's no readdir() in that codepath, the open(argv[x]) either succeeds or fails. Comparisons are done for creation exclusion (while recursing?), extraction inclusion, extraction exclusion... which corresponds to toybox tar's 3 existing calls to filter() with add_to_tar() calling filter(TT.excl), and then unpack_tar() doing both filter(TT.incl) and then filter(TT.excl). Both TT.excl calls should default to --no-anchor --wildcards-match-slash but the TT.incl call shouldn't (but currently does because I only implemented one filter behavior). The man page implies incl should default to --anchored --no-wildcards --no-wildcards-match-slash...

Sigh, I can just compare my argument with the global variable to distinguish the two cases, and set the default that was. It's ugly, but having the caller (redundantly!) specify the defaults is also ugly, and having an extra agument to distinguish the modes when I can just test for it... Wanna get this finished and move on to the next thing.


January 24, 2023

It's been a while since I've had a significant visual migrane.

The experience is not raising any positive nostalgia.

Not a productive evening.


January 23, 2023

Checked in [LINK] the probably correct but not actually tested DIRTREE_BREADTH code (which at least didn't cause regressions in the test suite) this morning, but haven't used it to implement tar --sort yet because I still have 2/3 of --wildcards in my tree. Which is actually a half-dozen options because there's --no-wildcards-match-slash and so on.

Urgh, why is tar.c not using FNM_LEADING_DIR instead of the constant? I did not leave myself a comment about WHICH build environment barfed on this. The fnmatch.h header is in posix but this particular constant isn't, It's unsurprisingly in glibc, it's in bionic (which says it got it from openbsd), it's in musl. Boot up freebsd-13 under kvm... that's got it too. And Zach got me a mac login... it's there as well.

Ok, is it a 7 year time horizon thing? The date on the line according to git annotate is 4 years ago, so most likely 7 years has expired by now if that was the case? (It's not a kernel thing, it's a libc thing. Annotate on musl's fnmatch.h says it's from 2011, that's a full dozen years ago.) Eh, try constant for the macro and see who complains...

Oh wow. It's glibc that complains. It wants #define ALL_HAIL_STALLMAN to provide the constants, but on Bionic and FreeBSD and MacOS they're just there without magic #defines. And it's the same constant value everywhere. Right, #ifndef in portability.h time, maybe posix will catch up somewhere around 2040...

Yay, dreamhost fixed it. My two posts about it to the list didn't wind up in the web archive and I was all ready to take up my sword again... but it's because I sent the message and the reply to "lists@landley.net" which is not a real address. Hopefully google and archive.org will start populating again at some point.


January 22, 2023

That tar --xform test failure which only happens on musl is because musl still doesn't have regexec(REG_STARTEND). So it's just a new manifestation of a known failure, eating another round of debugging time because 10 years ago Rich explicitly refused to implement something even the BSDs have.

Sigh. I'm eventually either going to have to fork musl or drop support for it. I should just switch that date test back on. There are multiple "yup, musl and musl only is broken, this even works on BSD" cases already. The test suite needs a MUSL_IS_BROKEN flag on tests, or something...

A tech writer recently boggled at the pointless "undefined behavior" in C compilers written by C++ developers. And here's a rant I edited out of a post to lkml:

The C language is simple. The programs you write aren't, but the LANGUAGE is. C combines the flexibility of assembly language with the power of assembly language: it's basically a portable assembly language, with just enough abstraction between the programmer and what the hardware is actually doing that porting from x86 to arm isn't a complete rewrite. You manually allocate and free all resources (memory, files, mappings) and all sort of stuff like endianness, alignment, and word size is directly whatever the hardware does. In C, single stepping through the resulting assembly and matching it up with what your code does isn't that unusual. I've gone looking at /proc/self/maps on a sigstop'd binary and objdump -d on the elf executable to figure out where it got to, and in C you _can_ do that.

C++... isn't that. The language is DESIGNED to hide implementation details, all that stuff about encapsulation and get/set methods and private and protected and friend and so on is about hiding stuff from the programmer. Then when implementation details leak through anyway, try to fix everything by adding more layers (ala "boost") on top of a broken base, but that's like adding floors to a skyscraper to escape a cracked foundation. It's still static typing with static allocation i(they're insanely proud of tying stuff to local variable lifetimes and claiming that's somehow to garbage collection) and it's GOING to leak implementation details left and right, so they have buckets of magic "don't do that" prohibitions which they cargo cult program off of. Most of C++ is learning what NOT to do with it.

C was simple, so C++ developers hijacked compiler development and have worked very hard for the past 15 years to fill C with hidden land mines so it can't be obviously better than C++.

C is a good language for what it does. C++ is a terrible language. The C++ developers have worked tirelessly to make C and C++ smell identical, and as a result there's a big push to replace BOTH with Rust/Go/Swift and throw the C baby out with the C++ bathwater.

Haven't heard back from dreamhost, so I've submitted ANOTHER support request:

http://lists.landley.net/robots.txt prevents Google from indexing http://lists.landley.net/pipermail/toybox-landley.net/

I did not put http://lists.landley.net/robots.txt there and cannot delete it.

The contents of http://lists.landley.net/robots.txt are:

User-agent: *
Disallow: /

Would you please delete this file, or change it to allow Google to index the site? I do not have access to it.

Here's hoping THAT is explicit enough for them to actually do something about it. Third time's the charm?


January 21, 2023

Properly reported the qemu-mips breakage. That list may be corporate, but it's not the wretched hive of scum and villainy linux-kernel's turned into, so maybe... (Yay, there is a patch, and it Worked For Me.)

So what DIRTREE_BREADTH _should_ look like is something like...

  1. The initial callback (which always happens) returns BREADTH, and the calling function populates the ->child list one level down the same way DIRTREE_SAVE would.
  2. The second callback has ->again set to DIRTREE_BREADTH, which lets you sort the children. When this one returns, it recurses into those children unless you returned DIRTREE_ABORT. This recursion frees each child if its initial callback didn't return DIRTREE_SAVE.
  3. The DIRTREE_AGAIN callback is handled normally, although the children were already freed if not SAVEd.

Hmmm, instead of checking for DIRTREE_BREADTH a lot the "populate children" loop should just pass a NULL callback while accumulating children... Sigh, I need to stress test DIRTREE_ABORT to make sure A) it returns from anywhere, B) it doesn't leak memory. Except most of my actual users don't choose the abort path, they continue on despite errors: tar, rm, cp...


January 20, 2023

We have a dishwasher again! Exact same type as last time, so it looks like nothing has changed but so much work went into this. (Ah, that old story.) The install guy set it doing an empty pratice run first, but then we have so many dishes to wash...

Jeff is trying to set up an sh4 development environment so he can come up with mmu patches and send them to linux-kernel, and I've been feeding him the trail of breacrumbs I've laid out with mdm-buildall and mkroot and so on, even using my prebuilt binary system image tarball the network didn't work for him, and that's becaue I'm using an older qemu version than he is.

Building QEMU from source recently broke network support for all platforms by splitting it out into a separate package your distro has to install for you. Because obviously the ability to talk to the network is not a standard thing a VM would want to do. This now requires "libslirb". There's an existing slirp package, for the serial line internet protocols slip and ppp, which has nothing to do with libslirp that I can tell. Luckily devuan has a "beowulf-backports" repository alongside all the others, which I can add (why didn't the OS install do that?) to get this libslirp-dev package. I'm still annoyed the IBM mainframe guys who took over QEMU development when kvm displaced xen as Linux's standard "hypervisor" are suddenly demanding it, but at least I can get Jeff unblocked now.

Mainframe punched card culture should not be allowed to turn functional software into bloated "enterprise" crap: qemu-system-arm64 (ahem, I mean qemu-system-aarrcchh6644) is A HUNDRED AND TWENTY FIVE MEGABYTES. Dynamically linked! That can't be right. You can tell Fabrice Bellard moved on long ago, and was replaced by a committee.

And test_mkroot.sh says mips is still broken... because the ethernet hardware isn't binding even WITH the library installled. And that's... because an endianness "fix" broke big endian for pretty much the entire PCI bus. Sigh. Vent about it all and move on...

Ok, tangent du jour beaten back down, let's circle back to the toybox design issue I'm frowning at. What notes did I leave myself:

why are recurse and handle_callback split?
  dirtree_add_node(): clear design, yay
    - maybe add callback as argument to dirtree_add_node()?
  dirtree_handle_callback: 

stages:
  fetch dir, initial callback: returns DIRTREE_BREADTH
    fetch children, via recurse with BREADTH.
      problem: closed fd already? (don't close for BREADTH)
    breadth callback: returns DIRTREE_RECURSE
      traverse children now
        call handle_callback on each?

Which means: DIRTREE_BREADTH isn't that hard to implement, but the existing code has three functions that really seem like they shouldn't be split that way?

One sharp edge is that handle_callback() is opening the dirfd for recurse, but then recurse is closing it, which is NOT a happy lifetime rule.

I think the reason for all this tangle in the first place is I was trying to recurse the data structure without making the FUNCTIONS recurse, so it didn't eat an unbounded amount of stack when descending into a tree of unbounded depth? (Especially nasty on nommu.) Except that pretty much means having all three of them be a single function, because otherwise they're calling back and forth between each other. Or having one function that calls the others in a loop, which isn't what it's currently doing.

In any case, "implement breadth first search" and "reorganize this to not be designed wrong" really need to be two different passes, otherwise I'm here for a while...


January 19, 2023

Ha! The dirtree.c plumbing shouldn't have seperate DTA_BLAH flags for the "again" field to distinguish different types of callbacks, it should reuse the existing DIRTREE_COMEAGAIN, DIRTREE_STATLESS, and DIRTREE_BREADTH bits. (The "again" field is a char so can only hold the first flags, but I can reorder the DIRTREE flag list as necessary so the ones that cause callbacks are all at the start. Nobody else cares which flag is which, that's why there's macros.) This way, the again bits are the same as the reason for the callback: no flags is the initial "we found and populated a struct" callback you always get when callback isn't NULL, then BREADTH is "finished populating a directory with implicit DIRTREE_SAVE but did not descend into it yet, so now would be a good time to sort the children", and then COMEAGAIN call would be the final call on the way out of the directory after handling all children. (STATLESS doesn't cause a seperate callback, but is set on any callback when stat isn't valid.)

I should rename DIRTREE_COMEAGAIN to just DIRTREE_AGAIN (it was a Simpsons reference), but my tree's too dirty for comfort, need to check other stuff in first.

For BREADTH child callbacks are deferred until traversal: if the initial no-flags callback on the directory returns DIRTREE_BREADTH the plumbing should populate all the child structures without making any callbacks on them yet, then it does a callback on the same dir again with DIRTREE_BREADTH, then traverses the child list doing normal callbacks but freeing all the non-dir children after each callback returns, and then traverses the now-shortened list again handling the directories it needs to descend into...

Hmmm, that's not what gnu/dammit tar is doing, though. It's populating and sorting the list, then traversing it but descending into each directory as it encounters it in the travesal. Which isn't a true breadth-first search, it has ELEMENTS of breadth-first but... Ok, the return codes from the callback functions need to control order. Maybe if the DIRTREE_BREADTH callback returns DIRTREE_RECURSE then we descend into it now, and if not we do the second pass thing? Hmmm. I've got DIRTREE_SAVE, DIRTREE_RECURSE, and DIRTREE_BREADTH, and can return a chord of any of them to mean what I need it to, the question is what's the most obvious way to signal what I need it to do? What ARE the use cases?

This needs some pacing and staring into the distance....


January 18, 2023

Sitting at HEB with a stack of beverages I just bought (refill on blueberry energy cylinders, the checkerboard teas are back in stock, and there was a good coconut water coupon today)... but no snacks.

I miss japanese grocery stores and conbini. The conbini of course had rice balls and steamed buns and even microwaveable hamburgers if you wanted to get serious. The grocery store near the office had lovely little 100 yen sandwiches, which were just two pieces of cheap white bread with some filling (I usually got the strawberry jam or tuna varieties), crimped in some variant of a panini press that cut off the crusts and sealed the edges, and then presumably run through a nuclear reactor to sterilize them so it has multi-week shelf life. (Like mythbusters did to sterilize those tortilla chips in the "double dipping" episode: conveyor built moves the product past a strong radiation source is basically a non-heating microwave that kills all the bacteria with a few seconds of intense gamma radiation. The expiration date on the package is when the sandwich dries out slightly and is less tasty, I never had one actually go bad.) We could totally do that here in the states, we just don't: some variant of laws, culture, inclination, and capitalism optimizing for profit over all else.

Ok, tar --sort needs DIRTREE_BREADTH to do breadth first search. I could instead do DIRTREE_SAVE to populate the whole tree up front, then sort the whole tree, and then traverse the resulting whole tree, but don't want to because A) directories changing out from under us are less icky if you do it all in one pass, B) I've already got the openat() directory-specific filehandles for local access (I can open "file in this directory") in that initial pass. A second traversal has to either re-establish the openat() filehandles, or create/open big/long/path and potentially hit PATH_MAX issues. Since I don't have existing plumbing to do either of those yet, as long as I have to to write new plumbing ANYWAY I might as well implement the DIRTREE_BREADTH stuff I have some existing design stubs for.

DIRTREE_BREADTH brings up the DIRTREE_COMEAGAIN callback semantics: to enforce a specific traversal order I need to sort each directory's contents before descending into it. I reserved a DIRTREE_BREADTH flag back at the start but never implemented it, and I now have _three_ users of this plumbing that I'm aware of (ls, find, tar) so sounds like time to implement it. (Whether or not I poke ls.c with a stick afterwards remains an open question.)

Looking at find -depth is.. sigh. The toybox find help text describes -depth as "ignore contents of dir" and the debian man page describes -depth as "Process each directory's contents before the directory itself" and I don't remember if posix even has -depth and I probably need to spend an hour or two on this rathole, but I haven't got spare cycles for right now. (And I've already REVIEWED this one multiple times, so 99% likely I wouldn't be fixing the code but just updating my memory of it.) Anyway, -depth existing implies that _without_ that it's doing a breadth first search... which it demonstrably isn't in simple testing. Ok, find is NOT doing breadth first search. I thought it had an option for this, but no. It has an option to tell it what order to _act_ on what it's traversing, but it still descends into each directory it encounters when it encounters it.The ls.c code is taking manual control of the traversal by having the callback return DIRTREE_SAVE without DIRTREE_RECURSE so the traversal populates a directory's children, then it converts the linked list to an array, sorts the array, uses the array to re-link the list objects in the right order, then it iterates over the sorted list and calls dirtree_recurse() again on each directory entry.

So I want dirtree_recurse to assemble the list, call a sort callback on the directory that can reorder the children, and then traverse them and descend. Which is a different callback from the current DIRTREE_COMEAGAIN callback? Do I need a third dirtree->again flag value? It's got 1 (callback on directory after processing all contents) and 2 (DIRTREE_STATLESS returning a file we can't stat), which are set/used as constants without macros defined for them. A third means macros, what would... DTA_AGAIN and DTA_STATLESS maybe?

Hmmm... but IS this callback a different one than DIRTREE_COMEAGAIN? It sounds like DIRTREE_BREADTH means: 1) DIRTREE_SAVE a linked list of a directory's children without recursing, 2) call the DIRTREE_COMEAGAIN callback on the directory, 3) traverse the saved list... doing what exactly? When are these freed? If we free them the step 3 traversal how do they ever get used?

Ok, I think I do want a third flag: DTA_DIRPOP lets you sort a directory after it's populated, and then we call with DTA_AGAIN on each entry right before we free it. Except the find -depth question comes in: does the directory count as occurring before or after its contents? That's a question for the sort function... ah, ok: while traversing the list, do a DTA_DIRPOP call before descending into it, DTA_DIRPOP|DTA_AGAIN after populating it, an then DTA_AGAIN without DTA_DIRPOP before freeing it. Silly, but it gives the callback multiple bites at the apple while still having generic infrastructure actual do the traversal.

And this is basically a wrapper function before the existing add_to_tar() dirtree callback that checks the flags and does sorting stuff as necessary, but otherwise calls the other callback. And you only insert the second callback when doing --sort. Ok, that seems feasible?

Implementing is easy, figuring out WHAT to implement is hard.

Darn it, one of the commands that came up in need of tweaking when I change dirtree semantics is chgrp... which was never converted to FLAG() macros. But chgrp.tests needs root to run meaning I want to run it under mkroot and that whole BRANCH of development is... several pops down the stack.

My _development_ plan has circular dependencies. Gordian knot cutting time, let's do it "wrong" for a bit just to clear some things...


January 17, 2023

My sleep schedule has been creeping forward towards my usual "walk to UT well after dark and spend the wee hours at the university with laptop", but I got woken up at the crack of dawn by sirens, flashy lights, and engine sounds right outside my window because the big house on the corner caught fire, and between something like 7 fire trucks and the police blocking off the street at both ends it was Very Definitely A Thing even from bed. I got up to make sure there wasn't incoming danger to us, and then I was up...

Kind of out of it all day as a result. Got a nap later, but "5 hours then FORCED UP" is something I may be too old to handle gracefully...

1pm call with Jeff to go over the Linux arch/sh patches, and the mmu change that apparently motivated the latest round of dickishness.

Elliott wants --sort=name, so looking at that. The man page has a short -s right next to it, which... "sort names to extract to match archive". What does that _do_ exactly? I'm already going through the archive in the order the names in the archive occur. There's not much alternative with tar. You can pass a bunch of match filters on the command line, but it's going to encounter them in the archive it's extracting, and thus extract them, in the order they occur in the archive. Tar != zip, it's not set up to randomly seek around, especially when it's compressed.

Sigh, my tar tree still has 2/3 of a --wildcards implementation in it, and does not currently even compile. Plus a bunch of test suite tests the host passes but my version doesn't. Need to finish that or back it out...

And when I do full tests against the musl build, tar is failing the "xform trailing slash special case". Which I don't notice when it's skipping the xform tests because it's using non-toybox sed (as happens on "make test_tar" unless I do special $PATH setup), and which I don't notice when testing a full glibc build because it works there. 95% likely it's musl's regex implementation, but... what specifically is diverging?

I would have an easier time with this if I remembered exactly what the "xform trailing slash special case" IS. October wasn't that long ago, but I checked this in as part of a large lump after days of work and there were a bunch of tests? It's searching for "^.+/" which... ^ is start of string, . is single character wildcard, + is * except "one or more" instead of "zero or more", and then / is presumably a literal / except it says "special case" here... Sigh, was this in the tar manual?

The example at the very end of that page is about specifying multiple sed transforms on the same command line, the first of which is NOT TERMINATED PROPERLY. (I.E. --transform='s,/usr/var,/var/' is missing a comma at the end.) And they repeat it twice the same way. Is this a doc mistake they cut and pasted, or does their implementation accept that? I'm afraid to check, and have NO idea how to deal with it if their implementation DOES allow it but normal sed doesn't. Maybe circle back to --xform after implementing the new stuff...


January 16, 2023

Ok, here's how I could cheat on the toysh "read" builtin: the case I care about optimizing is "while read", and the "while/do/done" block has an entry/exit lifespan. I can have the "while" cooperate with "read" to cache a FILE object. The read has to save it because "-u fd" is a read argument, but the while gives it someplace TO save it with a longer lifespan than the individual read call, and passing out of the "done" lets us know when to free the FILE *. Hmmm, I could store it in sh_blockstack's char *fvar with an evil typecast, that's not used by while... I'm dubious. Need to pace about it more. Probably best to implement just the slow path first. (There are SO many read options... timeout, length with and without terminator, -s puts the terminal in raw mode... I'm gonna need to back and implement array variable support in everything at some point? How do I stage this sanely...)

Oh hey, Greg KH is _also_ yanking most of the classic graphics drivers from linux-kernel. It REALLY sounds like linux-kernel development is collapsing and they're throwing code overboard as fast as they can. I hope that's NOT the case, I really thought we had another 5 to 10 years before that happened, but if Linus has decided to retire early because his daughters are all off to college... Let's see, his and his three daughters' birthdays are the easter egg values in "man 2 reboot" LINUX_REBOOT_MAGIC2, which are:

$ printf '%x\n' 672274793 85072278 369367448 537993216
28121969
5121996
16041998
20112000

So Linus is 53 (december 28, 1969) and his _youngest_ daughter is 22. Yeah, he's probably recently become an empty nester, and may be "quiet quitting" to go do other things with his life. And Greg has been waiting DECADES for the opportunity to do to Linux what Elon Musk is doing to twitter. Like an alcoholic buying a distillery. Sigh.

My annoyance with current linux kernel development is "Stop breaking stuff. Can the things that used to work still work?" And the reason we CAN'T have a stable kernel that doesn't shed features is... Greg Kroah-Hartman! Who many years ago proudly wrote a document named stable-api-nonsense about how the concept of Linux EVEN HAVING a stable driver API so you could keep portable divers between versions the way Windows did for many years... Greg said that's a crazy idea that Linux would never ever do. Userspace can still run a static binary from 1996, the kernel can't load a module from 9 months ago. Partly because GPL, and partly because Linux MUST be free to completely rewrite all the plumbing every 18 months to gain half a percent performance improvement with worse latency spikes. And now Greg's deleting a bunch of working drivers that are too hard to maintain under his insane regime. Wheee...

Sigh. Speaking of spiraling narcisists, did you know that Elon Musk got the idea of going to mars from a science fiction book the Nazi rocket scientist Werhner von Braun wrote in 1949, in which the emperor of Mars was named "Elon"? Back in the 1950s the reason Musk's grandparents gave for leaving canada for apartheid south africa was they perceived a "moral decline" in Canada (Wikipedia says "Most of the recorded student deaths at residential schools took place before the 1950s" so Musk's grandparents left Canada's about when the mass kidnapping and murder of native children declined, and instead they traveled halfway across the world to participate in Apartheid). So there's a nonzero chance Musk was named after that character in the 1949 German book, since his family was VERY familiar with a wide range of otherwise obscure nazi materials. So of course various Musk fans are now going "famous rocket scientist predicted Elon would be emperor of mars!" and I'm going "you have the causality here exacty backwards". Why do people keep thinking the man's ever had an original idea? That's NEVER been how he works...

My grandfather also interacted with Von Braun, they worked together on the Apollo program. (My parents met on the apollo program, because my father dated his boss's daughter.) The story grampa told me was that Von Braun's most important contribution to the US space program was statistical failure analysis. Grampa never mentioned the NSA until he got _really_ low while my mother was dying of cancer in the other room, shortly after my grandmother had died of her chronic lung problems (emphysema and eventually lung cancer, from years of smoking back before I was born). They'd had three kids, I never met Uncle Bo who volunteered to fight in vietnam over Grampa's strenuous objections and died there when his helicopter was shot down. Grandpa was now outliving a second kid and not taking it well, and started by complaining about how his hearing was shot and his big interest that got him into electronics had been crystal radios and audio. He was telling me how the allies recorded sound on magnetized metal wire but it got cross-talk when spooled and you couldn't splice it if it broke or got a kink, but they captured desk-sized audio reel to reel tape recorders from nazi bunkers which were a MUCH better design: built-in insulation between the magnetic layers in the spool and the tape could be cut cleanly and attached together with scotch tape on the back, and some of the GIs shipped a couple of them to the states where Bing Crosby paid to have them reverse engineered (and vastly simplified) so he could ship tape reels around to radio stations instead of constantly flying to give live performances, and this became the company "Ampex". Grandpa also told me how he did cryptography during the war creating one time pad vinyl records of "music off the spheres" radiotelescope recordings of ionized particles hitting the upper atmosphere sped up to be good random static which completely drowned out the voice unless you had the exact same record to do noise cancelling on the other side (stacks of these records were carried across the atlantic via courier, each one smashed and the pieces melted after one use). Churchill and FDR used these to securely talk to each other live over transatlantic cable, and this proceeded naturally to grampa venting about being blackmailed into joining (what became) the NSA after the war because they were going to draft him and put him on the front line in Korea if he didn't "volunteer", and then not being able to get out for decades until some idiot almost got him killed in Iraq in the 1980s by trying to hand off intelligence to him in his hotel room while he was there as a technical expert for General Electric upgrading (and bugging) the Iraqi phone system. (Apparently the various spy services are the best technical recruiters, finding you companies to work at. Well, they were decades ago, anyway. My take-away was "don't get any of that crap on you, you'll never get out again", and I learned it from my father's simple defense contracting.)

Oh hey, Dreamhost replied. They escalated to somebody who DID NOT BOTHER TO READ MY SUPPORT REQUEST. Not even the usbject line, which reads "Re: The robots.txt you put on lists.landley.net (which you won't let me log into) blocks google."

On 1/15/23 23:48, DreamHost Customer Support Team wrote:

> Hello,

> Thank you for contacting DreamHost Support! My name is XXX and I'd be happy to assist you with your concerns.

> With regards to the discussion list service, the last time this service was touched was last year in July when we had a maintenance on it to where we upgraded the services to new hardware. This didn't change much of how the service functions, though, as we're still running the same Mailman version as before under 2.1.39.

The robots.txt file is not technically part of mailman. Mailman runs in a web server, and that web server is serving the robots.txt file.

> About the http://lists.landley.net/listinfo.cgi page, that page has been disabled for a long time now.

I noticed. I complained about it at the time.

> The list overview page for the discussion list service was disabled over 5 years ago, actually.

Yes, as I told you in my last email. Closer to ten, really: https://landley.net/notes-2014.html#20-12-2014

> So, that page posted the "The list overview page has been disabled temporarily" message for a very long time now.

What does the word "temporarily" mean here?

> Unfortunately, that cannot be edited, but you already have your list archives set to public, so they can all be accessed here: http://lists.landley.net/listinfo.cgi/toybox-landley.net

Yes, I know, they are linked from https://landley.net/toybox on the left. But if people go to the top level lists.landley.net page, they do not get a list of available lists, and every couple months (for many years) people ask me why, and I tell them "because Dreamhost is bad at this".

For comparison, if I go to http://lists.busybox.net I don't need to remember the exact URL of the list I want to look at, because there is a navigation link right to it. That is not true of the toybox project, and I can't fix it, and my stock response to everyone who asks is "because Dreamhost is bad at this". Your service makes my open source project look bad to the point it's a FAQ.

The top level index page is especially nice if I'm sitting down at a different machine than I'm normally at and using a standard web browser to see if there are new posts, because remembering the full URL with the "dash between toybox and landley but the dot between landley and net and also a dot between listinfo and cgi"... that's tricky to do from memory.

> Since it's public, clicking the "Toybox Archives" link will open up the archives for that list for anyone that finds it.

I know how mailing lists work. I use them. If you looked at the mailing list in question you'd see I last posted to it on thursday. The "enh" posts are from Elliott Hughes, the maintainer of the Android base operating system for Google. He's the second most active poster to the list in question. I used to have other mailing lists for other projects, but they ended or moved off dreamhost "because Dreamhost is bad at this".

> As for the robot.txt file,

It's robots.txt.

> your 'lists.landley.net' sub-domain for the list does not use a robot.txt file.

Because it's robots.txt, as defined in the IETF RFC documents: https://www.rfc-editor.org/rfc/rfc9309.html

Point a web browser at: http://lists.landley.net/robots.txt

Do you see the file there? The file is wrong. The result returned by fetching that URL (which I CUT AND PASTED INTO MY LAST MESSAGE TO YOU) prevents Google from indexing the site. I do not have control over this file, for the same reason I had no control over the "temporarily disabled" message. It is a thing Dreamhost did on a server I do not have direct access to.

> In fact, on the mailman server, the services are not actually under the list sub-domain. That's just the sub-domain that all of your lists are managed under.

Do you see the "which you won't let me log into" up in the subject: line of this email, from my original support request?

In the message you are replying to I explained that "landley.net" and "lists.landley.net" are on different servers and I don't have access to the lists.landley.net one to fix what's wrong with it. You are repeating my problem statement back at me.

> But, on the mailman server, each list has its own set of configurations and files. For example, the stuff for the 'toybox' list is under the 'toybox-landley.net' location on the mailman server and has no robots.txt file.

When you fetch the URL from the server, there _is_ a robots.txt file. (Which you spelled properly this time.) The text "temporarily disabled" probably wasn't in the toybox-landley.net subdirectory either. The mailman install has templates for shared infrastructure.

This implies that it's a global setting, and you have thus blocked google search on EVERY mailman domain that Dreamhost serves. (Which I suspected was the case but don't know what other server pages to look at to confirm it.)

> It's just a sub-domain DNS record that points to the list server for where the list is managed.

Yes, I know. I managed my own DNS for the first few years you hosted my site, until I took advantage of your free domain renewals as part of the bundle.

I'm sure there was a little "yes I am experienced at web stuff" radio button selector when I submitted this help request? It did not have a "I ran my own apache instance for about 10 years, have also configured nginx, and even wrote my own simple web server from scratch in three different langauges" option, but still. (The httpd implementation I wrote last year is at https://github.com/landley/toybox/blob/master/toys/net/httpd.c because I needed something to test the new wget implementation with, so I did a simple little inetd version. Haven't wired up CGI support yet but it's got about 1/3 of the plumbing for it in already.)

The problem isn't that I don't know what's wrong, it's that I do not have access to fix it. I thought I'd explained this already, but I can repeat it.

I can SEE the robots.txt file. So can google. It is there. It should not be.

> And lastly, I'm afraid that our list services are not configured to run through HTTPS and there are no plans on getting that updated at this time, unfortunately.

Yes, I know. But that isn't _fresh_ breakage, so I'm living with it as part of the general "dreamhost is bad at this" Frequently Asked Question.

But Google _could_ find my mailing list entries a year or so back, and can't now, so Dreamhost adding a bad robots.txt is fresh breakage. (Dunno how long the google cache takes to time out when a new deny shows up?)

Given that the project I'm maintaining through that mailing list is Google's command line utilities for Android (I.E. their implementation of ls/cat/set etc as described in https://lwn.net/Articles/629362/ ) that's especially embarrassing.

> This would be quite the project as it would require an upgrade of Mailman, likely to version 3, which is quite different from version 2. So, the list admin page can only be accessed through HTTP. I'm very sorry about that.

Eh, I'm used to it.

I don't _think_ Android has entirely dropped support for non-encrypted URLs yet, only for certain api categories. (Which sadly broke my podcast player when upgrading to Android 12 no longer let it load http:// podcast files, only https.) I think you still have a couple more years before your mailing list infrastructure becomes entirely inaccessible from phones: https://transparencyreport.google.com/https/overview?hl=en

That uptick to 100% in the chart when Android 13 came out is a bit worrying, but I haven't bought a new phone in a few years and mine is only supported through 12. _I_ can still access it. (And from my Linux laptop, of course. No idea if random windows or mac users still can though. Safari's policy and chrome's policy and mozilla's policy don't advance in lockstep, but I hear rumblings.

Most websites have put mandatory http->https forwarding in place where accessing http just gets you a 301 redirect to https for _years_ now. Try explicitly going to "http://cnn.com" or "http://google.com" in your browser, it will load the secure page. It can't _not_ do so.

The rise of "let's encrypt" (nine years ago according to https://lwn.net/Articles/621945/ ) was what finally let people start deprecating the old protocol in clients, because sites no longer have to pay for a certificate so even the third world organizations running a solar powered raspberry pi on their cell phone towers can afford https now.

> I hope that helps to clear things up.

No, it doesn't. The robots.txt file excluding * for / still needs to be removed so Google can index my mailing list posts like it used to do.

> Please contact us back at any time should you have any questions or concerns at all. We're here to help!

The concern I expressed in the subject line is still not fixed.

I'd guess they did this because they didn't have any other way to manage server load, and their servers are underprovisioned. I suppose if they're truly this incompetent and have no other solution, I can set up a cron job to scrape the lists.landley.net web archive and mirror it under landley.net? It's EXTREMELY SILLY to do so, but I can just add that to the FAQ I guess?


January 15, 2023

Oh hey, Greg Kroah-Hartman is also removing the RNDIS driver from Linux, which is how Android does USB tethering. I wonder when Linus stopped being maintainer? The glory hound's been trying to drag the spotlight onto himself for decades now, but used to get told "no" a lot for hopefuly obvious reasons. Honestly, he's half the reason I don't post to lkml anymore. Al Viro was less abrasive: I'll take honest distain over two-faced self-aggrandizing politics any day.

I have some domain expertise with USB ethernet: a couple years back Jeff and I implemented CDC-ACM USB ethernet hardware for the Turtle boards, which could talk to Linux and MacOS but not Windows because Windows doesn't support CDC-ACM. It's a reference implementation from a standards body, but does NOT have a generic Windows driver because Microsoft wants money from each hardware vendor to be "supported". To test it we got a beta of a driver from somebody that made it work for half an hour at a time (before you had to unplug it and replug it because the driver was an "evaluation" version that timed out), but Microsoft charged $30k to sign a driver for Windows, and each is specific to a vendor ID and model number. Microsoft chose to have no generic driver for the protocol, only drivers for specific devices, so each hardware vendor had to pay microsoft $30k each time they needed to update their driver. (They claim they eliminated unsigned drivers for "security", but it's a profit center for them.)

Everybody Jeff talked to suggested we implement the RNDIS protocol instead, which is something Microsoft invented but both Mac and Linux supported it out of the box, and that one DOES have a generic driver in Windows that doesn't require $30k periodically sent to microsoft. Switching our hardware to RNDIS didn't look hard, we just hadn't done the research to make sure there weren't any lurking patents. (PROBABLY not? https://web.archive.org/web/20120222000514/http://msdn.microsoft.com/en-us/library/windows/hardware/gg463298.aspx says "updated 2009" and "assumes the reader is familiar with Remote NDIS Specification; Universal Serial Bus Specification, the Revision 1.1" but that document has been carefully scrubbed off the internet, the oldest I can find is 5.0. Because implementing against the old version is a prior art defense, so the old version is yanked.

The protocol was all in the FPGA bitstream, the actual USB chip we'd wired to the FPGA pins was just a fancy transciever that didn't even know about packets, and USB 2.x "bulk" protocols are all the same packet transfers with different header info. We never got around to prototyping it, we ran out of time shortly after we got the CDC-ACM version working (including our own TERRIBLE userspace driver that just spun sending data to/from a memory mapped I/O interface into the kernel's "raw packet" plumbing, improving THAT was our next todo item but the benchtop prototype was 2x SMP so the driver eating a processor affected power consumption but not performance). Jeff and I both flew out of Tokyo, and a year and change into the pandemic the funding for that project ran out, so it got mothballed without doing a proper production run, and we just didn't get back to it. But using RNDIS was the easy fix, and it's what everybody ELSE in the industry did, including Android's USB tethering.

Now Greg KH seems to be saying "we're losing features left and right, our collaping development team can't maintain the stuff we've already got, so let's flex OUR market muscle to out-influence microsoft". Or something?

I suspect Android's response will be "USB tethering is no longer supported on desktop Linux then, oh well, here's a Linux driver for RNDIS if you want to make it work". I haven't asked Elliott, but I remember when USB file transfer between my Linux laptop and android phone used to be really simple... and then it was replaced by some Microsoft protocol I could theoretically install an elaborate Gnome program for which never worked. (Or I could install the Android Development Kit, enable the debug menu in my phone, and use ADB file transfer from the command line. I've had to download a new copy of the android tools from their website every time I've needed to get that to work, because version skew.) Linux on the Desktop is not a commercially significant target market, we get _courtesy_ at best.

Even years from now, it would still be WAY easier for the J-core guys to ship an out-of-tree Linux kernel module than externally add a driver to Windows without paying them $30k annually-ish. Stuff like the Steam Deck could 100% use an out of tree driver if they needed to. Greg is making vanilla linux development smaller, but who's really suprised? He was the author of the kernel's "Code of Conflict" after all, and Linus was the one who apologized on behalf of the community and very publicly went to therapy to dig the community even a little way out of that hole, not Greg. The aging development community was emitting distress signals in 2013, and again in 2017, and now it's 2023...

(Yes I know Greg wrote "Android has had this disabled for many years so there should not be any real systems that still need this." My phone's running Android 12, I just tethered to check and dmesg said "rndis_host 3-1.2:1.0 usb0: unregister 'rndis_host' usb-0000:00:1a.0-1.2, RNDIS device". Oh, and hey, there's a more convenient way to configure it than I've been doing. I honestly don't know if Greg is clueless or lying, but does it matter? He is Confidently Wrong White Male.)

USB 2.0 shipped in 2000 so it's fairly recently gone out of patent (hence predictable badmouthing from for-profit manufacturers TERRIFIED of commodity competition from cheap generic hardware; the instant anything becomes available for open royalty-free implementation in it MUST BE DESTROYED. The oldest RNDIS documentation I could find says "updated 2009" (not authored, updated, it's older than that) and "assumes the reader is familiar with Remote NDIS Specification; Universal Serial Bus Specification, the Revision 1.1" but that document has been carefully scrubbed off the internet, the oldest I can find is 5.0. Because implementing against the old version is a prior art defense, so the old version is yanked. It is entirely possible that it recently DID go out of patent... and thus must be destroyed. How that idea made it from one of the Linux Foundation's largest contributors to one of the Linux Foundation's most prominent employees, I couldn't speculate, but he's sure confident about it.

RNDIS isn't tied to a specific USB generation (it's a packet protocol going across a transport), but USB 2.0 should be out of patent now (the spec is dated April 27, 2000) and that chugs along around 40 megabytes per second, which is still a quite useful modern data rate: over 20 parallel 4K HD netflix streams, over two gigabytes per minute, just under 7 hours per terabyte. It's about 1/3 the _theoretical_ max rate of gigabit ethernet (which I never get), and we were implementing it full speed on hardware running at... 60mhz I think? Either 4 bit or 8 bit parallel bus into and out of the chip, moving multiple bits per clock. A USB-powered device talking USB-2.0 RNDIS ethernet isn't hard to implement. Our CDC-ACM implementation fit in an ICE-40 with space left over.

I'm grinding through some of those email files from yesterday, trying to identify all the patches sent to the linux-sh list (grep '^+++ ' seems a reasonable first pass there once they're in individual files), but thunderbird saved all the files with the current date so it's not easy to filter for relevance. So I'm doing for i in sub/*; do toybox touch -d @$(date -d "$(toybox dos2unix < "$i" | sed -n 's/^Date: [ \t]*//p;T;q')" +%s) "$i" || break; done (as you do, yes gnu/dammit date gets unhappy with \r on the end of a date string, apparently), and I get an error message:

date: invalid date ‘Mon Sep 29 01:50:05 2014 +0200’

And I'm going... wha? Cut and paste that string to toybox date and... yes, it fails too. First guess: click back in xfce's little calendar widget, September 29, 2014 was a... sunday. Seriously? Sigh. Ok, FINE. Oddly, that date's not from the headers, it's from an inline patch which means... how is my T;q on the sed not triggering? (Back before I added that, date was complaining that multiple concatenated dates with \n were not a valid date...)

Ah, my sed is wrong. It expects a space after the date and that message has a tab in the headers, so it continued on and pulled one from a "git am"-ish patch in the body of the message. Ok, fix that and check that they all convert... yup, now they do.

Huh. You know, _technically_ netcat UDP server mode could handle one packet and then go back into "listening for new connection" mode, which would solve the 'locks itself to a specific source port' issue. That wouldn't work for child processes: the reason it's handling UDP packets the sameway as TCP connections is so we can pass off stdin/stdout filehandles to child processes. Which is where the "no notification of when a connection with no flow control closes" problem comes from: we'd need some sort of keepalive packet and there's no obvious place to insert that (if the kernel hasn't got a flag we'd need a Gratuitous Forwarder Process of some kind). The reason I didn't do that before is I don't want two codepaths to implement the same thing. Really, my use case here for interactive mode is "Linux net console". Does that send from a consistent source port even across reboots? Hmmm...

At this point I honestly expect healthcare.gov to KEEP sending me emails after today: "You missed the deadline, it was yesterday! How could you!" Yes I did try Obamacare one year, but at the moment I have the classic "VERY NICE health insurance through spouse's work" arrangement, in this case Fade's graduate program through the end of the summer, and then maybe we'll do that Common Object Request Broker Architecture thing to extend it a bit if she hasn't found a job yet, at which point it's _next_ healthcare.gov enrollment period). Alas there's no obvious way to tell obamacare's automated system that A) I'm currently good, B) you are basically useless in Texas because republican assholes bounced the subsidies and sabotaged implementation, C) I schedule doctor's appointments when I visit my wife up to minneapolis because the hospitals _here_ are collapsing unless all you need is a $100 visit to a nurse practitioner in a strip mall to get regulatory permission to purchase pills from a pharmacy, which are all over the place now. (How much of that collapse was covid and how much was foretold in legend and song is an open question. Two answer the follow-up questions: 1) Yes it's intentional, 2) if you don't work for a billionare-owned company getting a UTI costs more than a car and potentially more than a house so you will put up with ANYTHING to keep your job and they have less competition from small businesses and independent contractors. Guillotine the billionaires.)


January 14, 2023

I wonder if there's some way to get mastodon to do the green check mark thing? If you view source, I've had the link up top for a while now, with the magic rel="me" thing that's apparently an important part of it, but it just doesn't register? (I was reminded by updating the page links for the new year...)

Always odd when I get a request to do a thing I'm in the middle of doing. Yay, I'm on the right track? Not quite sure how to reply... "Um... yeah."

While Fade was here, heading out to poke at my laptop usually meant I'd use like a quarter of my battery then head back. Getting the low battery warning comes as a surprise after a few weeks of Not Doing That.

Dreamhost forwarded my support request to a higher level tech. That's nice. Unlikely to hear back before monday

I am once again impressed by how broken Thunderbird is. This needs some context. So Rich Felker theoretically maintains Linux's arch/sh but he hasn't updated his linux-sh git repo in over a year, and Cristoph Hellwig unilaterally decided to delete Linux support for the architecture despite plenty of people still using it and having an active debian port and so on. He didn't just suggest it, but posted a 22 patch series to remove it. (The charitable explanation is he's doing a don't a "don't ask questions, post errors" thing and putting the onus on US to object loudly enough.) Of course Greg KH immediately jumped up and went "I am deciding" because he's like that, but in THEORY Linus still has the final say here, and has not weighed in last I checked? And of course the motivations for the removal are contradictory: the primary complaint is it hasn't been converted to device tree (which is true of a lot of stuff), so the reply is be sure to remove the stuff that IS using device tree. Thanks ever so much.

The guy who maintains the Debian fork has tenatively volunteered to become the new maintainer, and one thing he'd need is all the patches that Rich chronically hasn't applied for years now. (Jeff informed me of this, and has volunteered to help the new guy, but will NOT say so on the list, and I quote: "Not going to engage with LKM toxicity in any way, got permanently away from that way back in 2002." So I connected them in private email and am very tired of doing that. But I still haven't posted this set to the kernel list myself, so can't exactly blame him?) So a useful concrete thing I can do is grab the accumulated linux-sh patches that have gone by on the list. So I'm giving it a go.

The first problem is gmail is crazy, and only ever keeps ONE copy of a message when I'm sent multiple copies with different headers, which means when I get emails cc'd to linux-kernel and linux-sh I only get one copy and which list-id it is is semi-random. (Usually linux-sh because that server has fewer subscribers so sends my copy out faster, but not reliably.) In each mbox in which I _don't_ get a copy, reply threads get broken, and if I ever wanted to put together a cannonical toybox history mbox file (and start a quest chain to eventually insert it into Dreamhost's web archive to fix the gaps) I'd have to check my toybox folder AND my inbox AND my sent mail (because I don't get my OWN messages sent back to me through the list either). But that's not FRESH stupid.

So I've done a search on my linux-kernel folder in thunderbird for messages with linux-sh in the "to or cc" field, which defaulted to searching subfolders but ok. Some of those subfolders are architectures or subsystems I follow (linux-hexagon and such, linux-sh is a seprate top level folder in my layout because I checked it regularly), but most of those subfolders are previous years of linux-kernel that I moved messages out to because thunderbird not only melts if you try to open a folder with a few hundred thousand messages in it, but email delivery slows down because the filters appending email TO those large mbox files somehow scale with the size of the mbox file they're appending to, and having linux-kernel as a regular destination gets noticeably slow every 3 months or so, and email fetch CRAWLS after 9 months without reorganization. So I have to periodically do maintenance to keep thunderbird running by moving messages into yearly folders to fight off whatever memory eating N^2 nonsense is in thunderbird's algorithms (a name and an offset in an mbox file doesn't seem like THAT big a struct, but it is in C++). Thunderbird's, "click then shift click to highlight a bunch of messages, right click move to other folder" plumbing ALSO scales badly with lots of messages (the "swap thrash" threshold is somewhere around 40k messages, which is much faster with an SSD but really not good for it, and the actual OOM killer kicks in somewhere in the 60k-90k range. There's something N^2 in their algorithm maybe? Yes the right click menu popping up even with 20k messages selected can take 30 seconds; it's a chore). But again, that's not FRESH stupid.

Thunderbird's search results window presents a list of messages but doesn't let me right click and DO anything to them. (I can double click one at a time to open in a new window, but not what I want here.) Instead I have to create a virtual "search subfolder", which has a pink icon and populates itself slowly as it re-performs the search (of each subfolder) each time you go into it, but otherwise seems to act as a regular folder. Fine. And after it had stopped adding messages clicking on the last message in the list pegged the CPU for 45 seconds before it showed me its contents. FINE. So eventually manage to I highlight all the messages in the pink folder, right click and get a menu, tell it to "save as"... and the resulting destination pop-up doesn't give the option of making a new mbox file, it wants to save them as individual messages. Ok. So I give it an empty folder to do so in, and...

Here's the FRESH stupid: a thousand empty files show up at once, with no contents yet, 30 seconds later the contents fill out in the filiesystem but I also get a pop-up saying "couldn't save file". Because it tried to open all the files it was writing IN PARALLEL and ran out of filehandles. (Or maybe loop to open them all, loop to write them all, loop to close them all? Why would anyone do that? The default ulimit -n is 1024, the default HARD ulimit is 4096 filehandles per process without requiring root access to increase. Don't DO that.)

Remember how I said Mozilla was not a real open source development organization? They are BAD AT THIS. So is the Free Software Foundation, the Linux Foundation, and even Red Hat. Capitalism mixes badly with open source development, even when it a nominal foundation claiming to shield developers from capitalism. Red Hat inflicted systemd on the world for profit (we're not allowed to opt out), the FSF zealots became as bad as what they fought, and the Linux Foundation and Mozilla did that wierd 501c6 trade association thing (a for-profit nonprofit tax dodge) where they're endlessly fundraising to provide exclusive members-only benefits.


January 13, 2023

Fade is on the airplane (as is This Dog).

Sigh, I do not have time for kernel shenanigans, but I guess I need to make time. Grrr. (There's a maintained debian port, Hellwig. Stop it. I didn't even post the bc removal patch to the list! I should do so, every release from here on...)

And Wednesday's question has been answered: the reason the middleman couldn't manage to pay my invoice THIS time is because "our policy is to pay invoices in arrears rather than in advance". Which is news to me because I was previously doing quarterly invoices (not risking TOO much of the money to a single transaction) and they paid Q4 in october. This time I invoiced for 2 quarters at once (hoping not to go through a multi-week debugging/negotiation process QUITE as often), and... Sigh. (They have literally one job. This is the fourth time it has not gone smoothly.)

The Executive Director of the middleman went on to suggest "if there is a strong reason for invoicing in advance, please do let us know, and we may be able to make specific arrangements for this — such a binding you to an agreement as a consultant."

I replied:

I invoiced for 2 quarters this time largely because each of the previous 3 invoices had some sort of multi-week issue. I honestly did not expect this one to go through smoothly either, but was at least hoping to deal with the problem less often. (I left a good chunk of the sponsorship money in there because I'm still not ENTIRELY convinced it won't vanish in transit again and maybe not come back this time.)

Now that I know the fourth roadblock is bureaucracy, let me know when and how much I'm allowed to invoice for to conform with your procedures, and I'll do that then. I'm assuming invoicing for Q1 would still be paying me in advance, so... March? (In previous quarters I got paid for the quarter we were in, but now that I'm aware of "policy" I'm assuming that no longer works either. Can I invoice for Q1 now and get paid March 1, or do I have to wait to submit and approve the invoice?)

As for whether I'm a flight risk, I've been working on https://github.com/landley/toybox/commits/master for 16 years (ala https://github.com/landley/toybox/commit/13bab2f09e91) which is longer than github's existed (hence https://landley.net/hg/toybox). Every commit in that repo was applied to the tree by me, and I personally authored... grep says 3642 of them. Even the current mailing list goes back to 2011 (http://lists.landley.net/pipermail/toybox-landley.net/) and dreamhost is terrible at mailing lists (https://landley.net/dreamhost.txt and https://landley.net/dreamhost2.txt and no I don't know where the threading info went back at http://lists.landley.net/pipermail/toybox-landley.net/2013-August/thread.html but after https://landley.net/toybox/#12-21-2015 and https://landley.net/toybox/#23-07-2022 and such I'm not asking).

Most of that time toybox was a hobby project I did around various day jobs (https://landley.net/resume.html). Google decided to use toybox in late 2014 and I kept working on it as a hobby for another 7 years. I am very grateful to them sponsoring me to focus on this project, and have said so publicly multiple times including in the release notes (https://landley.net/toybox/#12-08-2022). Disclosure: before the sponsorship I did get the Google Open Source Award twice, which came with a $200 gift card each time.

I suppose I could always get hit by a bus or have a stroke or something, but I'm not sure how signing a contract with you would affect that?

How WOULD the middleman perform oversight? Do they have any idea what success looks like? The only other guy who gets cc'd on this sort of thing is Elliott, and even I can't reliably find stuff like that again 6 months later. (Would they assign somebody to read my blog? Would that HELP?) Eh, KLOCs I suppose. Judge the value of a car by the weight of metal added to its construction...

While trying to google for a link writing the above, I noticed that lists.landley.net is no longer visible via google at all, and traced it to Dreamhost adding a robots.txt blocking... everything. I didn't change anything, and don't have ACCESS to change anything (remember: it's a shared server and they don't let me log in directly, everything happens through a web panel). I have opened a support ticket.

Oh goddess:

Subject: The robots.txt you put on lists.landley.net (which you won't let me log into) blocks google.

Hello Rob,

Thank you for contacting the DreamHost support team, I'm sorry you're having this issue, I will be happy to help. After checking your site under landley.net, I was not able to find the robots.txt you've mentioned, so to check the rules and offer you solutions. Have you deleted the file to prevent blocking Google crawling your site?

Please, have a look at our article on how to create a robots.txt file that is convenient for you https://help.dreamhost.com/hc/en-us/articles/216105077

I hope this troubleshooting and information was useful for you. Please, don't hesitate to contact back the support team in case you need it.

They didn't even read the TITLE of my support request, did they?

My reply:

> After checking your site under landley.net, I was not able to find the robots.txt you've mentioned,

Because lists.landley.net is not the same web server as landley.net. Your mailing lists run on a different (shared) server which I don't have direct access to, and which I can only interact with through your web panel.

Your server has been mildly broken for years, such as refusing to give a list of available mailing lists under https://lists.landley.net (which has been "temporarily disabled" for over a decade).

But sometime in the past year or so the robots.txt on lists.landley.net (which is not landley.net) changed, so that:

https://www.google.com/search?q=site%3Alists.landley.net

Says "no information available on this page", and when I click "learn why" under that it goes to:

https://support.google.com/webmasters/answer/7489871?hl=en#zippy=%2Cthis-is-my-site%2Cthe-page-is-blocked-by-robotstxt

> Have you deleted the file to prevent blocking Google crawling your site?

I would love to get access to lists.landley.net to fix stuff there, but the lack of that has been a persistent issue dealing with you for some years now:

https://landley.net/dreamhost.txt
https://landley.net/dreamhost2.txt
https://landley.net/toybox/#23-07-2022

I haven't even bothered to ask where the thread information for older months went:

http://lists.landley.net/pipermail/toybox-landley.net/2013-August/thread.html

(It used to be able to indent those, but not anymore.) But far and away the BIGGEST problem with lists.landley.net is you can't access it via https but only http, which means mailing list administration sends a plaintext password across the internet for every page load. (Because the Let's Encrypt certificate for landley.net isn't available to the shared lists.landley.net server.)

> Please, have a look at our article on how to create a robots.txt file that is convenient for you https://help.dreamhost.com/hc/en-us/articles/216105077

I know what a robots.txt file is. But I do not have access to change any of the files at https://lists.landley.net. I can only ssh into the sever that provides landley.net (ala www.landley.net) because the different domain name resolves to a different host.

> I hope this troubleshooting and information was useful for you.

Not really. Here is the issue:

https://landley.net/robots.txt - 404 error
https://lists.landley.net/robots.txt - ERR_CONNECTION_REFUSED
http://lists.landley.net/robots.txt

  User-agent: *
  Disallow: /

Meaning Google cannot index the site. It USED to index the site, but it stopped sometime during the pandemic, because of you.

I hate having to explain people's own systems to them. It's embarassing for both of us. I also dislike having to reenact the Dead Parrot Skit. ("If you want to get anything done in this country you've got to complain until you're blue in the mouth.") I feel there should be a better way, but I'm not good enough to find it.

Walked to the table for the first time in a while. (I was hanging out with Fade in the evenings, and mostly staying on a day schedule with her.) Pulled up the list of pending shell items... and then spent the evening editing old blog entries since new year's.

My blog plumbing (such as it is) has a slight year wrapping issue: I switch to a new filename for 2023. The rss feed generator takes the html file as input and emits the most recent 30 entries in rss format, using the stylized start of new day html lines to split entries. Which means if the December entries aren't appended to the new year's file they'll prematurely vanish from the RSS feed, but when I DO append them I keep forgetting to delete them and I think some previous years might STILL have the previous december at the end?

There's always a temptation to cheat and not edit/publish January for a week or two, so that when the RSS feed updates everybody's had plenty of time to notice the old stuff, and anybody new checking it won't see a questionably short list. Not that I need MORE incentive to procrastinate about a large time sink...


January 12, 2023

Fade flies home tomorrow, mostly spending time with her.

The ls.c stuff (fallout from this) is harder than it needs to be because ls --sort has a lot of long options (and I added a couple more). The current help text looks like:

-c  ctime      -r  reverse  -S  size     -t  time   -u  atime   -U  none
-X  extension  -?  nocase   -!  dirfirst

And I went "well of course that should be comma separated values with fallback sorts, just like sort itself does!" and that's... tricksy. Each of those can be specified as a short option (which doesn't save the order it saw them in), and you can presumably mix short and long options, and I dowanna re-parse the list each time because that feels slow but I don't have a good format to put it in?

Eh, data format's not hard: array of char that's the offset of the flag value for the sort type. Break the comparison out into its own function and feed it either toys.optflags or 1<<sort[i] in a loop. If I ensure flag 0 isn't interesting (it's currently -w, not a sort option) then it's even a null terminated string (of mostly low ascii values, but still). But the design and user interface parts are still funky: the longopts would accumulate as fallback sorts and the single character sort flags should switch each other off? No, it's more complicated than that: you can do ls -ltr with is reversed mtime, so they DO chord at least sometimes... Actually, "reverse" is specifically weird. Sticky. It should ALWAYS go last because otherwise it has no effect.

No, the really WEIRD chording is -cut with or without -l. (I was working on this before, I know I was, and I document COMPULSIVELY, but it's not in the blog. Is it on the mailing list? One of the myriad github pages? A commit comment? Did I make the mistake of typing it at someone in IRC and setting the little mental "it's been written up!" flag? Who knows...)

(Once upon a time the #uclibc channel on freenode, where all the busybox developers hung out back in the day, was logged to a web page, which I believe went down when the guy hosting it went through a bad divorce, and in any case freenode got bought by a korean billionaire who did to it what the muskrat is doing to twitter. I still sometimes think "written in irc" means "I can find it again later", but have mostly trained myself back out of that these days.)

Anyway, the issue is that the ls man page (and behavior) is nuts:

-c     with -lt: sort by, and show, ctime (time of last modification of
       file status information); with -l: show ctime and sort by  name;
       otherwise: sort by ctime, newest first

So -l disables -c's sorting behavior and you add -t to get it back. Same for -u. That's horrible historical nonsense and I need to make it work, but where does --sort ctime and --sort atime work into this mess?

As always "how to implement" is easy and "what to implement" is hard.

Sigh: [-Cxm1][-Cxml][-Cxmo][-Cxmg][-cu][-ftS][-HL][-Nqb] is tangly. But lib/args.c hasn't got a [-Cxm[1log]] syntax and there haven't been other callers for it.


January 11, 2023

I have invoiced the middleman! Let's see how it fails to work THIS time.

Staring at the bugs shell fuzzing found. And ls.c. And the shell "read" command. And the shell "command" command, because command -v is where scripts/portability.sh barfs trying to run the test suite under toysh.

Not really making a lot of progress on any of them, but looking. Oh, and I should read that git format documentation...


January 10, 2023

Finally cut a toybox release. And then updated the date in news.html to the correct day AFTER cutting the release. (The tarball and tagged commit still say the 8th. Always something.)


January 9, 2023

I have reached the "revert and rip stuff out" stage of release prep: if it's not ready, punt it to later.

Disabled the date test in "make tests" again: not shipping a fresh toolchain this time. Put bc and gcc back in the airlock because not demanding people build a patched linux this time either. More FAQ updates. I can't get the ls.c work in this time...

Right wing loons are flipping out about a possible ban on gas stoves, which means Fuzzy has been involved in an argument online where somebody insisted it was impossible to make proper custard on induction, so we have a big pot of custard now. It's lovely. Peejee has a want. (Cat, YOU may be spry and feisty but your kidneys are 19.)

Peejee had custard.


January 8, 2023

Found a problem with "make sh_tests" where some of the early tests weren't testing the right shell. There's a context switch before which you can do "sh -c 'test' arguments", and then it switches to having all the tests run through sh -c _for_ you, to ensure it's all being tested in toysh rather than being "eval" under whichever shell the test suite is itself running in. (On Debian, bash. In Android's case, mksh.) You can manually wrap tests before that yourself, but I found a set of tests before the switch but weren't wrapped, and moved them after the switch so they happen in the proper context... and some fail. Now I've gotta fix unrelated stuff before running the test suite gets me back to seeing the failures I was debugging before Progress of a sort, I suppose. But it puts us firmly into "punt this whole mess until AFTER next release" territory.

Oh hey, right after I tried to pivot AWAY from working on toysh, Eric Roshan-Eisner ran a fuzzer on toysh and found several ways to segfault it. Fixed some low hanging fruit, punting the rest until (say it with me) after the release.


January 7, 2023

I kiiiinda wanted to get "make test_sh" passing its test suite this release. Not happening, but I have multiple dirty sh.c forks I'm trying to check in and delete. The release is the time to finish unfinished things and clean up what you can.

The next "make test_sh" failure was a simple fix [editorial note: while trying to add the link to the blog I realized I'd checked it in to the WRONG TREE: pushed now] but the next thing that test tried to do is call the shell builtin "read"... which I haven't implemented yet. Taking a stab at it now, but there's a design problem: it does line reads but lets read -u substitute a different file descriptor. Hmmm...

Strings are hard, and that includes efficiently reading lines of input. This is why I had get_line() all those years: byte-at-a-time is slow and CPU intensive, but larger reads inevitably overshoot, and you can't ungetc() to an fd. (Well you _can_ but only to a seekable fd, which does not include pipes or tty devices.) This is why the ANSI/ISO C committee invented the FILE * back in the 1980s: somewhere to store a buffer of data you block read and save the extra for next time. But shells don't USE file pointers, they use file descriptors, both for redirect and when spawning child processes.

This isn't AS bad because pipe and tty devices return short reads with the data they've got, so when a typing human is providing input MOST of the time the computer will respond to the enter key before you press the next key. And piping between programs, each printf() turns into a seperate write() system call which sends a batch of data through the pipe and if the read() at the far end receives that data before more gets sent (and concatenated in the pipe buffer) then it hasn't read ahead part of the next line there either. But if you DO type fast (or something like "zcat file.gz | while read i" happens) then the read gets extra characters that go on the next line, but the read returns and the next read happens only knowing the file descriptor. (If you're wondering why you see "echo $X && sleep .1 && echo $Y && .sleep .1" in some places in the test suite... generally this sort of thing. Even that's not ENTIRELY deterministic if the system's heavily loaded enough.)

This same problem would also screw up trying to provide input to a command, such as echo 'fdisk blah.img\nn\np\n1\n\n\nw' | sh because the FILE * stdin used to read the fdisk line will read ahead to the rest of the data in the input block, which is then NOT provided by file descriptor 0 to the fdisk child process, because it was aleady delivered and is waiting in an unrelated buffer. (I bothered the Posix and C99 guys about querying how many bytes of data are waiting in the buffer so I could fread() them back OUT and pass them along, about like my tar.c does when autodetecting compression type from pipe input. You read data and then need to DO something with it, can't go back into the file descriptor.

(If you COULD unget data into a read-only file descriptor, that would be a security issue. Unix semantics generally do make sense when you dig far enough, because everybody's had 50 years to complain and usually would have fixed it by now if it was actually wrong.)

All this reminds me I'm ALREADY mixing FILE * and stdin in sh.c because get_next_line() takes a FILE * argument, but that's always been a placeholder function. I need to implement interative line editing and command history, which I should tackle soon but it's not a _small_ can of worms to open and I want to get the shell more load bearing before putting a big neon "welcome" sign out front. But the interactive stuff should use the input batching trick I introduced to busybox vi years ago to figure out what is and isn't an ANSI escape sequence, which I already iplemented in lib/tty.c scan_key_getsize() and is STRONGER reliance on input batching. (It would be lovely if I could fcntl() a pipe to NOT collate data written to it in its buffers, but this seems to be another "optimization" I can't turn off. It would also make testing dd SO much easier if I could do that...) Anyway, scan_get_getsize() is always doing 1 byte reads to assemble the sequence from a file descriptor without overshooting, because "interactive keyboard stuff" really should not be a big CPU/battery drain on modern systems. (He says knowing that "top" is a giant cpu hog that really needs some sustained frowning to try to be less expensive. I dunno if it's all the /proc data parsing or the display utf8 fontmetrics or what, but something in there needs more work.)

I suppose I could try to lseek() on the input, and do block reads when I can and single byte reads if I can't? The problem is the slow path is the common case. I don't want zcat file | while read i to be an unnecessarily slow CPU hog, and the FAST way is using getline() through a FILE * (or writing my own equivalent; generally not an improvement). Which doesn't work for -u, and if I wrapped that in a FILE * where would I save the cache struct between "read" command calls? How do I know when it's done and I can free it? Can of worms. Redirect and FILE *stdin aren't going to play nice together, but what's the BETTER way?

Sigh, I'm not entirely sure what the corner cases here ARE. Coming up with test cases demonstrating it causing problems is a headache all its own. And some of those corner cases I'm pretty sure bash suffers from and their answer is 1/10th second sleeps.

I don't WANT two codepaths, one for stdin and one for -u other than 0. That's just creepy.


January 6, 2023

Elliott didn't like xmemcmp() so it's smemcmp() now. (Yeah, I know he's right, just... trying to get a release out!) Bugfix from somebody at google for sh.c (yay, people are looking at it). FAQ update...

The new dishwasher is not arriving today, supply chains are failing to supply, or possibly chain. New estimate is the 20th. Fuzzy is very tired of doing dishes by hand, we have purchased paper blates, cups, bowls, and plastic utensils.


January 5, 2023

Day 2 of Actual Release Prep, which _seems_ like it's mostly just creating a big page of release notes but is actually "go through each commit since the last release and re-examine them", which is second pass review (more design than code per se) and a MASSIVE tangent generator. It always winds up taking multiple days to actually get a release out, and that's AFTER I've done a full mkroot build-and-test recently on the kernel version I plan to release with, using the toolchain I plan to release with. (I.E. no blocker bugs.)

The toolchain issue's a bit wonky this time, because llvm version skew broke my ability to rebuild the hexagon toolchain with the script that worked last time, but I need to rebuild musl to include the patch for the issue Rich refused to fix but which otherwise breaks the date test that runs on glibc and bionic but not musl. (I enabled the date and gunzip tests in "make tests" because they worked fine on glibc and bionic, and only after I'd checked it in realized that date was still disabled because the musl bug was a pending todo item.)

It's a one line fix to musl, but Rich won't do it because Purity or something, and after many years fruitlessly arguing with Rich I just put the simpler workarounds in my code, and otherwise patch musl in the toolchains I build. I wave a patch at Rich once so it's not MY fault when he says no, same general theory as linux-kernel. (Except there I tend to do second, third, even fourth versions when I get engagement. Less so when they're ignored.)

Speaking of which, my musl patches are still inline in scripts/mcm-buildall.sh but I've moved the kernel patches from scripts/mkroot.sh out to a separate repository. I should philosophically collate my patch design approach at some point, but I'm not holding up the release for it now.

In general broken out patches are better in case other projects want to pick them up. Squashfs was widely used out of tree for many years before lkml deigned to notice, and some of the Android stuff still is I think? Back on Aboriginal I had a patches dir with linux-*.patch in it. For this release I'm probably putting 0001-*.patch files for the kernel in the mkroot binaries release dir because "apply these to your arbitrary kernel version" seems easier than collating two otherwise unrelated kernel trees via git. But how much of that is what I'm used to vs what other people are used to? (I mean I HAVE a branch published on github, but have to redo it each release I do a musl thingy on and then can't delete the old ones if they're load bearing, which is non-collated cruft I dowanna encourage/accumulate. "Probably gonna delete this later" is not a good ongoing policy.


January 4, 2023

Rant cut and pasted from the list to the blog:

I'm not a fan of over-optimizing compilers. My commodore 64 had a single 1mhz 8 bit processor with 38911 basic bytes free, and it was usable. I'm typing this on a quad processor 2.7 ghz 64 bit processor laptop with 16 gigabytes of ram, and this thing is completely obsolete (as in this model was discontinued 9 years ago: they were surplussed cheap and I bought four of them to have spares).

Performance improvements have come almost entirely from the hardware, not the compiler. The fanciest compiler in the world today, targeting a vintage Compaq Desqpro 386, would lose handily to tinycc on a first generation raspberry pi. Hardware doubled performance roughly annually (cpu was 18 months but memory and storage and stuff in creased in parallel) and each major compiler rewrite would be what, 3% faster? The hardware upgrades seldom broke software (rowhammer and specdown meant the hardware didn't work as advertised, but that's an obvious bug we worked around, not "everything intentionally works different now, adjust your programs"). Every major gcc upgrade had some package it won't build right anymore and the gcc devs say we shouldn't EXPECT it to.

Part of this attitude is fallout from the compiler guys back around 1990 making such a big deal about the move from CISC to RISC needing instruction reordering and populating branch delay slots and only their HEROIC EFFORTS could make proper use of that hardware... and then we mostly stayed on CISC anyway (yes including arm) and the chips grew an instruction translation pipeline with reordering and branch prediction.

I'm aware this is a minority view and I'm "behind the times", but if I wanted the tools to be more clever than necessary I'd be up in javascript land writing code that runs in web browsers, or similar.

This difference of viewpoint between myself and people maintaining compilers in C++ keeps cropping up, and I have yet to see a convincing argument in favor of their side. They're going to break it ANYWAY.

I'm currently editing the December 28 blog entry about tidying up the html help text generator, and I realized a corner case I hadn't handled: nbd-client says "see nbd_client" which doesn't exist. (Public dash version vs private underscore version because C symbol generation.) Sigh. Ok, fix the help text generator AGAIN...

I keep nibbling at the release, but... time to start writing release notes. Ok, git log 0.8.8..HEAD and... there have been a few commits, haven't there? Lot to go through. But first, the hardest part: picking a Hitchhiker's quote I haven't already used.


January 3, 2023

Working to make ASAN=1 make test_sh pass, which is whack-a-mole. The address sanitizer's a net positive, but it's a close thing at times. (Gimme a stack trace of where the problem OCCURRED, not just where the closest hunk of memory was historically allocated.)

Refrigerator dude stopped by and vacuumed the refrigerator coils, which were COVERED with cat hair. Showed me how to do it myself, not that I own a vacuum cleaner. (Tile floors. Big flood back in 2014. The only carpet in the house these days is throw rugs we take outside and do a sort of bullfighter thing with.)

The outsourced washing machine guy called and said the symptoms I'm seeing on the model I have means the circuit board's almost certainly fried, probably had water leak onto it, which with labor is something like $800 to replace (Bosch is reliable, but not repairable), and getting a new dishwasher of the same model and having the old one hauled away is basically the same price, so there's not much point him coming out to look at it and billing us for not being able to help. (Professional repair dude, no shortage of work.) Thanked him and Fade ordered a new one from the people we got the replacement washer and dryer through, they think it'll be here Friday.

Yes, I am aware I did the refrigerator thing because "they're already coming" and then it was two seperate servicebeings. There was meta-upsell there, apparently. Unlikely to use Sears again, which is convenient since they only barely seem to still exist as a kind of temp agency.


January 2, 2023

Finishing up pending toysh fixes. I was redoing math stuff and it's always more work to figure out what I was doing when I leave myself half-finished commits in the tree. I can see what the code there is doing, but have to work out what I MEANT it to do and examine how much I got done to figure out what I left out. The design work is as always the tricksy part: is there anything I didn't think of at the time, or thought of but didn't implement, or thought of then but am not remembering now? (There's no such thing as sufficiently copious design notes that do NOT include a finished implementation. Not that the implementation by itself is always sufficient either, which is why there's code comments and commit comments and blog entries...)

Not gonna manage to merge expr.c in with $((math)) this release, and I'm not 100% sure it's doable (well, worth doing) at all. And the big missing piece seems to be a floating point version of this same code. Python seems to do arbitrary precision math: 1<<256 and 1/1.7 resolve but 1.7<<256 is an error. Multiple codepaths!

Eh, punt for now. Close the tab, move on to the next...

The dishwasher died. As in the power button does nothing, acts like it's not plugged in but the outlet works when we plug in other stuff? (RIGHT at new year's. IS there such a thing as a Y2023 bug?)

Despite Sears having died years ago, Google Maps has a number for "sears appliance services" in hancock center. (In the middle of the parking lot on the map.) And when I called it I got... what's probably an indian call center, but sure. They have our file from when we bought the thing. And want $150 for a service technician to come look at it. Hmmm...

I'm not entirely sure how they upsold me on having my refrigerator serviced, but it was just an extra $50 and nobody's looked at it in ~5 years and I'd rather it didn't go out, so sure. Why not. (Fade thought we might as well, anyway. Long as the dude's already making the trip...)


January 1, 2023

Happy new year! The first in a while that isn't "the year of hindsight", "last year won", or "also that year". We are finally "2020 free", or at least experiencing a diet version thereof.

Grrr. The recent xmemcmp() changes in file.c left me with an open tab where I WANT to replace a bunch of memcmp(x, "string", 8) with #define MEMCMP(x, y) xmemcmp(x, y, sizeof(y)) so you don't need to specify the length, but unfortunately it won't quite work. Yes, sizeof() treats a string constant as an array and thus gives you the allocation size, including the null terminator. Some of the comparisons in file.c are checking the NULL terminator, and some aren't. Having two #defines for the two different cases pushes this out of net-positive mental complexity savings territory. Subtle enough the NEW thing becomes a sharp edge you can cut yourself on. The other is redundant/tedious but very explicit.

While reviewing them I did find a memcmp(s+28, "BBCD\0", 5) so once again no review is ever COMPLETELY wasted...

Maybe I should rename _mkroot_ to "dorodango"...


Back to 2022