Sigh, I'm trying to redo the toysh line input processing. The trailing \ logic is different from what bash does (ongoing thread on the list with chet about that), and I _also_ have a hack I've been meaning to clean up: because parse_line ignores completely blank lines, I'm having EOF feed in single space " " lines to flush pending line continuations, which is wrong for multiple reasons. For one, toybox sh -c 'echo hello\' has an space on the end instead of a backslash, which is TWO bugs (\ gets eaten, space gets added). For another, you can have MULTIPLE pending line continuations, and a single EOF line won't necessarily flush them. Except we return 1 when we need another line, and there's different reasons for needing another line, which behave differently: unterminated if or || flow control errors out, but unterminated HERE documents are terminated by EOF (with a warning in bash, silently in the defective annoying shell). And you can have more than one HERE document pending at the same time: cat << EOF1; cat << EOF2 or even just cat << EOF1 << EOF2 (no they don't append, stdin gets redirected twice so the first one is dropped, but 3<<EOF would let you read from fd 3... Oh, and HERE documents are seekable, I should add a test for that. It writes to a deleted temp file to get a seekable filehandle that frees its contents automatically when closed. Classic unix filesystem semantics...)

My line input logic was removing trailing newlines right at the start, but I can't do that because an escape at the end of a line that was NOT ended with a newline gets preserved. (Well, not RELIABLY by bash, but I poked Chet about that. The -c processing is still magic.) So now I've got to propagate that \n through and it's essentially trailing whitespace which I'm already mostly handling, but something somewhere's likely to break. Plus NULL pointer, empty string, and string that only contain whitespace being DIFFERENT is why that "send in a line with a space in it" hack happened in the first place...

I'm also patching the HERE document logic to terminate all outstanding HERE documents at EOF. It can still return "nope, I need more" for unterminated flow control, at which point the caller errors out because there is no more, but the caller can't distinguish "need more HERE document lines" from "need more flow control logic", it just returns 1 to ask for another line, so it has to do it within parse_line(). And I've had to add multiple goto statements to get it to work because the existing logic really isn't set up to turn into a loop. Multiple gotos are not elegant, it means there should be a loop here which would require major surgery to insert...

Multiple people are now trying to use toysh and sending me bug reports, but what I _really_ need to do is grind through the "ASAN=1 make test_sh" bugs because every time I hit an issue or major todo item I try to throw a test in there which bash passes and thus toysh probably _should_. And there are a whole lot of existing tests toysh doesn't pass yet, which makes adding new ones awkward. (I keep sticking them near the start so they trigger, but there should be some logical order to all this...)

The next test_sh failure is a double free when a command comes after a HERE document, ala ASAN=1 make sh && ./sh -c '<<0;echo hello' which did print the hello! It didn't warn that the HERE document hit EOF, but I can presumably add that.

ASAN says the second free happened on line 2923... which is in the function free_pipeline() so yes it would, wouldn't it? This being gcc, it doesn't say who CALLED that function, because gcc's ASAN is crap. And I can't use the Android NDK's ASAN because it's only available as a dynamic library, and if I dynamically link against bionic it's not available on the host so the binaries won't run. And you can't LD_LIBRARY_PATH your way around the dynamic loader being /system/bin/linker64". Elliott suggested I could symlink /system to somewhere in the NDK, but find android-ndk-r25c -name linker64 produced zero hits.

So backing up, the first free was in do_source() which calls llist_traverse(pl, free_pipeline) after run_lines() returns. Because we've executed all the stuff and it's done with it now... Ah, and then it frees the HERE document but the last entry in that is the EOF which was one of the arguments to the earlier command line that already got freed. (Because why copy it when we're

I need to enable the leak detector, and whitelist the EXPECTED leaks at exit. I know how to do the first, not sure how to do the second. Other than writing a debug function to laboriously free stuff the OS is about to free for us. I'm worried about accumulating leaks during long runs, not blocks of data with the same lifetime as the process. I want some sort of leak_forget() function that says "anything that's already been allocated is not interesting for leak detection, only show me NEW allocations after this point that don't get freed".

Alas, gcc's ASAN is abandoned crap, and the LLVM toolchain I have wants dynamic bionic installed on the host, and will NOT work static linked (for no obvious reason other than they either didn't think of it or didn't bother). So puppy eyes about adding stuff would add it to a context I can't use anyway.

Hmmm, I should try getting a dynamic bionic chroot working again. In theory the stdin panic fix in the _start code has made it in to the release version by now? (I should really learn to build the NDK from source. Too many tangent ratholes...)

Email from Chet: my trailing backslash line parsing is wrong in toybox (or at least doesn't match bash). I knew my line parsing is wrong and I'd have to redo it, but it turns out it's wrong in more ways than I was aware of. Hmmm...

Also bash -c 'cat<

Oh, and no matter how you fiddle with the priority, HERE documents always seem to eat their lines before line continuation logic does:

$ if cat << EOF; then
> blah
> EOF
> echo hello; fi
blah
hello

Which toysh is already getting right, but I want to make sure I have tests for. And also:

$ if [ $(cat) == blah ]; then echo hello
> fi << EOF
> blah
> EOF
hello

I.E. your REASON for requesting line continuation can vary from line to line, based on parsing the new input. And that $(cat) can't be evaluated until the trailing redirect has replaced stdin for the whole block, which I'm already getting right but the new changes can't break that, hence regression testing...

I mean, if you REALLY want to go down the rathole here:

$ bash -c 'echo $LINENO'
0
$ bash -c $'\n\n\necho $LINENO'
3
$ echo 'echo $LINENO' > weeb
$ bash -c '. weeb;. weeb;echo $LINENO'
1
1
0
$ bash -c $'. weeb;. weeb;echo $(eval $\'echo $LINENO\\necho $LINENO\');echo $LINENO'
1
1
1 2
0
$ bash -c $'. weeb\n. weeb\necho $(eval $\'echo $LINENO\\necho $LINENO\');echo $LINENO'
1
1
3 4
2
$ bash -c $'. weeb\n. weeb\neval $\'echo $LINENO\\necho $LINENO\';echo $LINENO'
1
1
2
3
2

That's why each "do_source()" has it's own pseudo-function context, because LINENO is often a local variable even without a function call, which sometimes gets reset and sometimes gets inherited as you enter/exit each new parsing context, and I need tests for all of it...

Flew back to Austin first thing in the morning, early enough in the day I could hang out with new/old laptop at Wendy's and the HEB tables.

Finally got the ls --sort tests in, and fixed more than one bug found by them.

For example --sort can handle csv arguments (in toybox, not in gnu/dammit) but when I fed it more than one sometimes it was looping endlessly because when you've already matched you don't do further sorts (which meant it wasn't advancing past those arguments it wasn't processing), but it needs to CHECK those future arguments to see if there's a "reverse" in there, so it was looping without advancing... (There's also "unsorted" which stops argument processing despite not having matched, but I can't TEST unsorted because it means the filesystem order leaks through and I can't control what that IS)...

Yesterday the dentist said the tooth should come out, and responded to my "it doesn't hurt" with a poke with a tool demonstrating that the nerve is alive and well and not protected by much and capable of hurting a VERY LARGE AMOUNT QUITE SUDDENLY.

This would be the SEVENTH tooth I've lost. (Four wisdom teeth and two on top next to the incisors removed presumably for cosmetic reasons as part of the braces years ago? I had a somewhat pronounced overbite as a teenager. My parents were really making those decisions at the time, I was in early high school.)

He didn't quite come out and say it, but when my wisdom teeth were removed this tooth was left without a matching tooth to chew against, and that's apparently bad for teeth. Which means I'm losing this one because they took out too many wisdom teeth back in the day. Those two removed in the front meant the braces shifted the rest forward, which made room to KEEP the top two wisdom teeth (which I pointed out at the time) and the dentists went "no, you don't want to leave a tooth with no matching tooth for it to work against"... but they DID, didn't they?"

Add in the TMJ I got because the braces for some INSANE reason involved a rubber band from my upper left to the lower right (across my tongue) for 6 months, and this means basically every major experience I've had with american dentistry has caused future problems. The braces made my jaw click and grind, the previous tooth removals left me with an orphan tooth that's now collapsing, and the 2013 experience in St. Paul left two of my front teeth looking obviously terrible. The cost each time was probably somewhere around ten grand each time (if not more adjusted for inflation)...

Anyway, had him grind off the pointy bit (which didn't hurt for about five seconds and then essentially repeated the first poke; he is a VERY good dentist to not have done further damage inside my mouth when I lurched like that, but hey: my cheek can heal now). And then I scheduled a follow-up apointment for a month from now. I was thinking of flying back to dogsit while Fade's at 4th street anyway. (Previous years she flew back from Austin and left Adverb with us, but she's staying in Minneapolis this summer to finish her dissertation so she can defend it in August, which is why I'm flying there to see her here so many times this year.)

I may have gone a little overboard. Somebody emailed me to track down a citation, and I replied with my usual "THIS IS A SPECIAL INTEREST OF MINE" enthusiasm:

> Is this yours?
> https://landley.net/history/mirror/cpm/history.html
> I'm citing some of it in a book I'm writing and I wanted to make sure it was you.

As the "mirror" states, it's a copy of an old page from geocities. Here's the original pulled out of archive.org.

Back in 1984 Gary Kildall was one of the original co-hosts of the TV show "computer chronicles" (until he became too busy with his company to continue). Here's an episode he co-hosted on "programming languages", and here's an episode on "operating systems". When Kildall died, the show did a retrospective on him.

Here's the "standard" interview with him (I have a copy of this book). And here's another computer industry pioneer reminiscing about him.

You should really watch the PBS series Triumph of the Nerds (which is based on the book "Accidental Empires", the presenter is the book's author).

CP/M only became the standard operating system for "S-100" systems. (Here's a song Frank Hayes, columnist for ComputerWorld, wrote/performed about the S-100 bus. Yes, it's a "C shanty". From the album "never set the cat on fire".)

The "PC vs Mac" of the day was Apple II vs S/100 systems (which started as clones of the Dec Altair: MITS manufacturing couldn't keep up with demand but they shipped a full schematic with every system were using off the shelf parts, so other people bought the parts and assembled them according to the schematic, and then started making improvements).

The company "Imsai" (that's the computer the protagonist of the movie "Wargames" had in his bedroom) convinced Kildall to break his OS into two parts (BDOS and BIOS, Basic Disk Operating System and Basic Input Output System), with the BIOS essentially being a driver package provided by the hardware manufacturers so the same BDOS could talk to disk and console hardware. That way, ALL CP/M machines could run from the same floppy disk, rather than having separate disks for each manufacturer.

All that was 8-bit, and since 1979 Kildall had been chasing multiprocessing (MP/M) as the next big thing (about 20 years too early, the cost of memory was a big limiting factor so at the time running multiple programs in parallel on the same machine wasn't _that_ much cheaper than buying multiple machines and networking them, although S/100 systems didn't have a "motherboard", memory expansion was on cards (which you can keep adding as long as you have slots), and even the CPU was on a card, so having a multi-processor system with two CPU cards wasn't a far-fetched idea, the trick was making it WORK...) so he basically ignored the 16 bit 8086 for the first couple years.

But a guy named Tim Patterson at Seattle Computer Products was working on a new 8086 board which was intended to run CP/M, and since DR hadn't shipped it yet he bought an off the shelf CP/M manual and implemented 16 bit versions of the system calls it listed so he had something to test the hardware with, calling the result "QDOS" (a play on BDOS, Quick and Dirty Operating System).

Tim had previously worked a summer job for Microsoft where he created their first hardware project (an 8080 processor card for the Apple II, allowing it to run Microsoft DOS instead of the one Steve Wozniak had written), and when Paul Allen realized that IBM's project Acorn was basically a 16 bit CP/M machine he and gates threw $50k at Tim (split with his employer) to buy QDOS from him, which they renamed "DOS 1.0"...

8-bit machines all had only 64 kilobytes of memory, and hacks like "bank switching" historically never made much difference. CP/M was about the best you could do on that generation of hardware. Paul Allen thought the PC that IBM was developing could do better and wanted to run Unix on it, so he licensed Unix from AT&T and contracted a small 2-man garage outfit called SCO (the Santa Cruz Operation) to port it to the Intel 8086 and Motorola 68000 processors (because IBM hadn't decided wihich it would go with yet), and called it "xenix" to indicate "we'll port it anywhere IBM needs it to go".

Then when they signed the NDA and got the hardware specs of the original IBM PC (because IBM wanted to put Microsoft's DOS in ROM as the PC's operating system, like the Commodore 64 and so on, and the CEO was on the board of directors of the Red Cross with the mother of William H. Gates III, "trey" to his friends, made an exception for "Mary's Boy". Microsoft was too small to qualify as an IBM vendor normally, that part is in the book "Big Blues" about the history of IBM, by the way)... Anyway, the IBM PC specs read "16k of ram expandable to 64k if you pay extra, and the ISA bus is just the S/100 bus with unused wires removed" (as in there were literally adapters that plugged the bigger cards into the smaller slots, no electrical or timing fiddling required, just shifting wires over)...

Paul Allen went "oh: you're going to run CP/M on it". But he and Gates had already expanded their ambitions to sell a bigger OS to IBM, and Gates said he knew Kildall and offered to set up the meeting with IBM, whereupon SOMEHOW Kildall got the impression that the meeting was in the afternoon but IBM got the impression that the meeting was in late morning, so Kildall was off at the airport flying his airplane (cessna probably?) to cool his nerves, and when the IBM guys unexpectedly showed up at his house (he worked from home) his wife paniced and called Gates who suggested that the company lawyer look over the NDA while Kildall got back from the airport, and as lawyers do he went "ew" and started negotiating terms, and since they refused to sign it as is the IBM guys went away empty handed before Kildall even got back from the airport, and the whole meeting was set back weeks...

Which gave Paul time to contact Tim Patterson and scrape up $50k to buy QDOS and offer to be IBM's "second source" with "their" 16 bit CP/M clone (filing off the Q and renaming it MS-DOS). IBM did the PC after their salesbeings saw Apple II running Visicalc on the secretaries desks when they went to meet with executives in otherwise pure IBM shops, and after allowing Digital Equipment Corporation and the PDP-1 to live (creating the minicomputer ecosystem) they vowed NEVER AGAIN. They estimated they had a year to flood the market and smother Apple before it got entrenched, but an internal process audit had just measured it took them 9 months to ship an empty box, so they had NO TIME to make the new product and get it to market. The head of the Boca Raton department offered to make one out of off the shelf third party parts they could order in volume with a phone call, which is NOT how IBM normally did things but this was an emergency and the CEO personally granted absolution and indulgences to the Boca team. IBM was a monopoly used to squeezing customers, so they carefully made sure none of these new suppliers could ever do monopoly leverage against IBM, by ensuring there was a second source for EVERYTHING, with their one unique contribution being the BIOS ROM (the thing Compaq clean room cloned). They hadn't been second sourcing the software (just the hardware), but hey: good idea! Another CP/M, sure thing.

Meanwhile, Kildall was a Navy instructor before he started Digital Research, which meant he knew about being a vendor to big bureaucratic institutions, and wasn't really keen on going there. It's lots of money, but most of it's pie-in-the-sky someday money after jumping through lots of hoops and years of delay, and to get there you need a dozen full-time staff just to navigate the bureaucracy. His company was a couple people running out of his house: he'd take free money if IBM offered it, but he already had a CP/M ecosystem built around the stuff he was already selling to existing customers, and this new thing was either part of the S-100 family or it wasn't.

So when the PC shipped, CP/M-86 was late, and when it arrived it cost several times what Microsoft priced DOS at. But the real nail in the coffin was that Paul Allen didn't give up on his dream of having this machine run Unix. Each new release (PC, XT, AT) could support more memory, and the 8086 processor could physically address up to a full megabyte. (The DOS 640k barrier was because they'd arbitrarily mapped I/O memory at 10x the original PC's memory capacity: 2/3 for RAM, 1/3 for I/O memory space. You had to move the VGA card's memory window in order to use more contiguous address space in your application, and even then you don't get ALL the space because I/O memory is still needed.)

DOS 1.0 was a bug-for-bug clone of CP/M (well, a 16-bit port of an 8 bit system, but otherwise identical). But for the DOS 2.0 release, Paul Allen added as many Unix features to MS-DOS as he could. You could now use filehandles instead of file control blocks, and stdin/stdout/stderr were filehandles now. He added unix-style subdirectories, although DOS 2.0 used "\" and "/" interchangeably because "dir /s" was how CP/M had indicated command line options, so DOS 2.0 let you use both "dir /s" and unix style "dir -s" with the / version deprecated, but he couldn't quite REMOVE it yet, so the syscalls supported both directory separators. And he publicly announced that a future DOS version (hand-wiggle maybe around 4.0) would just be Xenix with a DOS emulation layer for old programs. You'd need something like 256 to be worth it, and hey: you'd get multiprocessing for free. (Remember how Kildall was doing MP/M? Maybe not THAT crazy. For reference, IBM announced its "Topview" multitasking graphical desktop for DOS in August 1984, and the first version of the Desqview multitasker for DOS shipped in July 1985. If 8 bit systems max out at 64k, a 16 bit system with 128k of RAM running 2 of those 8-bit programs at once sounds pretty feasible...)

The new unix features in DOS 2.0 made it a way better programming environment than CP/M-86, so it wasn't just cheaper now it was BETTER, and CP/M-86 receded from use on the IBM PC. (And clones, Compaq had happened by now. The reason the IBM PC took over the world and the Apple II didn't is that when IBM sued Compaq they lost, but when Apple sued Franklin they won: https://en.wikipedia.org/wiki/Apple_Computer,_Inc._v._Franklin_Computer_Corp. That was the legal decision that extended copyright to cover binaries and thus invented "shrinkwrap" software, see also the 1980 Audio interview with Bill Gates (mp3 and transcript both linked from https://landley.net/history/mirror/#:~:text=1980%20audio ). The GNU project, IBM's "Object Code Only" announcement, and AT&T's post-breakup commercialization of Unix were all responses to Apple vs Franklin...)

IBM's competitive focus on Compaq and the hardware clones distracted it for years from the fact it had lost its second source competition on the operating system side when DOS 2.0 rendered CP/M-86 irrelevant. IBM shipped its own PC-DOS and Digital Research eventually came out with DR-DOS, but by then Microsoft was doing "CPU tax" contracts with motherboard manufacturers (see the 1995 antitrust trial under Judge Sporkin), and used aggressive bundling (buy X get Y for free, and you can't NOT buy X) to promote Windows and Office... But I'm getting ahead of myself.

Two things happened to derail the dos->xenix move:

1) the IBM PC/AT (developed in 1983, shipped August 1984) added a hard drive, so the DOS 3.0 release was mostly about adding hard drive support (the C: drive) rather than furthering the convergence with Xenix.

2) in 1983 Paul Allen came down with Hodgkins Lymphoma. (That's the same cancer Hank Green just got. It's one of the most treatable forms of cancer, but it IS cancer, and can totally kill you).

Nobody initially knew WHY Paul Allen was so sick (looked like overwork during the DOS 3.0 crunch), but Paul Allen owned 1/3 of Micorosoft's stock because Bill Gates was an asshole: they original wrote DOS for the MITS Altair, and the owner of MITS offered Paul a job working at MITS. When incorporating Microsoft Gates insisted he have 2/3 of the stock and Allen only 1/3 because Gates would be working at Microsoft full time and Allen only part time due to his job at MITS, and Allen agreed... and then immediately after that was signed, Gates asked Allen if he could get him a job at MITS. As I said: asshole.

But the ultimate asshole move was that while Paul Allen was working himself to death trying to get DOS 3.0 out fast and clearly sick but not yet properly diagnosed, Paul heard Bill Gates and Steve Ballmer (Microsoft employee #3, Gates' old poker buddy from Harvard Law School before they dropped out to work at Microsoft) talking to each other in the next room about how to get Paul Allen's 1/3 ownership of microsoft back when Paul died. They didn't want it going to his family, they wanted to figure out how to take it back.

When Paul Allen took a leave of absence to get cancer treatment, he never returned to Microsoft. The drive to switch everything to Xenix left with him, and Gates looked around for other people to copy technical agendas from instead. He saw the Apple Lisa (because Apple gave them an early unit to port their application software to), and tried REAL HARD to copy it but Windows 1.0 and Windows 2.0 were just pathetic. DOS 4.0, 5.0, and 6.0 offered nothing that DOS 3.0 hadn't. Gates teamed up with IBM to work on OS/2 which was IBM's attempt to port mainframe technology down to the PC space... alas, targeting the 286 instead of the 386.

IBM had bought the entire first year production run of the Intel 286 processor to keep it out of the hands of competitors (like Compaq), and was then stuck with a warehouse full of the slowest, most expensive, rapidly depreciating 286 processors ever made. That's why they refused to go to the 386 and even the IBM PS/2 was mostly 286 chips, they were trying to unload that backlog of 286 chips! (They eventually landfilled some portion of them, but it took YEARS.) In 1987 the Compaq Desqpro 386 was the first 386 PC because IBM the 386 had been out since 1985 and IBM still hadn't used it, and Compaq got tired of waiting. (As did IBM's customers.) So yeah, that's why OS/2 was so far behind the times that Windows 3.0 could get out ahead of it and establish a new programing API standard.

When David Weise made windows work years later on his own and against orders, the first person he showed it to thought he'd get in trouble for it because Microsoft was focused on OS/2. Microsoft never had a plan, they had a monopoly that let them fail repeatedly until they got lucky. Their "CPU tax" monopoly contracts forced manufacturers to License microsoft products for entire "product lines", meaning PC manufacturers who wanted to ever sell a Microsoft operating system on ANY machine had to put them on EVERY machine. They couldn't sell even a small number of machines without the preinstalled microsoft software, and microsoft fought a marketing campaign for years against "naked machines" because obviously the only thing anyone could do with a machine that DIDN'T have microsoft software on it was install pirated microsoft software. Microsoft's monopoly leverage also let them prevent other operating systems from being installed alongside theirs, and when Windows 95 came out they extended this to preventing IBM from installing OS/2 on any of its own PCs if it wanted any access at all to Windows 95. (See the 1998 antitrust trial with Judge Jackson.) But again, getting ahead of myself.

The death blow for Xenix was that after the 1983 AT&T breakup, when AT&T was commercializing unix, it sucked in code (without attribution) from all the third party unix variants and shipped it in Unix System III. (System V was a successor to System III, there was a 4.0 but it never shipped to customers.) This is why the AT&T vs BSDi lawsuit ended favorably for BSDi: they were able to prove in court that AT&T had sucked in THEIR code without attribution, and thus forced a settlement on AT&T. AT&T also did the same thing to Xenix, and when Gates found out Microsoft code was in an AT&T product without permission or payment he went BALLISTIC, but didn't think he had the legal heft to take on AT&T so instead he purged Xenix from Microsoft (it had been running their internal email system and so on) and unloaded Microsoft's interest in Xenix on SCO (which is how SCO wound up fully owning Xenix, they'd intially just been a subcontractor doing work on somebody else's IP, but they got it cheap), and basically developed a Dave Cutler level of Unix hatred going forward...

I note that back in the day I did a LOT of research on this for my rebuttal to SCO's second amended complaint against IBM, and xenix is all through it. The indented parts in green are mostly stuff I wrote, with a little bit from Eric, but the OSI position paper was his baby and the rebuttal paper was mine. The rebuttal links to a lot of primary sources, many of which have sadly gone away over the years but you can still pull most of them out of archive.org if you try...

(You should TOTALLY get a copy of Peter Salus' book "a quarter century of unix". And a copy of "Where wizards stay up late" which is about the formation of the internet. Soul of a new Machine and A Few Good Men From Univac are more tangential, but loads of fun.)

Oh, and the book "Hackers" by Steven Levy is the other half of this Ken Olsen Smithsonian interview, literally two halves of the same story with the TX-0 and so on.

Oh, and the first four interviews in the Intel section of my mirror are the four parts of the story of the birth of the microprocessor: Ted Hoff (the actual creator), Federico Faggin (who went on to found Zilog and the Z80 processor), Masatoshi Shima was their actual customer at Busicom (who many people say was the ACTUAL inventor of the 4004), and then their boss was Gordon Moore (of Moore's Law fame).

Then read "Crystal Fire" about the invention of the transistor. The second half of that book is about the creation of Silicon Valley (which exists because William Shockley was an utter asshole), and Gordon Moore is a featured player (part of the "traitorous 8" that bounced from Shockley to Fairchild to found Intel)...

Ahem: computer history is a hobby of mine. Here's a 2 part writeup (part 1, part 2) on some interesting plot threads I did a dozen years ago.

(I've been meaning to write my own book for years, but... too busy.)

Sigh, fell out of the habit of blogging during the week when I couldn't. (Nothing for my editing pass to elaborate on when I didn't leave myself a trail of breadcrumbs...)

Git log shows couple of shell fixes. I should get a release out then do a deep dive into shell stuff again, and try to get that properly finished.

Back on the horse. (For a definition of "horse" that involves taking my new laptop to the common work area in building 1 of Fade's apartment, which is playing a spanish cover of "Achy Breaky Heart" for some reason.)

The 'repeated hang" failure mode left me with a lot of vi :recover files where it prompts me which of the three .swp files to read, and I'm just zapping all that. There's a lot of pausing to stare at "am I deleting the .blah.c.sw? file or the blah.c file" before each one JUST TO BE SURE. (I have made that mistake. Less of an issue when the file is in git, and I'm just losing recent changes instead of trying to dig it up out of a USB backup drive.)

Sigh, the hard part of fiddling with a command like ulimit/prlimit is A) coming up with the new help text, B) coming up with test suite entires. Once I've got those, the CODE is generally pretty easy. Implementation is seldom the hard part, DESIGN is the hard part. What should it DO?

New laptop arrived. The freeze problem advanced to "happens 30 seconds after a reboot", so I ordered another of the same type I could just swap the hard drive into. (I have 3 such spares at home, but they're in Austin and I'm with Fade in Minneapolis.)

It's so CLEAN. Not covered in scratches and gunk, almost as if I HAVEN'T been dragging it around with me everywhere for a couple years. Same model (Dell E6230) but this one's refurbished and thus in a slightly different case (doesn't say Dell E6230 on it for one), and with this case I can't see the charge/disk LEDs with the lid open. Seems like a tiny thing, but kinda significant now that I'm confronted with its absence. Yeah, there's software versions up in the toolbar (which I have configured to only be visible with the mouse hitting the top of the screen), but I don't TRUST the software ones. I wouldn't have a band-aid over the laptop camera if it had a physical LED that lit up when it was powered independent of any software. The fact they refuse to do that stuff is why one of the first things I do with any new laptop is stick a band-aid over the camera. The pad protects it for when I want to use it, and when I don't it's NOT LOOKING AT ME. Grrr.)

I tried borrowing Fade's old macbook during the gap, which was a comedy of errors in and of itself. She dug it out of the closet, confirmed it worked, set it to charge on the counter, and went to work. I opened the lid to be confronted with a login prompt. Ah. Day 2: armed with the password I tried to ssh out to a linux machine to do some work and... none of the ones I can think of are configured to allow password, they're all key-only. (I have backups of everything... in Austin.)

It's pretty late in the day by this point (shipping estimated the new laptop would arrive yesterday, instead it came in after 3pm today), and by the time I'd rustled up an appropriate screwdriver and got the hard drive swapped and network access sorted out (registering the mac address with Fade's apartment's wifi... gave me an intercept screen asking me to log in? Seems redundant somehow. Oh well, phone tethering still works...) it's after 5pm. Old machine still has the bigger memory but I'm making sure this is STABLE before swapping more parts than strictly necessary. To be honest it's possible I could have fixed the old one with a can of compressed air, but I haven't got one here and am not entirely sure where to buy one (target?), and the hang problem going away and then coming back again is how I _got_ here. I want reliability, please.

Via the phone tether I'm downloading SO MUCH EMAIL... (Gmail's pop3 does about 1 message per second in 250-500 message chunks. Between linux-kernel and qemu-devel and so I, I get well over 1000 messages a day. This is likely to take a while...)

On the bright side, the time off probably gave my eyes time to adjust to the new glasses. (The myopia is the same, only the asigmatism has been changed to protect the innocent.)

Wrote up yesterday's broken tooth while email downloads. Not gonna backfill the rest because I didn't do anything of note and don't remember what most of it was anyway...

Still backfilling: this is the day I broke a tooth. Molar all the way in the back, bottom left side, next to where I got a wisdom tooth removed years ago. The tooth itself doesn't hurt, the magic japanese toothpaste is quite effective. Hydroxyapatite deposits more calcium phosophate on top of any exposed dentin and keeps the nerves protected behind bone equivalent... but it does nothing about the enamel, a large chunk of which is what broke off here, leaving a sharp pointy bit that's stabbing my cheek. The cheek hurts a LOT.

Regretting not getting to a dentist while I was in Japan, but while I trust the medical providers over there FAR more than the ones in the USA... there's still a language barrier, and my teeth are in terrible shape due to the extensive dental work I paid thousands for back in 2013. (The 6 month apartment I had for the Cray contract in St. Paul was right down the hall from a dentist, and I used them as a second opinion to say "yeah, those two front teeth that got chipped before they even came in because of that car accident when you were 5 years old smashing your baby teeth up into your gums so they've got grooves on the front? We we ALSO want to just drill all that out and turn it into fillings because it's weird and we're calling that cavities even though you yourself can't detect them in any way." So I went with it, and all the fillings they put in chipped to pieces and fell out entirely over the next 18 months, leaving me with large obvious holes in two front teeth. I paid a lot of money to get those holes, and felt really silly about it, but regular application of japanese toothpaste meant it didn't hurt and did not appear to be getting worse...

And now I need to wrestle with US dentistry for a _different_ problem. Dowanna.

The "battery charging while using laptop" problem is getting worse, I just had two reboots (well, freezes forcing me to reboot) in half an hour with nothing plugged into USB, and while basically just typing in a text editor and no cpu-intensive anything pulling power.

May 19, 2023

I originally had this as "April 31" but then my python RSS feed generator went "boing" parsing the date, because there isn't one. (Midnight according to my laptop is in the middle of the day in tokyo, it's a bit fuzzy which day I'm writing for over there...) So I moved it here because I didn't write a blog entry today, due to ongoing travel recovery:

Somebody in email said "Canada as a whole seems to be determined to be a branch plant operation of US-based multinationals," and I replied:

This too shall pass. Maybe not fast enough to benefit either of us personally, and when both the roman empire and the british empire receded they left behind a lot of scar tissue, but no empire lasts forever, and the USA is already pulling back from the world now we're a net exporter of oil (have been since 2019) and thus don't really care about having a trillion dollars of navy policing everybody's shipping quite so much anymore. We've still got aircraft carriers, but they can't be everywhere, and we've gotten rid of most of the smaller patrol boats that used to _be_ everywhere...

It's hard to take solace in bad things happening, but the currently powerful aren't going to stay powerful forever. The USA is facing the end of the boomers (1946 was 77 years ago, their refusal to hand off anything gracefully is going to cause a LOT of loss of institutional continuity), climate change (Houston's flooded twice more since hurricane Harvey, but "again" isn't as newsworthy), the exhaustion of the ogalalla and california's central plains aquifers underpinning the majority of our agriculture (don't get me started on crop monocultures), the collapse of the US health care system coinciding with the rise of antibiotic resistance, a dozen kinds of invasive species (Texas has "crazy raspberry ants" that are attracted to electromagnetism and will thus ball your wifi router), the reshoring of manufacturing (if our trillion dollar annual defense budget stops paying for the navy protecting container ships "for free" then floating everything from the far side of the world instead of mexico gets a lot more uncertain)...

Canada has its own issues to work out, but the worldwid fascist crazy should recede with the Boomers (they got BADLY poisoned by airborne lead in the gasoline for 50 years, and in 2/3 of who's left it's combining terribly with senility to go past "kids these days get off my lawn" into flat earth territory). Once that's past, then maybe the wretched survivors can start shoveling out. (Step 1: universal basic income, which yes will only happen over the Billionaires' dead bodies. So maybe UBI is step 2.)

Personally, I would like the new victorian prudishness and the Fosta/Sesta (Comstock Act II) nonsense to stop being imposed on every other country in the world, along with the USA's tendency to treat children as non-persons. In Japan quite small kids buy stuff in the store and go on the train by themselves. In Europe kids can have wine or beer at dinner as soon as they can walk. In the USA we look back in horror on the days of "latchkey kids" because now they're non-persons legally confined to a building every day where they go through metal detectors and are watched over by police with live ammo who randomly search their bags and lockers, they can be arrested and permanently removed from any family that allows them to be alone on the street two blocks from home. It's "for their protection" that they can't work or vote or drive, if a teenager sexts a naked selfie to another teenager they can BOTH wind up on a sex offender watch list for life, which is a change to the law the Supreme Court only made in 1982 by the way. Ronald Regan hijacked the federal highway funds to force states to raise the drinking age from 18 to 21 (after vietnam lowered it since Johnson and Nixon were drafting 18 year olds to die, so they-who-were-about-to-die protested their way into being treated like adults in other areas, which involved "the man" gunning down protestors).

So yeah, rooting for the american empire to collapse. When France invented the guillotine they had to work through Robespierre and Napoleon (liberty, equality, oh look a rich white guy has siezed dictatorial power again, rinse repeat), but it worked out for them in the long run, and they're currently braving tear gas to push back against late stage capitalists who want cheaper and more obedient servants. "We can't afford this" means you squeeze the rich harder. We went to the moon WHILE fighting the cold war, whether or not we could "afford it" wasn't the big question.

I would love to reach a point where I could take solace in GOOD things happening. Not just looking forward to the end of bad things and hunkering down to minimize the inevitable collateral damage...

Flying back to Minneapolis.

Sigh, this laptop power supply with the intermittent data connection works fine when I'm not trying to charge the battery _and_ use the laptop at the same time. Suspend it, charge it up, then keep it plugged in and use it: great. But when I don't do that, it has three obvious falure modes: 1) power supply gets VERY hot (the first time I noticed the _smell_ of volatile plastic compounds becoming airborne), 2) laptop toggles every 15 seconds or so between charge and discharge (which can't be good for the battery, and has corresponding screen rightening and dimming poer management weirdness sometimes), 3) if I forget and try to charge a USB thingy (such as my bluetooth headphones) while it's also charging the battery, the laptop freezes solid. (Probably a kernel panic that doesn't get marshalled through X11 into actually showing me the panic or even THAT it paniced. The kernel guys have been throwing functionality overboard like hot air ballons dropping sandbags, and one of the things they gave up on a while ago was "go into VGA mode or framebuffer and dump text to the screen when you get a panic". Because who cares about THAT?

I note that the OLD battery, which was smaller even before it lost 1/3 of its capacity due to age, had much fewer problems because I think its maximum charging current was lower (fewer cells). The battery charging logic goes "aha, I can feed THIS much power into the battery" and when the controller can't interrupt that and go "I said I could deliver this much power but I'd like to take it down a notch now" because the data line's gone walkabout again, Dell's charging logic goes all pear shaped.

Anyway, I tried to charge my laptop and phone before getting on the plane, laptop went suddenly catatonic and had to have the power key held down until it turned off, so I've lost all my open windows again. Sigh. Right as I was getting into a position to dig out and address the backlog.

Plane is 100% crowded again, and they made the seats smaller (again!). Front to back AND side to side, I'm trying to use laptop at awkward angle on the TV tray but it just plain doesn't FIT. And while I theoretically have a power outlet (I assume that's what the green LED on the seat in front of me near my right ankle is about), I can't see it well enough to actually plug into it. They turned the cabin lights off because it's an overnight flight, and my phone battery is fully dead so I can't use its flashlight (well I didn't really get to CHARGE it in the airport, did I?). The overhead reading light doesn't make it down there...

Not _much_ makes it down there. Air Canada added seats to their planes since pre-pandemic times, and reduced flight frequency so every international flight is 100% full. 13 and 1/2 hours in a space too small to pick up anything that fell onto the floor (unless I can hook it with my foot, I'd have to ask the guy next to me to get out of his seat, which means the person next to HIM would have to get up and stand in the aisle). The accumulated muscle cramps from being unable to move is not pleasant. Add in the usual 6pm tokyo departure time and the sleep deprivation is... getting unpleasant.

Dunno how much of this is Air canada and how much of this is post-pandemic late stage capitalist capitalist profiteering, but the days where I wrote most of the ps.c infrastructure on a flight back from Japan seem long gone.

Jeff wants me to meet Mike today and talk about future plans. I really, really, really don't want to, but there isn't a graceful way to back out of it. I strongly suspect they're going to try to pressure me into making more of a commitment to Jeff's company. (I'm not signing anything before Fade can read it. I also have no idea if Google wants to continue the toybox funding beyond what they've already done, but I'm not moving on from that until I've done all I can there. Jeff's project does not take precedence over MY project.)

Jeff hates when I say I'm working on "his projects", and insists that it's "our project". He has a "vision" that he's upset he hasn't ben able to explain all of to me because I keep getting derailed into practical things we need to do, but I'm not interested in infrastructure in search of a user and a big strategic goal that can't be concretely implemented. We've worked on and then left half-finished a dozen different pieces of technology. I care about what can get completed and put in the hands of users. If Jeff had funding for us to spend 5 years focusing on Basic Research in the vein of bell labs or xerox parc, great. But we don't. And I got enough swap-thrashing on toybox, thanks.

Still, I'm learning interesting stuff I didn't know before. And I'm FINALLY getting to the point where I know a LITTLE japanese. There have been anime dialog scenes where I followed multiple consecutive sentences! Yeah, ok, simple ones, but once there were FIVE sentences, in a row, that I understood almost all of. Alas, in actual interactions with japanese people, knowing "this is the point where the person running the cash register asks me if I need a bag" is still far more useful than my ability to parse the words...

The OpenLane git checkout is about 2 gigabytes. We went on a "deleting stuff we can prove isn't needed" spree... and the result is 2.1 megabytes of actually needed scripts. The .git directory is over a gigabyte, full of old long-deleted churn. They also checked in every project that's successfully built against this thing INTO THE OPENLANE REPO. (Remember when uClibc had a test suite containing every package that had ever successfully built against uClibc, and the invocation necessary to make it work in the new context? That test suite turned into the "buildroot" project. Well OpenLane has something similar, and it's FAR BIGGER THAN THE ACTUAL PROJECT.) This is pufferfish territory, the project is making itself look big, but once you cut through the cloud of squid ink there's not actually much there.

I need to fix up the toybox shell because people are using it, which means I need to finally add the command line editing and history (without which it seems way less finished than it is because monkey brains conflate user interface polish with functionality, and yes that includes _me_).

Command line editing is adjacent to the chrunch_str() logic in exactly the same way fold() is, namely that "backpace eats how much" is the big missing piece of both. Which is a nonobvious question to answer because the HARD part is tabs advance by a variable amount based on where they started. (Also, nonprintable characters are TRAILING, which is the dumbest thing the unicode committee ever did. A printable character does not FLUSH pending nonprintable characters, the printable character comes first and is them modified by following characters, being REDRAWN ON THE SCREEN multiple times in some instances, which also means you're never sure you've finished a stack of combining characters until you've read PAST it and gotten a character (not byte, utf-8 sequence parsed to unicode point!) that is NOT part of this one, which you then need to unget and process seprately in the net go 'round the loop. When you've got a string fragment, you CAN'T know. It could end in an unfinished utf8 sequence. There could be a combining character following it. Pretty much the only thing that DOES tell you it's done is newline... and then what does combining characters at the start of a line MEAN exactly when there's NOTHING FOR THEM TO COMBINE WITH? (What, do they combine with an implicit NUL? What would that mean? The last newline has an umlaut!) It's REALLY STUPID because they did it BACKWARDS.)

But it's what's there, so we must cope.

I keep wanting to do a "how to bloat code" presentation, starting from the classic K&R "Hello World" and then shifting over to C++ with accessor functions doing exactly the same job but passing the data between all the various "enterprise" style contexts, demonstrating the full range of "it's not code reuse the first time" nonsense, and generally showing that you can have a very large amount of infrastructure that LOOKS like it's doing something but isn't really.

I'm reminded of this becuase the skywater toolchain turned their README from a text file into one of those "markup that generates HTML" things, yes of course it has stylesheets, and along the way they added a lint variant to check the validity of their markup, then of COURSE they factored it out into a subrepo which does a "git submodule update" at build time. (So you check out the repository, and then when you run the build it checks out more repository within the build.)

Remember: this is a README. Historically, this was ONE SMALL TEXT FILE. There's no build infrastructure for a text file. None of this is NEEDED, but this crap it's metastasized into is pulling in a chunk of another one of Mithro's projects, which is doing dependency checking against the host to make sure various packages and versions are installed... except it's checking RPM and we're running it on a Debian system so it's not finding the magic red hat package names out of the wrong kind of repository. And since they did a "-include" this whole mess should just DROP OUT if the repository isn't installed... but they're installing it WITHIN the build.

Someday I should do a proper writeup on why Google's OpenLane project stalled...

tl;dr: OpenRoad is a DARPA funded initiative, which works. OpenLane is fundamentally a small number of shell scripts that call the OpenRoad tools to do their thing in order, with reference to the Sky130 PDK which is basically "fonts and CSS to make a mask for this fab". (A fab is sort of a really high end printer. We're submitting a job to it. The job is literally a big data file.)

OpenLane is a partnership between Google and SkyWater (which used to be Cypress Semiconductor) to create an open toolchain for Sky130. Google hired the guy who did QFlow to work on the tools part, and then subcontracted much of the fab integration work to a company called "Efabless" which has an existing business taking people's design files (mostly in Verilog) and converting them into something the fab can accept. Which means if Efabless were to succeed at what Google's paying them to do, it would undercut their existing core business. There are two big projects here: OpenLane is a set of control scripts that call the OpenRoad tools in the right order to perform tasks, and the other is "Skywater PDK" which is a data dump from the fab. Tim Edwards is running a giant pile of fixup scripts in the Skywater PDK build because the fab's data dump is horrible (there's like... OCR errors in it or something?) And the resulting PDK is subtly broken half the time, although somebody found that if you run make TWICE, the result is usually good after the second time. (But the only way to determine if the result is good is to build the REST of the toolchain around the resulting PDK, then build your project with the resulting toolchain, then test result. Which is time consuming and labor intensive.)

The guy at Google running this is Tim "Mithro" Ansell, who is writing his own build system to do some portion of all this, except he doesn't seem to have done this before in a nontrivial way so doesn't really know what success looks like? He's a fan of the concept, but not a veteran. Jeff (who has done this before) keeps telling him "you need to do this" and getting dismissed as silly, and then 6 months later they realize they need to do what Jeff was telling them. Kinda like the RiscV guys, really...)

So Mithro grabbed chunks of his symbiflow project and stuck them to the Skywater PDK builder, specifically he's grabbed anaconda, which long ago used to be Red Hat's system installer. It was the large python program that would run when you booted an install CD (or floppies) that would partition and format your disk and let you select what type of Red Hat system you wanted to install, and would then install all the packages. This was back before Fedora and Enterprise happened, they replaced it with something else rather than rewrite the large pile of Python 2 code in Python 3. But the old 1990s Red Hat system installer seems to have spun out into its own project maintained by yet another proprietary company producing source-under-glass that you can see but would be crazy to try to build or install yourself) and is using it to confirm prerequisites are available in the local RPM repository. On Debian systems that don't use RPM, this doesn't find much.

Specifically, it's complaining that "yosys" isn't installed. It is, and it's in the $PATH, but since Mithro's slurped-up symbiflow plumbing that's installing a proprietopen fork of Anaconda didn't install it, it's not finding it. If it just tries to call "yosys" it's there, and presumably "yosys --version" might say if it's new enough, but instead Mithro/symbiflow/anaconda/openlane runs a large pile of python 3 which returns an incorrect answer.

Note: it doesn't have to do ANY OF THIS AT ALL, becuause if yosys isn't there then you should get an obvious build break where the last line of output is an attempt to run yosys and getting a file not found error. This is basically the "assert" problem where the bug IS THE EXISTENCE OF THE ASSERT.

It looks like if you remove that whole subdirectory, the enclosing makefile should just work becuase it has - before the include to skip the nonexistent file, and then the $(wrapper) variable drops out and it just calls the rest of command line. Seems worth a try, anyway. So I'm trying to remove the git repository, and I chopped the makefile target that clones it out.

Except he didn't just clone it there, he added it as a submodule, which means it's getting cloned already. So I need to remove it from the Makefile AND remove it as a submodule from the parent repository. Except the makefile of the parent repository is checking it out. (Remember yesterday's "how do I remove a submodule"? Because me not checking it out didn't prevent THIS from checking it out.)

One of the subrepos needs to be patched, which means we check out our own copy and tell the build where to find it. And there's even a make variable for this! Except autoconf is marshalling data from variable to variable, and if you track it back the top level configure is setting it with a hardwired path.

The whole project is like this. They keep making layers of infrastructure and then hardwiring it to do specific things. There are obviously multiple teams working at cross purposes here, and the PROPER fix would be to RIP OUT all the stuff that is we can PROVE is not doing anything. But you don't show progress in a fortune 500 company by REMOVING code. Code has a dollar value attached, generating more of it is always progress. Code gets depreciated and amortized, not _deleted_. Deleting it costs MONEY. Creating more is profit! IBM's KLOCS and so on...

We, on the other hand, are trying to get something to WORK.

So git grew a "fatal: detected dubious ownership" error whenever you cd into another user's directory and try to "git log" a repository. Not a "warning", but "I stubbornly refuse to perform the requested operation". So far the only fix is to sudo and run git as root, where it doesn't care about permissions.

That's really stupid. Barfing this way when WRITING to a repo is one thing, but I'm cd-ing into another user's directory and trying to "git log" and "git show" individual commits there. (I could tar the repository and extract a copy in my home directory so it all belongs to me, but that's deeply silly. And inconvenient.)

I don't know if google has deteriorated to the point it can't find it, or if there's no way to fix git other than to build it from source with this test patched out. Luckily, there's part of a git implementation in toybox, and this would be a reason to finish and use it.

In the meantime, it makes debugging a build that runs as a different user extra-annoying... and even more brittle than I thought? Darn it, the sudo workaround isn't load-bearing: if I do an "env -i PATH=$PATH git log" as root, I get the "fatal: dubious ownership" abort again. It's something about HAVING RUN SUDO that git goes "oh well if you really mean it". Actually doing so AS ROOT is not allowed by git. (I mean, I could destructively "chown -R root:root .git" but then the original user couldn't use it. Before finding out git was treating sudo as magic, I was thinking the right thing to do here is create an LD_PRELOAD library that wraps stat() to patch ownership to always equal getuid(), but even that won't fix it?)

I am so tired of myopic git developers. The reason I stopped maintaining kernel.org/doc is when kernel.org had a breakin (because one of the devs was ssh-ing in from a windows machine, and once you'd logged in the server wasn't that secure internally across users) they locked the barn door after the horses had escaped by removing generic ssh support, including the ability to rsync over ssh. I pointed them at a way to make ssh explicitly call rsync, with forced prefixes and everything, but they weren't interested: they'd homebrewed some horrible wrapper tool that ONLY let ssh run git (nothing else), so to update the website I had to check everything into git, including things like the gigabyte video file (USB driver writing tutorial) they'd removed after the breakin which I wanted to put back online. If the file then moved elsewhere (it was eventually uploaded to youtube) it would STILL be taking up space in the .git directory both in my local copy and on the server, forever. But they were deep in "If all you have is a hammer, everything looks like a thumb" territory, and I gave up and moved on...

Aha: I asked on the #git channel on freenode, and the magic invocation is "git configure --global 'safe.directory=*'", and they pointed at a reference for why the stupid happened. (And they confirmed it's checking SUDO_UID, which is just wrong.) Yay, found a way to make it stop.

Long argument with somebody online who claims that statically linking an initramfs into the kernel is weird, and that in 20 years of messing with Debian and Ubuntu they've never encountered it, so obviously nobody does it because their experience is universal. And apparently contradicting them was considered insulting. (Insisting that having used debian and a derivative of debian makes your experience universal, thus everyone else is weird, was apparently NOT insulting. Go figure.)

I can't link to it because it wasn't cc'd to a mailing list, only to Jeff. Mike texted to yell at me about "burning bridges", which means Jeff forwarded it to him. I'm also informed that Jeff has apologized to them on my behalf, which was not something I'd asked for (or been aware of at the time).

A lot more "huh, Google can't find this" instances during said exchange, which kinda undercut my point that what I was doing is not unusual. The deterioration of Google is getting alarming. But I did manage to at least dig up a few interesting numbers, which I can cut and paste here:

My perspective is skewed. Not just because I'm the guy who wrote the initramfs documentation in the kernel back in 2005, but I also maintain the command line utilities of Android and used to maintain the command line utilities used by Linux routers. That means I hear from those communities a lot, and they're orders of magnitude bigger than desktop Linux. That's not hyperbole: this page estimates there are 33 million active Linux workstations, and 1.6 billion active Android devices. Add in about ~750 million routers of which around 91% use Linux and "somewhere over 500 million" seems a reasonable guess, bringing the embedded total from just those two sources over 2 billion active installs.

So I regularly hit things that are "weird" for 33 million installs and "normal" for a couple billion. It's hard for me to convince the developers who make those billions of devices to show up even briefly on linux-kernel because they got tired of being called weird, and being seen as pushy when they try to explain. And if they won't show up, out of sight out of mind. (They think _I'm_ weird for still engaging with the kernel community at all.)

(I didn't even go into the PC hardawre space, where Red Hat claims to have a 33% share of the "worldwide server market" although that's in terms of who's paying for their OS, not installs. In terms of seats, all conventional Linux distros together are collectively 2.1% of desktop installs, behind ChromeOS at 2.2%. Windows is over 74%, Mac is 15.3, neither NOTICES those two. And in the PC "cloud" space... it's still Windows at 72%.)

Working on toybox stuff today instead of Jeff's thing, but I no longer feel safe huddling in my hotel room (and they're cleaning the room today anyway), so I went to the Hello Office, and when Jeff arrived he got mad that I didn't immediately stop working on the toybox thing and start working on his thing instead, and he left abruptly and angrily. I took the train to Akihabara to try to find a coffee shop there (it's the other part of tokyo I'm familiar-ish with), but didn't bring an umbrella and got caught in a rainstorm. (I have an umbrella in my hotel room and we found FOUR cleaning up the hello office, I am not buying ANOTHER ONE). Holed up in a random not-mall space, but it didn't have good seating to use laptop with so I mostly watched stuff on my phone. Eventually a lull in the rain let me take the train back, except I was most of the way to Shibuya before I realized I was going the wrong way down the Ginza line. The train was too crowded to pull out laptop there either. Got back to the hotel eventually.

Yesterday's entry was too long so I moved the battery tech description here. I have actually learned a lot about battery technology this trip:

Step 1 was to stare at boards taken out of the OLD system being salvaged and repurposed. (Well, diverted. The batteries were ordered for another project but never actually installed. The pandemic messed with shipping logistics or some such. I think they were going to be used at a wind farm in another country, but didn't make it there?) The existing system has thousands of batteries in over a hundred big cube things, each cube is sort of an industrial garden shed meant to live outdoors. It's a proprietary Chinese design made from imported western chips that require an NDA to get programming specs for from the (german?) chip vendor to figure out what we can salvage and what we have to reimplement. Each battery management board attaches to a case containing 52 Lithium Iron Phosphate prismatic cells: big blue rectangles with two terminals on top like a car battery, each roughly 15x10x4 centimeters and weighing a little over a kilogram.

The resulting pack of 52 is in a big (aluminum?) case, kind of a horizontal silver version of the Monolith from 2001, which is too big to fit in the elevator to the office, and most of them are a 2 hour car ride away anyway. (Mike has a car but Jeff and I don't. The guy who salvaged the batteries bought a plot of cheap land out in the countryside to store them. It's not anywhere near a train line, and does not have much in the way of hotels either. You know the parts of rural japan that have a lot of abandoned houses and entire towns with no one younger than 65 because of the declining birth rate? Yeah that. Jeff and Mike went out there and fetched stuff a few days before I flew in to Tokyo, but did not bring back an actual battery. Just an assortment of easily removable electronics and lots of photographs.)

So terminology: "battery" is a collection of cells, and "cell" is an individual anode/cathode pair (with electrolyte and separator), in this case those big blue rectangles ("prismatic cells"). The battery cells are wired in series because Lithium Iron Phosphate chemistry produces 3.2 volts (plus or minus ~10% depending on how charged the cell is; the voltage rise/drop is actually how you tell when you're done charging and discharing the battery). So each prismatic cell holds a LOT of power (over a hundred amp-hours) but produces a tiny voltage. Wiring them in series adds up the voltages of each battery, so 52 x 3.2 = 166.4 volts, at a LOT of amps. (Each monolith is roughly like a Ford E-transit battery I think? Same ballpark anyway, I don't have the numbers in front of me.) And each cube has a couple dozen of them: it was a VERY big battery farm.

So the board attached to the front of each of these 52 battery monoliths has four AFEs, which stands for "Analog Front End". It's a big analog to digital converter that measures voltages, and each AFE has a 28 pin connector hooked up to it through a zillion little resistors and capacitors. Those pins come from the batteries, the general idea is to have a connection before/after each cell so you can measure the voltage put out by just that one cell, and if it's higher than it should be while you're charging the battery, you can route current around it through the same pins so it doesn't charge up as much as the others in the string. Except the AFE chip can only divert like 1% of the charge current around the battery, so it's just a LITTLE bit of balancing, but it can happen each time you charge, and you can choose to stop early on either the charge of discharge if some cells are hitting an end stop and others aren't yet, sacrificing collective capacity to avoid damaging any of the individual cells.

This is how all battery management systems work, people do youtube videos about this. You start with balanced cells when you assemble the battery pack, and then do a tiny amount of balancing each time you charge them to _keep_ them balanced. If they're all from the same production run and have been linked together since, they should only really get UNBALANCED due to slightly uneven heating. But the home users making their own battery walls mixing and matching scavanged cells with very different origin and history, their battery management systems will have a LOT more work to do, and may not be able to keep up. Hopefully not an issue here.

So anyway, this chinese design is 4 copies of the AFE chip vendor's reference design board glued together (literally; they sell it on their website and it looks _identical_), with a different 5th board on one end (haven't found what they copied that from yet), and then the whole thing laminated under at least a milimeter of plastic to keep moisture out (and maybe electrical insulation). The 4 AFE chips are daisy chained together talking some SPI/serial variant, and then for the 5th board the SPI connection goes to a microcontroller. The microcontroller is from the same company that makes the AFE chip, and it's more or less a motorola 6800 from the 1970s with a bunch of SRAM and flash bolted on.

The 6800 is an 8-bit predecessor to the 32-bit m68k from the Amiga and Macintosh. The MOStek 6502 in the Commodore 64 and Apple II was to the Motorola 6800 what the Zilog Z80 was to the Intel 8080. In both cases, engineers who worked on the earlier one left to form their own company. So a 6800 SOC with 128k of sram is roughly equivalent to a Commodore 128, albeit clocked a bit faster 30 years later.

So that 5th board section is the controller, and at the far end of THAT board is a CANBUS connection to the outside world. CANBUS came from the car industry and is also used in manufacturing automation. The problem is, CANBUS is just a "read value from address, write value to address" protocol that tells us nothing about what's being said. The chinese manufacturer's board design is all under NDA (we recognized the AFE reference board they copied because there's a picture of it on the chip manufacturer's website), and the proprietary-is-good chinese company built their stuff out of chips that are all themselves under NDA. (There are perfectly good non-NDA AFEs on the market, but that's not what they chose to use.)

Even if we felt up to reverse engineering an assembly dump of the biggest program that could fit in a Commodore 128, we can't get at it because this NDA SOC has a fuse you blow to prevent reading the flash back out. (What this system is doing is generic and well understood, and the patents on Lithium Iron Phosphate batteries themselves expired last October (although that's mostly about manufacturing more cells, not managing them). But every design decision in the electronics has been about obfuscating what they're doing to protect largely nonexistent intellectual property. They want to SEEM unique and magic because you can't tell what they're doing, or interoperate with any of their existing stuff to repurpose it.)

So EITHER this control SOC is a dumb translator passing on the AFE info to whatever is at the other end of the CANBUS connection, OR this is where the battery management program lives that's measuring the voltages and making the bypass decisions and reporting how "full" or "empty" the battery is so it doesn't overcharge or undercharge and damage any of the cells. It's either a passthrough or it's the brain, no idea which.

Solution: build a new one and replace the whole board. Which also has the advantage that when these repurposed wind farm batteries run out, we can order more prism cells and put them in our own case (Jeff found an aluminum fabricator in japan that would do nicely), and make our own battery systems.

Oh, the old battery cases are also water cooled. (Well, half water half ethylene glycol.) The big cube shed things have an elaborate climate control system, and the spec sheets say that these batteries operate within about 3 degree celsius temperature range. This is partly because they packed a LOT of batteries tightly together into each cube (and run a LOT of power through them), but also because they apparently didn't want to do the math about how the batteries behave differently at different temperatures. (Which Jeff has papers on with graphed curves and math... but the chinese engineers apparently didn't bother.) Which is funny because the 28 pin connector only NEEDS 14 wires to measure the battery voltages (52/4=13, so 12 between each battery and 2 at the ends of the string), and most of the rest are probably temperature sensors? If we're using the same case and wiring harness for the initial deployment we want to reuse those temperature sensors, but... no documentation. Gotta go poke at stuff with voltmeters and crack one open to find what's actually there and look up data sheets...

All this stuff has to get re-certified to be hooked up to the grid, we need to find or create documentation for the parts we keep. (Jeff also wants to find a "current shunt" for measuring the whole battery, because adding up the individual cells isn't good enough. It's apparently somewhere in all this.) Personaly, I'm uncomfortable mixing enough electricity to run a car with conductive liquids, but it's what's already there. Cracking them open and then deploying the result is a thing we would rather not do, so we want to just replace the electronics on the front without opening the case. (Also, Lithium Iron Phosphate is WAY SAFER than Lithium Ion. I would not want to do ANY of this with Lithium Ion. The downside is LiPo4 only has half the energy density of the best Lithium Ion, but it can go through a LOT more charge/discharge cycles without losing capacity. "Lasts fifty times longer and puncturing a single membrane doesn't result in a three hour fire water won't extinguish" is rather a nice trade-off.)

Anyway, coming up with a plan for what to do with all that was "Milestone 2". Initially the goal for that was "design a demonstration prototype unit" (and milestone 3 is building/delivering like 3 prototypes they could show to people), but we wound up debugging through enough of the original electronics (and out the other side) that we came up with a scalable manufacturing plan for all-new replacment parts.

At which point I assumed we would actually start making stuff, but so far...

Jeff wanted me to come along to Shibuya for a meeting with Mike and PK today so we can go over business plans for fundraising. Because of course. I mentioned not wanting to talk to Mike, and Jeff went through several variants of "that's not good for me", "you can't do that", "get over it", and "suck it up and deal" (none of them phrased _quite_ that way), and I went along rather than argue.

I think Jeff's position here is "ha ha, Mike just _threatened_ to have you arrested and presumably deported and barred from the country, it didn't actually happen, so no harm no foul". My position is "Mike showed me who he is". There's probably some divergent neurochemistry in there, what with me being an ADHD poster child and all. (Growing up I was diagnosed "hyperactive and gifted". They hadn't invented ADHD yet. This isn't exactly rejection sensitive dysphoria because I never wanted Mike's approval, he's a friend of Jeff's whom Jeff finds useful for running the business that gives us the opportunity to work on the interesting tech. That's not the relationship jeff wants me to have with mike, and "I work on interesting tech with you" is not the relationship Jeff wants me to have with his business, either. But I started working for him in October 2014, it's 7 and 1/2 years later, and I can't think of a single thing we worked on that actually got deployed. We have not shipped ANYTHING to a customer except prototypes and demonstration units. As with Linux on the Desktop and making Android self-hosting, I keep grinding away and want it to work, and we get closer. There's a bunch of good side effects. But I am no longer trying to organize my household finances around it succeeding: for the moment Google is paying the bills so I can focus on toybox, and THAT is plenty of challenge for me. I don't know how long that situation will last, and am trying to make the most of it. This is VACATION TIME from that.)

I want to learn tech stuff from Jeff, and he's got a lot of great projects to do. I was hoping this trip we might reopen the VHDL to implement the barrel processor and fourier engine functionality we were talking about before Covid happened. Making an ASIC work through Sky130 _or_ ArtAnalog/TSMC would be great too. Jeff's also talked about doing a fresh j-core implementation starting over with a tomasulo/scoreboard design so it can do multi-issue. But we're not working on any of that, because we have more important things to do... as in chase money. I always get lured here with promises of tech work, and then we do a big fundraising document.

This time the document was called "milestone 2", and there was at least a lot of high level technical design work involved as we worked out how to recondition a load of batteries someone wants to repurpose from storing power at a windmill farm into individual combini and factory power walls. Only 6 of Japan's 17 nuclear reactors have reopened since Fukushima, meaning they leaned hard back into fossil generation without really PLANNING for that to happen, and in the past couple years the cost of Japanese electricity has tripled as fart gas got expensive due to Vladimir Putin's dick being too small. Load shifting from overnight to daytime is now potentially a big cost savings, and that's a market with legs anyway as wind+solar ramps up, so let's get into the battery management system business! Sure, why not, sounds like fun. I've been watching prudetube videos from Will Prowse and such about this sort of thing for years anyway, I'd love to learn more.

When I first went to work for Jeff in 2014 his company Smart Energy Instruments was trying to retrofit the electrical grid with sensors so we could feed a lot more wind and solar into it. I'm big into renewables and getting off fossil fuel, this IS my idea of fun. I very much want to see this project succeed, and grow into a sustainable business.

In the first ~2 weeks of this trip I learned how battery management systems work, although I doubt I could quite reproduce all the math myself. We've confirmed we can do a new one from off the shelf chips that don't require an NDA, plus technology Jeff has lying around from previous projects. Yay! The result was... a document that goes to somebody who gives Jeff money for having completed the project milestone. (But that somebody is not a "customer" and Jeff was angry that I kept calling him that.)

At the end of my original trip it looked like we were just about to actually starting building stuff, so I agreed to stay a couple more weeks. Then as soon as the trip was extended, we did the Open Project stuff to make gantt charts, and today's meeting I didn't want to attend was about preparing for a fundraising round.

After the meeting in Shibuya, Mike wanted to talk to me about setting up a meeting where he and I go over a new contract for me to come back to work for Jeff's company full-time. I was noncommittal. I'd rather NOT be arrested and deported before the 17th.

Travel arrangements for the potential talk in taiwan this summer went a bit off the rails yesterday, because when they originally asked about airport selection I thought they were talking about the DESTINATION airport not source (I didn't recognize airport code IAH)... so they booked me out of Houston. Oops. That's a 2.5 hour drive away from Austin (if I still had a car, bit longer by bus). I asked if they could add a connecting flight since a quick check of commuter flights from Austin to Houston shows a bunch for around $70... and they offered to refund me the $70 when I got there. I was too tired to cope and thought I'd try again in the morning. (Actually I went "maybe I could just send them a video of my talk, and not go in person because even if they can't amend the itinerary I'm sure they can still get a refund this far in advance...", and was up until 3am doing an outline.)

The money isn't the problem. The USA's insane security theater is the problem. Airports these days, you're supposed to budget 2 hours to get through security, and if I arrive on a different itinerary than I'm continuing on I have to go through security AGAIN (and collect my luggage and re-check it), which means my one hour layover adds 2 more hours, and I still have to arrive 2 hours early in Austin, plus the actual austin to houston flight, plus me getting up and going to the airport in Austin, which means my ~1:30 pm depature out of Houston is now something I should leave home for around 6 am. It's now a red-eye flight with something like 17 hours of travel before I arrive in a strange country to deal with a new kind of customs check and trying to find the hotel. (Tokyo I more or less know my way around now, and can recover from inevitably getting lost more than once on my way anywhere. Taiwan I've never been to, I'm assuming it has a rail system or busses of some sort? To... a hotel? Somewhere?)

What I'd MEANT to work on last night back at the hotel (instead of outlining a talk I don't have to give for months yet as a way of venting "dowanna deal with this right now" anxiety) was getting Jeff an initramfs that extracts a tarball into a subdir and then does a proper switch_root into that. Which means teaching switch_root that it's not only partition boundaries block file deletion, it should also skip the destination directory. Oh, and it wasn't doing the mount --move on the existing partition mounts, thought it was but apparently I hadn't implemented that.

Doing this means the j-core developers don't have to rebuild the kernel each time to change the root filesystem contents. (They never taught the j-core bootloader to load an eternal initrd.gz file because doing that with device tree involves either patching an existing device tree on the fly or doing the device tree overlay thing, and we just never got around to it.) Upgrading switch_root is a good toybox thing to add to the release I'm preparing anyway, seemed like a good thing to prioritize.

So after about 3 hours of sleep, I got up and started working on that, and an hour later Mike called up to yell at me that I was making "us" look poor by arguing over $70 (the conference that invited Jeff and me to speak had offered to cover travel, I took them up on it, I hadn't yet responded to the email offering $70 instead of amending the itinerary). When I told him my position on that, he shifted to yelling I hadn't left the hotel in days (not true?) and that I didn't have the work for Jeff done yet which he had CALLED INTERRUPTING ME WORKING ON. (And which I do best in the hotel because our Hello Office is hot, stuffy, windowless, and full of trash.)

They're paying for my trip but not paying me a salary. I like tokyo, hadn't been here in years, from my point of view I got a free vacation to Japan and am helping out friends, but I've GOT a day job working on toybox, and am trying to keep up with that while I'm here. I want to see them succeed, but this ain't paying the mortage (or the $8k to clean the mold out of the vents back home).

I told Mike that his call had interrupted me doing the thing he had called to yell at me about not doing, and that I was going to go back to doing it, and hung up and muted my phone for a bit. Twenty minutes later I noticed his Signal message: "Rob pick up the phone or I will get you arrested by the police". (Did I mention Mike is a japanese citizen and I'm not? On an unrelated note, when I first got ADHD meds I didn't know they weren't allowed in Japan: different schedule levels and not honoring foreign prescriptions and so on. When Jeff and I cleaned up the office I found all sorts of old pre-pandemic things, many of which I'd asked Mike about the status/location of before said cleanup. Back when I left the apartment in Japan expecting to return _before_ the end of the pandemic, I left behind two suitcases worth of stuff which Mike moved to the office when the lease on that apartment finally expired and they chose not to renew it.)

So at that point it was me trying to get the code done and sent to Jeff before the police showed up. I got it done, tested, checked in, and an updated image (with build procedure) emailed to Jeff. I then left the hotel room (still sans police), met Jeff at the Hello Office, and walked him through the build so I was sure he could reproduce what I'd done.

I'm sure other things happened later that day, but I couldn't tell you what they were. No, police didn't show up. (I dunno if Mike was bluffing or merely changed his mind. I was actually looking forward to maybe getting to go home early, I'm kind of regretting extending my stay from the 3rd to the 17th if it's going to be like this, and NOW the complication with the Taiwan itinerary thing has switched to "I asked them to fly me in from the USA and out to Tokyo, but I'm no longer sure I could/should come back to Tokyo"...)

Cleaned out the spam folder again. For some reason, gmail has decided that all of Elliott Hughes' posts to the musl mailing list were spam. (Not the REST of the threads he's replying to, just him. And not his posts to me or to the toybox list.) Yes, he's a google employee emailing from a google.com address. *jazzhands*

Finding stuff wrong while doing release notes, as you do. (Nothing highlights gaps and weird cornercases like writing documentation.)

I should record my talk proposal writeups instead of just entering them into various call for papers websites where the ones that aren't selected vanish. (I mean, I SHOULD do them as online talk videos, but it's really hard to motivate myself to talk to camera by myself in a quiet room. As the twelth doctor said in the episode Heaven Sent: "I'm nothing without an audience". I was doing ELC but the Linux Foundation's really done a number on that one and I haven't been able to brace myself for it since 2019.)

Slogging away at toybox release notes. I've done a new entry skeleton that I may leave commented out at the start of the page because I haven't entirely been consistent in my category headings. I should probably ALMOST CERTAINLY switch the index.html link to point to the "about" page (which tries to explain what the project is and why) instead of the "news" page (which is long technical gibberish release notes and not a good first impression; yeah proof of life but not a gentle slope to ascend instead of a vertical cliff).

Oh goddess. You know how news coverage and articles always seem authoritative until you read something you already know about, and then there's multiple obvious errors? I just read the Wikipedia[citation needed] article on Bionic. Lots of "that's just wrong", "that's almost a decade out of date", "Elliott fixed that because of _me_", "no you can just do this instead"... I may need to go lie down. (And I'm not even a Bionic developer!)

Jeff and I cleaned up the Hello Office by dragging most of its contents out into the conference room down the hall and then putting 3/4 of it back and throwing out the rest. (Well, right now it's a pile of trash in the middle of the office because the building's trash room is only open for an hour in the mornings, and we didn't find a box cutter for the boxes so may need to buy one. But I'm calling it a win anyway. Three hours of lifting and hauling. I do NOT get enough exercise.)

During the shoveling I unearthed a mysterious CD which turned out to have the professional photo I had taken (japan still has that service, like Sears used to) for my now-expired zairyu card, which means my own attempt was not strictly necessary.

Woo, 5000th commit to the toybox repository! I feel I should have some sort of celebration. (I bought an instance of the famed famichicki from famimart. Tasty, but very greasy. I prefer their teriyaki grilled chicken breast to their fried offering. I guess both technically qualify as famichicki, but the fried one is the meme.)

I am REALLY TEMPTED to add a new option to toybox echo so it can split the arguments it's printing with newline instead of space. There's a lot of "ls blah/*/blah | xargs | blah" to glue things together, but the other way has to use "echo blah/*/blah | tr ' ' '\n'" which is awkward (quoting both arguments!), but not awkward ENOUGH to add "echo -N blah/*/blah" and open the whole "should I try to teach busybox and coreutils about that" can of compatibility worms.

Yes, I added "test -T 37" recently, because I care that the file descriptor is OPEN, not that it's necessarily a terminal. And I couldn't figure out how to do that otherwise... Sigh, figured it out: I can just do "2>/dev/null <37" instead and the shell will error if the filehandle isn't open (because the dup2() fails). Alright, remove that then. (This is why releases take so long: writing documentation reveals needed code changes, and both blogging and release notes count.)

Under the weather today. The kebab place Mike likes to go has very spicy food, and I killed a roach that crawled over the table while I was eating. This morning my digestion was not happy. Go figure.

Working on closing tabs for a toybox release. So many tabs...

I need to rebuild toolchains with musl-1.2.4 but I never did bisect why sh2eb won't build under newer gcc, and I can't ship new toolchains without that. (Well, ideally I want to rebuild the hexagon llvm too, which had its own version skew a while back. I also want to redo mcm-buildall.sh to not need mcm, at which point I can probably stick the new replacement in mkroot.

And I need to finish mkroot/README and update the mkroot faq.html entires...

Some text I cut out of my reply, as "not helpful". (I have a venting-about-lkml budget I try to stay under, and that message already had plenty. Here, no such limit. Well, I spent so much of the rump administration venting about what the ruling nazis were doing that I forced myself to only do it on odd numbered days and left the even ones for technical stuff, but A) much higher limit, B) that was people's lives rather than just niche drama.) Anyway, what I wrote was, with URLs moved into actual links because blog instead of non-HTML email:

I was going to point you at the last kernel commit with "oppenlander" in it so you could confirm which email to use, but I have a repository here going back to 0.0.1 and his name's not there despite the patch submissions. He's not a regular in the clique, so nothing he submitted ever got in. That's linux-kernel for you.

(For all my faults, I historically _have_ managed to get code into linux-kernel. Largely because I'm really old so have been around longer than a lot of the grognards gatekeeping these days, and I was even technically the Documentation maintainer between commits 01358e562a8b and 5191d566c023. And I understand the whole "if you want to get anything done you have to complain until you're blue in the mouth" Dead Parrot aspect of the project the author of Squashfs eloquently explained ("a closed community which know everyone worth knowing by sight") ten years ago when Linux Weekly News asked if the Linux Foundation had completed its purge of all hobbyists from the open source development process, which it had. They've ossified a LOT more in the 10 years since Philip Lougher wrote that...

So yeah, happy to submit patches to someone who will actually talk about the code and not the bureaucracy+politics (he says, venting about the bureaucracy+politics).

Jeff and Mike are turning a big todo list I made into Open Project Work Items. I'm sitting with my laptop doing other stuff, but available in case they need to ask questions about the todo list.

I have an old rant about open source being unable to do user interfaces, and it's about how any time it's faced with a user interface issue the process melts down into one of three distinct failure modes. I know I blogged about it but couldn't remember which year off the top of my head, so I googled for "landley three distinct failure modes"... and then put quotes around "landley" because I recently learned that it's silently substituting in random misspellings for words it doesn't think are popular enough... and my blog STILL does not show up in ANY of Google's hits. Nor does the copy of the rant I put into the aboriginal linux about page, which I was reminded of when I looked at the talk version of the rant I gave years ago at ELC and I had that about page version up on the screen.

Google found NONE OF THAT. Despite all three containing the phrase "three distinct failure modes", and two of them being on landley.net. Google search is not healthy. It's kind of concerning, twitter going away is one thing, but Google Search will be _missed_. (They're panicing about chatgpt, but NOT about rapidly losing competence at their original core business. It seems to have started about when they laid off those 12,000 people.)

Today I learned that Open Project (and presumably whatever the generic name for crap-like-jira is) has "stories" and "epics", and an epic is a collection of stories. (Like the Epic of Gilgamesh... which seems kinda unique and nobody ELSE calls a collection of stories an epic? It's usually a series when it's not a trilogy. Kevin Feige is trying to brand the MCU iron man to endgame collection as "the inifinit saga".) This "epic" naming is pretentions enough I'm actually slightly nauseous. I would go out of my way to avoid meeting the people who decided on that naming.

Still getting emails for the "Austin Tech Happy Hour", which was a vaguely interesting thing many years ago. It seemed like a good idea to maybe meet some people on the same side of the planet, now that all the local LUGs I knew broke up; I went three or four times, don't think I actually met anybody I wound up seeing again). But at some point it grew a cover charge to keep the riff-raff out, and I really don't feel the need to pay $10 to attend a gathering of people I don't know at a bar, thanks. (Meeting random strangers with shared interests in-person is what giving talks at conferences is for. And science fiction conventions. In THEORY it's what meetup.com was about, but all the ones I tried to attend of those were "oh no, you're not allowed to enter the building without paying" nonsense too. And all those SxSW events that supposedly didn't require a badge, I stopped trying those YEARS ago because I never once got in. They were either full to capacity from preregistrations I couldn't access without a badge, or just plain "it said it didn't need a thousand dollar badge but does". As with the twitter blue checks, it's not the ability to afford it that's the problem, it's that the kind of people you're selecting for means I don't want to meet them.

Alas, my normal daily schedule involves sitting quietly in various corners reading and/or writing things, with the occasional long walk by myself. I often have _extensive_ correspondence with people at least a thousand miles away, but have to go out of my way to exchange ten consecutive words with anybody in the same town who I don't actually live with. There's a reason I founded more than one science fiction convention back in the day. :)

Darn it, glibc's wcwidth() is returning at most 1 for every character in toybox, never 2 even though when you cat tests/files/japan.txt it's all hiragana characters of width 2 (visibly measurable against an ascii text line above it). I'm trying to rewrite fold.c to do unicode properly and the glibc apis don't work.

Jeff is deeply enamoured of a pointy haired management thing called "OpenProject", so we spent HOURS yesterday setting it up so he can do gantt charts in it. Except the admin account doesn't work because it immediately goes "cross site scripting!" which turns out to be because the browser doing https is not enough, the openproject application ALSO has to have access to your let's encrypt keys. (Why? I don't know. Third base.)

This thing is the kind of "open source" you see when a corporation produces regularly updated abandonware. It has no community. There is no libra chat channel for it. Googling for things about it produces hits on their site and nowhere else (although with the sad state of google search I'm not sure what that proves).

A recurring error in our attempts to set up OpenProject is that their git integration breaks apache, which refuses to start because "OpenProjectGitSmartHttp" is a made up word its config file parser doesn't know. Googling for that word finds a closed bug report on the Let's Encrypt website where the Let's Encrypt people say "this is not our bug, ask openproject". There's also a bug report on the OpenProject website where somebody said it broke, and someone else replied "yeah it broke for me too", with no response and no fix. The bug report is from 3 years ago.

We EVENTUALLY figured out that the magic word is exported by the subversion integration code, so if you enable git integration WITHOUT also enabling subversion integration, it CAN'T WORK. (I repeat, this project has no developer community except employees of the company producing it, and THEY want you to run their magic docker where everything is preinstalled for you and you do not touch their proprietary inexplicable secret sauce "open source" code that you're crazy for trying to install/configure yourself.)

And of course if you enable the svn integration it breaks apache for a DIFFERENT reason, so we just switched them both off for now.

I also noticed that the gmail account Jeff set up for me years ago, which I'm only logged into on my phone, hasn't been inactive like I thought. When I open the gmail app on my phone (only thing logged into it), it says "auto sync is off", and I have to pull down to load to see if there's new mail. This is why I haven't gotten a new mail notification from it since last year. BUT if I try to turn auto sync on, I get a full-screen pop-up saying this doesn't apply to just gmail but will also flush all my photos to google's cloud so they can scan them on behalf of ICE and the TSA. There's no obvious way to enable "tell me when I get new email" without "send my contact list and location history to the governor of texas whose boomer supporters can sue you for a million dollars if they think your wife had a miscarriage". Hell no. I don't want to sync my photos, contacts, location history, I don't want it uploading (let alone retaining) the voice samples from speech-to-text (which I KNOW it can do locally because it does it in airplane mode, I don't know what Rossman is on about? Or is this one of those "it _can_ operate independently but there's no way to tell it not to upload everything anyway" things like I'm having with the email client?)...

So I'm writing a new unicode aware fold and I'd just like to say that posix really needs to move past the Y2K bug and enter the 21st century at some point. They have a "-b" meaning "interpret as bytes", but do NOT really handle the "not that" case.

Backspace is defined as reducing the column count by one, but unicode characters can have variable width (including zero for combining characters which should logically come BEFORE the character(s) they combine with but don't because somebody REALLY STUPID was on the unicode committee, I'm assuming from Microsoft). So in THEORY backspace should remove the number of columns consumed by the last printable character.

In practice, the flush-and-forget approach to output when toybuf fills up is a problem because we may have to backspace into it... unless we record how wide each column of output was? I mean, that's just a malloc of length -w (or shorter if we want to get fancy), AND avoids having to back up through utf8 to find the last printable unicode character.

Talked to Jeff about whether I should bump my flight back. We're getting a bunch done, and Fade and Fuzzy don't... strongly object. I enjoy Tokyo, and get about as much toybox work done here as I do elsewhere (the better work environment balancing out the extra demands on my time, although I hope fixing the mold in the vents back in texas changes that going forward).

I'm basically getting a free vacation in Japan, modulo not really seeing much of it outside of late night walks (tried to walk south to the beach; there's no beach, it turns into an industrial harbor sort of thing. Oh yeah, city. Right...) but I'm STILL trying to learn nontrivial amounts of the language.

Everybody keeps saying that "the food is so much healthier here you will lose weight", but 24 hour combini with tuna mayo rice balls and sweet milk tea EXACTLY the way I made it growing up (which was dismissed as absurd by everyone else, where's the Fools I'll Show Them All lightning when you need it, VINDICATION!!1!ichi!) it hasn't exactly worked out that way. I have two relevant Claire Ting videos queued up but would have to shuffle luggage for multiple minutes to clear space... so long walks. I wonder if there are any swimming pools available?

I'm eyeing a toybox release. It's overdue, I know, but there are so many things I'd like to get IN said release. Still, 6.3 came out and I should more or less try to stay synced with kernel releases...

The downside of this lovely office is I have no ADHD meds here, because they're not legal in Japan, and I'm really starting to notice.

Sigh, SMB_SUPER_MAGIC is still lying around, and it got moved to staging in commit 2116b7a473bf and then removed in 939cbe5af5fb in 2011, which was TWELVE YEARS AGO and yet the debris is still not just in the kernel tree, but in the header files exported to userspace. Oh hey, and USBDEVICE_SUPER_MAGIC is gone too (commit fb28d58b72aa back in 2012), but the symbol's still exported in the header. Oh, and last time I was poking at this, Novell Netware went away in commit bd32895c750b but they still have NCP_SUPER_MAGIC in the header.)

The observation in The Cathedral and the Bazaar that "with enough eyeballs all bugs are shallow" has been demonstrably untrue of linux-kernel for at least that long. There are not enough eyeballs because the kernel community is unwelcoming of newbies who would go over the obvious with fresh eyes and thus point out stuff like that. Just like the geriatric Unix community it replaced, it's now all old farts with long white beards and suspenders telling everyone who will listen about the glory days ~25 years ago.

Huh. I've got "fat" as 0x4006 in my list, and can't find that in the kernel source (not current, not 4.0, 3.0, 2.6, 2.4, 2.2, or 2.0). It came from a patch from Hyejin Kim but I have no idea where he(?) got that from? There's "msdos" and "vfat" (both 0x4D44), but no "fat" using 0x4006... And the 4d44 constant was added in linux-0.9.7 in 1992.

Right, posted the patch with a jazzhands comment, poked the github request to see if that fixed it for them, and punted on a BUNCH of questions. (If I identify smb do I say "cifs" or "smb3", both of which are driver names you can mount it with but... different behavior? msdos vs vfat is another but there's never reason NOT to use vfat these days that I'm aware of...) What I should really do is come up with a Horrible Sed Invocation that just extracts this data from the kernel source so I can regression test, but I'm not up for it right now. (In part because grep -ho 'register_filesystem[(][&][^)]*)' -r * | sort -u | wc -l says there's 97 of them, and in part because the first one grep finds is in arch/s390/hypfs/inode.c and I really can't bring myself to care about that one at the moment. And because grep 'static struct file_system_type .*_fs_type = {' -r * | wc returns 83 hits rather than 97 meaning this is NOT quite regular enough to make it easy.)

Fade finally tried one of the cans of The Dintiest Moore I left behind (I asked her to order a flat while I was there), and is a fan. It's the only american product I've encountered that makes serious use of Demi-Glace. (I don't know what non-demi glace would look like. Full glace? Ask the french. Highly boiled cow.)

Watched a frustrating history of gasoline video, which both had good historical information and repeated debunked lies out of old industry press releases verbatim.

A hundred years ago, Standard Oil worked out that mixing about 10% ethanol into gasoline prevents engine knock. All the lead in tetraethyl lead EVER did was make it PATENTABLE, because ethanol (which is the kind of alcohol humans have been drinking for thousands of years) already existed. The lead served no other function in the mixture EXCEPT to make it patentable. Tetraethyl lead is four ethanol molecules connected to an atom of lead (resulting in a molecule shaped like a swastika), and when you heat it you get back four ethanol molecules, plus a free radical of lead which goes out the tailpipe. It otherwise behaves EXACTLY like mixing ethanol into the gasoline would (which was the goal of developing the compound), and when it was finally restricted by the EPA they replaced it in gasoline with pure ethanol. Old engines that COULD use leaded gasoline (because they didn't have a catalytic converter, which the lead binds to covering over the catalyst surfaces that otherwise break down incomplete combustion products like carbon monoxide and nitrous oxide and so on), all those old engines worked JUST FINE with "unleaded" gasoline, and people only thought the stuff with lead was "better" because of years of advertising lying to them and causing placebo effect performance evaluation.

The airborne lead also made people exposed to it measurably more stupid, which is combining badly with senility in the current Boomer generation as age-related neurological degeneration overcompes their ability to compensate for a lifetime of nerve damage from massive pediatric and chronic lead exposure. (This is why everyone fled the cities to the suburbs, they moved upwind so they could breathe! But it was only RELATIVELY better, the air of the ENTIRE PLANET was poisoned (airborne lead was like acid rain and the CFCs that caused the antarctic ozone hole).

Keep in mind that organic lead compounds are generally even worse than metallic lead, because the human body is better at absorbing organic compounds and bringing them inside cells. So both tetraethyl lead itself and the free lead radicals going out the tailpipe in a cloud of superheated moist carbon monoxide and so on... that may have poisoned the Boomers WAY more than the largely inert residue it's broken down into 30+ years later. Some compounds are worse than others: the movie Erin Brocovich talks about Hexavalent chromium being WAY MORE TOXIC than other chromium compounds, and the research chemist Karen Wetterhahn was killed by a couple drops of dimethyl mercury poisoning her through her glove. The leaded gasoline profiteers were intentionally putting lead into volatile organic compounds that people would inhale, the neurological damage the Boomers suffered from this is manifesting VERY STRONGLY in their senior years.

Seriously, I wrote about this at length, with citations to multiple articles about it. Water samples taken in the middle of the pacific ocean had 20 times as much lead near the surface as the same location a few hundred feet down. Blood lead levels were SIX HUNDRED TIMES higher than samples from ancient egyptian mummies, and children absorbed 5 times as much as adults did. The Boomers were the first generation to grow up surrounded by cars, and it HURT THEM BADLY. In their 20s they could mostly compensate. But as they slowed down in their 40s the brain damage really started to show, and now that they're turning 70 two thirds of them are losing all touch with reality. This is not a case of oligarchs being better at manipulating people than the Railroad Robber Barons of the Guilded Age of the late 1800s, this is a population of lead poisoned vegetables ripe for elder abuse. Ten years ago they were falling for nigerian prince email spam, and now it's fascists finding them useful political cannon fodder. If even the rich and famous regularly suffer from elder abuse, imagine what the wider population of brain damaged Boomers is undergoing. Boomerdom going full nazi is because they literally have brain damage, which means our best chance to pull out of it and clean up afterwards is to outlive them.

Back to the frustrating video: when he later goes on to talk about "oxygenates" like ETHANOL... he does not connect the dots. This was not a new discovery. Thomas Midgely and his bosses understood this JUST FINE a hundred years ago. They chose to poison LITERALLY BILLIONS OF PEOPLE around the world entirely for profit. And then when Oil Industry stopped needing "the Ethyl Institute", the think tank reorganized itself into The Tobacco Institute to defend poisoning OTHER people for profit. And when that ran out, they reorganized into a bunch of global warming denialist think tanks to continue to kill people for profit.

Billionaires love to profit from fascists, and gerontocracy collapses into facsism, and we're suffering from both right now. On the geronotocracy thing: Hitler came to power in Germany because the previous President of Germany, 86 year old World War I veteran Paul Hindenberg, made him Chancellor in 1934 to shut him up (ahem: in hopes sharing power would appease him). Hindenberg was then manipulated into signing an emergency declaration ONE MONTH LATER giving Hitler's edicts the force of law which were not subject for judicial review for the duration of an emergency that lasted until Hitler said it was over. 6 months later Hindenberg died, at which point Hitler appointed himself president AND chancellor. Hindenberg was the same age as Dianne Feinstein (who is still in the senate), 3 years older than Nancy Pelosi (who is still in congress), and the same age Biden would be at the end of the second term he just announced he's running for. At least all those guys are OLDER than the pediatric (but not chronic) lead exposure from gasoline.

Oh good grief, now the guy in the video is on about ethanol coming from plants that absorb carbon dioxide: STOP IT. All matters is whether it's fossil carbon or not. Plants taking carbon out of the atmosphere for SIX MONTHS before it goes right back into the atmosphere does not change the amount of carbon in the atmosphere in any meaningful way. Mining operations that take carbon that's been underground for millions of years and release it into the atmosphere, THAT is what permanently increases atmospheric carbon. I do not care about rearranging deck chairs on the titanic, either you're mining fossil carbon or you aren't. (The problem with "carbon sequestration" is finding someplace to put it. A trillion dollar industry digging up carbon from miles underground is kinda hard to run in reverse at the DESIGN level...)

Reading press releases is not research.

Sigh, archive.org decided to commit seppuku during the pandemic (let's aggro every major publisher by putting their books online for free!) so I should definitely mirror the institutional memory post in my own computer history archive before it goes away. (Yes IP law is stupid the same way car-centric cities are stupid, but running out into traffic is not the answer.)

Finally got the turtle board running the 6.3 kernel and current toybox (increasing my kernel patch stack to 10 patches in the process), and... there are bugs. For some reason, ctrl-C doesn't work in the console which means oneit isn't doing the switch from /dev/console to /dev/ttyS0 (well, ttyUL0 there) properly. Another problem is that "ps" produces no output, even though I can cat files out of /proc and see the raw data it should be transforming into output.

Alas, I still don't have a proper nommu test system set up under qemu, and sneakernetting sd cards over to the turtle board for compile/install/test cycles is... really hard on the fingernails. I burned out (gummed up?) one SD card adapter already and bought a new one that's REALLY TIGHT, and have been slowly chipping bits of plastic off the ridge at the end getting the sd card back out. (The turtle board itself does the push-to-click thing but the laptop end uses a microsd-to-sd adapter, unless I want to dig up a USB adapter which is worse. I've already trimmed my fingernail to be less pointy in hopes of chipping out LESS plastic, but it's a question of degree.)

Sigh, at some point I need to do this dance with QEMU's virtual cortex-m board so I have a nommu test environment that runs under qemu, which should make regression testing this a lot easier. The problem is I don't have a "what success looks like" reference version there. Maybe I can beat one out of buildroot? (Or make puppy eyes at Geert Uytterhoeven about coldfire, that's a nommu target qemu theoretically supports as well, although I recall getting a kernel/board config to match with nontrivial amounts of RAM and useful peripheral devices didn't line up last I checked. Sigh, I should learn to modify QEMU, but just haven't got the spoons.)

The magic to stop vim from intercepting the mouse, thus preventing the terminal from letting me copy and paste text between a screen session at the far end of ssh and a local window, is the colon command "set mouse=" with nothing after it. There may have been a small rant.

I'm not just merging the j-core turtle board config into mkroot, and cleaning up mkroot general in preparation for cutting a toybox release, and also testing the 6.3 kernel. Of course there's kernel config weirdness. Kernel commit 3508aae9b561 memorializes a lot of config changes back around v5.8 that I wasn't paying much attention to at the time. IOSCHED_CFQ became IOSCHED_BFQ, IOSCHED_DEADLINE seems to have replaced the NOP one (always configured in), and MMC_BLOCK_BOUNCE went away because you can't switch off the bounce buffers anymore. MTD_M25P80 got merged into MTD_SPI_NOR.

Dirty trick: I can detect NAME=VALUE in the mkroot microconfig format and automatically insert lines other than =y or =m without needing the separate KERNEL_CONFIG mechanism... Except that the value can in theory have a comma in it. (None of the ones I'm using yet do, but they CAN.) Hmmm, I suppose I can come up with an escape mechanism for the comma? And then NOT have an obvious example of it in the file. Hmmm... The alternative is keeping the second mechanism for passing in raw lines despite nothing in the file currently using it. Or waiting for somebody to complain, which... isn't really better here because said complaint is likely to turn into "oh I can't use this" rather than "I'd better report this to the maintainer". Hmmm...

(I can backslash escape quotes and spaces, but can't backslash escape commas because the escape gets eaten before that parsing happens. I could transpose it with another character but that's black magic. I could say an assignment has to be the last thing on the line so it eats commas but I've already got multiple assignments in one config. Hmmm...)

And the air conditioner service guy back in Austin found mold in the vents. So we have to make an appointment with a Mold Remediation Specialist. Great. Well, that explains why I'd feel so tired five minutes after getting home, and had so much trouble getting a good working environment there and preferred to do all my work out at a fast food table or at the university.

(This is exactly why we had very expensive specialists come after every flood with HUGE BLOWERS and refrigerator-sized dehumidifiers drilling holes and spraying gallons of chemicals into the walls: DID NOT WANT MOLD. Didn't really think of the air conditioner vents, where condensation is kinda normal. What's in there for them to eat, anyway? Dust, I guess...)

Anyway, I'm here in tokyo, where mold smells completely different anyway. (The clothes I left hanging to dry in the apartment needed some serious re-washing.)

It's so easy to just spend the ENTIRE DAY in an APA hotel room, and ignore the outside world. I shouldn't, because it's Tokyo out there and I really like tokyo, but it's SO QUIET. (APA is apparently the middle three letters of Japan, at least on their posters... Which is weird because here it's Nippon. Medieval Dutch and Portugese traders asked OTHER countries what those islands over there were called and "Japan" seems to have emerged via consensus from multiple languages playing telephone, and then the same insane map makers who named two whole continents after Americo Vespucci went sure, "Japan", sounds great).

I figured out why Jeff can't stand them, or the windowless Hello Office: the pandemic gave him claustrophobia. Being in enclosed spaces too long gradually increases his stress levels and he needs to go OUT somewhere. I can relate, but am personally experiencing the opposite here. Let me work!

The main limiting factor is Jeff calling me up and pulling me in to his projects, but he is paying for the trip.

The old j-core ethernet driver was just too messy to submit to mainline. It's not secret, but Jeff outsourced it to some cheap Russian programmer (the lowest bidder) years before we ever met and only like 1/3 of it is actually relevant. It's got all sorts of debris from IEEE time synchronization and such that were never completed. We should really write a new one, but never got around to it.

That said, the last time it got forward-ported was 5.8, and we'd like to use it on current (6.3) kernels, and I bisected the FIRST build break to commit adeef3e32146, which made a field const and added a gratuitous new API to change it. There's a bunch of commits (bb52aff3e321, 0f98d7e47843, 9a962aedd30f) converting drivers to the new API, so it wasn't too hard to fix it up. The other breakage (b48b89f9c189) removed an argument from a function, and was easy to fix up.

Bisected the "Turtle works now" bug to commit 5d1d527cd905, which was a rewrite of the RCU plumbing for the networking code that starts "Using rwlock in networking code is extremely risky..." So yeah, I'm willing leave that part to the professionals. The symptom they saw was soft lockups, it fixed our boot hang, calling it good and moving on.

I have discovered LaserPig on Youtube, who answers the question "What if Sheogorath, Daedric prince of Madness from Skyrim, did youtube videos about the war in Ukraine in character as an extremely drunk farm animal with strong opinions about military history and equipment". I discovered him via a team up with the "oh god you've reinvented trains again" guy who keeps photoshopping Elon Musk into a clown outfit.

Trying to have mkroot more gracefully straddle the patched vs unpatched kernel issues, and also get the init script to work nicely in both QEMU and a chroot/container. Added test -T to check if stdin is open, test already has -t but I don't care if it's a tty because a chroot with redirected stdin/stdout (or piped through something) is fine and does not need to be replaced with /dev/console.

Sat down to figure out why the current vanilla kernel broke on turtle, and... it's fixed? It smelled like an alignment issue (unaligned access), and maybe it got perturbed so it's aligned again? (Or else there was a bug that hit somebody else and they fixed it?) Either way, that means the revert commit in Rich's j-core patches is no longer needed although I'm still gonna track down what fixed it so I know. (If bad alignment got perturbed into place again, it'll be back.)

And now that there's a new arch/sh kernel maintainer I'm looking at those to see what's still relevant. Vladimir Murzin's commit was merged into vanilla. There's one adding extra percpu memory... why? Rich does not actually provide descriptions with his patches, so I have no idea what actual PROBLEM he was trying to fix, and the kernel's attempts to describe this plumbing are not enlightening.

Ok, generic commits that still apply: 4c7333b0fb9e, 53ac9fc75ae0, 262e1e5884da and could maybe go upstream as-is. Commit 155d2abffb8b is jcore-specific (the clock thing), I think it's generic-ish and could go into the vanilla tree? (Need to test that the turtle board as is still works with it.) The ethernet is 186e1d80a89b and 666583fa6d5d, gratuitously split in two for no obvious reason.

I'm test building on my turtle board, by repeating:

for i in ../toybox/000[34578]*.patch; do echo $i; patch -p1 -i $i || break; done
sed -Eis '/select HAVE_(STACK_VALIDATION|OBJTOOL)[^_]/d' arch/x86/Kconfig
patch -p1 -i ../linux-sh/0001-percpu-km-ensure-it-is-used-with-NOMMU-either-UP-or-.patch
patch -p1 -i ../linux-sh/0001-revert-790eb67374-to-unbreak-j2.patch
mkroot/mkroot.sh CROSS=sh2eb LINUX=~/linux/github
sudo bash -c 'mount /dev/mmcblk0p1 /mnt && cp root/sh2eb/linux-kernel /mnt/vmlinux && umount /mnt'
sudo microcom -s 115200 /dev/ttyACM0

Bash command line history comes in handy there, cursor up a few times and hit enter, take out the sd card put it in the holder, put it in the laptop, run the thing, take it out of the holder, ka-click it into the board, plus in usb, frown at boot messages, rinse repeat.

Bisecting stuff is awkward (why the above [34578] skips some numbers and a couple patches are broken out as individual lines). Much annoyance because git insists that old is good and new is bad. You're never searching for where something got FIXED, only for where a bug was introduced. Therefore, to find the commit where the turtle board started working again (without reverting commit 790eb67374 in the patch above, that's the "it just started working again" thing) I had to call the one that does not boot "good" and the one that boots to a shell prompt "bad". Because git.

For extra fun, I have to build each one without the revert first to see if I get output, and then build it again with the revert to make sure I DO get output and it's not a different bug. So the cycles are a bit slow.

And mkroot is set up to always do a full build. I can do incremental builds out of tree, but then I have to hack the config file to point to the initramfs directory via absolute path and I dowanna.

The hotel rooms in Japan are lovely. Jeff says they drive him nuts, but they're giving me something I haven't had nearly enough of: quiet well-lit isolation with a desk and an outlet and internet access where I can get work done without interruption. (Especially now I don't have to be out of them by 10am, at least 3 days out of 4.) A 5 minute walk away there's tea the outright weird way I grew up drinking it (cold, sweet, with milk: confusing BOTH sides of the atlantic), and cheap tasty rice triangles (onigiri) which I finally figured out the intended way to open so the seaweed goes around the rice. (There's a pull tab that peels away to split the packaging with a plastic thread down the middle, and then you pull it equally off both sides so the seaweed stays in place. Gets seaweed crumbs on the desk, but otherwise works great. The point is the seaweed and rice are separate until you open them so the seaweed stays dry and crispy.)

I hadn't actually _installed_ the qemu targets I rebuilt back at the end of march, and now that I'm trying to test them "mips" still isn't working. and I don't remember what specifically I built (I can see the git log but it doesn't help), so I think I need to pull and rebuild ust mips, which is probably still "./configure --target-list=mips-softmmu"? (The QEMU devs have a terrible habit of breaking their API for no obvious reason.)

I need ldd to fix up mkroot/root/dynamic, which runs after populating the airlock. Alas Elliott strenuously and repeatedly objected to toybox containing an ldd capable of running that loop, because it wouldn't invoke glibc's dynamic linker to find where something is currently loaded into memory when you haven't loaded it into memory. How a cross compilation running ldd on a mips binary is supposed to tell you where that library is currently loaded into an x86-64 system, I couldn't tell you, but the FSF keeps making their binaries fatter and fatter. The one in uclibc never did this.

Back in 2020 a third article about the greying of Linux came out, but it's already fallen off the web and you have to fish it out of archive.org because "Linux has gone the way of Unix, maintained by crotchety greying grognards who scoff in all directions outside their insular little niche" isn't really NEWS anymore.

It's one of those days where I'm skittling along a giant dependency chain doing twenty minutes work on one thing and then going "but first I need to do X" and getting five things deep before I go "what can I actually FINISH AND CHECK IN RIGHT NOW".

The most recent email with the guy who needs me to update scripts/root/dynamic wound up with him being able to use the prebuilt musl toolchains I provided (he was saying Linux= instead of LINUX= but it's case sensitive), but I should still poke at the "dynamic" target because mkroot shouldn't REQUIRE musl, which led me to asking whether static linking in bionic is working yet (mkroot didn't used to be able to use it because of the "segfault with no stdin" bug hitting PID 1, which got fixed upstream but hadn't made it into the NDK yet, and there's a new NDK (r25c) so I downloaded that and extracted it and went "huh, creating the cc symlink I've been doing works but seems silly because there's no OTHER tools prefixed like that in that directory anymore, where did they go? There are a bunch prefixed with llvm- except there's no llvm-ld in there...") and so on down a rathole I've also parked and stepped away from because NOT RIGHT NOW.

But I tried to build with that in a fresh directory and defconfig of course barfed with bionic because it hasn't got the shadow password plumbing, except I redid lib/password.c and friends in my tree (the new one doesn't USE the shadow.h nonsense awkwardly bolted alongside the original user/group stuff by shadow-utils back in the 1990s) and I really need to test and check that whole rewrite in, except it's big and intrusive so copying it to a new fresh directory for proper testing required some investigation: "git diff lib" says that lib.h, password.c and pending.h (which I deleted as part of this work) are the changed files to marshall over, but the new password.c has three functions (get_salt, read_password, update_password) which grep says are used by: passwd.c, su.c, login.c, mkpasswd.c, chsh.c, groupadd.c, groupdel.c, sulogin.c, useradd.c, and userdel.c. And in my big dirty working tree the changed files are passwd.c, mkpasswd.c, chsh.c, groupadd.c, and groupdel.c, so that's what I should coopy to the new tree and try to build.

Except to test this stuff I need a mkroot build (not letting it write to my /etc directory as root just yet, thanks), and I ALSO have a toybox directory where I'm moving mkroot out of scripts/ and into it's own mkroot/ subdirectory (where I can give it its own README), and there are two edge cases that I'm not sure whether I should move: 1) mcm-buildall.sh and record-commands.

Design-wise scripts/mcm-buildall.sh remains a rough edge because it populates the ccc/ directory at the top level, not under mkroot.sh. The problem is one again "lifetime rules" (you don't rebuild the toolchains every time you rebuild mkroot). So... does it stay in scripts/ or does it move to mkroot/ with mkroot.sh and test_mkroot.sh and the scripts/root directory? It's not really part of toybox, it's an important dependency for mkroot (CROSS= there is what expects the ccc/ directory), and mkroot is what has the plumbing to download external packages (via mkroot/root/plumbing) so it kind of _does_ need to be in there... But if it IS in there then the README is hard to write, because the logical sequence of scripts is then 1) cccbuild.sh, 2) mkroot.sh, 3) test_mkroot.sh. But 99% of the time, you don't RUN cccbuild.sh. Heck, most newbies will probably download binary toolchains because it's a pain.

The other thing is I want to rewrite mcm-buildall.sh so it doesn't use Rich's musl-cross-make repository anymore and is its own standalone cccbuild.sh instead, because Rich doesn't reliably maintain musl-cross-make (the last commit to it was just over a year ago), and it's really not helping much anyway. The Linux From Scratch partial build script I posted to the toybox list last month builds a gcc variant without jumping through that many hoops, and I'm leaning towards just doing my own build directly rather than working out how to feed configuration stuff through Rich's plumbing to the gcc build. I've already added a couple of my own patches to his that he won't take, and have a couple more queued up that I poked him about but he ignored. (That said, I believe he and in his family are still touring Indonesia? I type this from Tokyo, can't throw glass houses at anybody, but I try to stay in touch. He's been insufficiently communicado for a while now.)

And then there's the whole "llvm toolchains" can of worms I need to reopen at some point, which musl-cross-make is no help at all about... I suppose the pending rewrite is a good excuse to leave the old one in scripts/ for now?

ANYWAY, I'm trying to write up the new README, starting from the ancient README back when it was a standalone project, and the FAQ entry (which is another thing I need to update before checking in the move; I should probably leave a symlink from scripts/mkroot.sh to ../mkroot/mkroot.sh in the tree).

Oh hey, today _is_ the every-fourth-day that they clean the room. When I asked the guy at the front desk what time I had to be out by, he said it was tomorrow.

And lo, I have my laptop available again (yay adapter), a quiet hotel room (APA is now only cleaning the rooms every 3 days so I can stay in it all day if I like), and rather a largeish todo backlog. Let's see:

Upgrade test suite so gentoo can run it.
  Request filesystem type, umount -l.
  ldd chroot https://github.com/landley/toybox/commit/e70126eabef8
Finish lspci -x fallout.
  Check compression? https://github.com/landley/toybox/issues/386
  http://lists.landley.net/pipermail/toybox-landley.net/2023-April/029520.html
Finish cgroup stat support.
  https://github.com/landley/toybox/issues/423
Yifan Hong's continuing tar weirdness:
  https://android-review.googlesource.com/c/2536710
Peter Maydell qemu Malta patch?
Tom Lisjac (and previous guy) want scripts/root/dynamic
  https://github.com/landley/toybox/issues/418
David Legault, fold tests. (Promote fold?)
  https://github.com/landley/toybox/issues/424
vmstat for zhmars
  https://github.com/landley/toybox/issues/422
sizeof(toybuf)
  https://en.cppreference.com/w/c/language/_Alignas
fix sh2eb mkroot build (toolchain and kernel)
gzip --rsyncable
  implement deflate, implement rsync...
Ongoing cleanup of mdev.c started on plane due to /sys/block poke.
  http://lists.landley.net/pipermail/toybox-landley.net/2023-April/029525.html
Finish the cp -s work so I can do install -T
Try to beat a multi-console thing out of mkroot+qemu to test oneit change
  http://lists.landley.net/pipermail/toybox-landley.net/2023-April/029531.html

Pretty sure I've missed multiple things there. Plus I _was_ planning on cutting a release before visiting Tokyo. And there's the Linux From Scratch automation script so I can go back down the aboriginal path of making a self-hosting toybox environment...

Ah right, there are no three prong outlets in Tokyo. And I brought a three prong laptop charger. That's inconvenient. My plan to program all morning until the sun came up (what with being waaaaay off this timezone in my sleep schedule) hit a bit of a snag there.

Met with Jeff in his office, unboxed, disassembled and reassembled oscilloscope, talked about about his battery project, went out to dinner with Mike and some of Mike's friends in Shibuya where we went to a chinese-run restaurant that allows smoking indoors, where I found out that after a few years of not trying to eat while breathing cigarette smoke I've lost my tolerance for it. (As in "mouthfull of food and lungfull of air combines to convince my brain I've got a mouthful of cigarette ash, and forcing myself to swallow triggers a nausea reaction that lasts all night." That was not fun.)

It's a lot easier to get programming done on a plane that ISN'T 100% full. Getting up at 5am after finally adusting back to a day schedule doesn't help either. I had grand plans for the 14 hour uninterrupted block, but don't have the focus.

Huh. Given the way adler32 works, if you're just looking for a run of zeroes at the bottom and it's 16 bits or less... you don't need the whole algorithm. It's just "add up the bytes modulus by the largest 16 bit prime".

That really seems unreliable? I mean... ok, fast. But "runs of zeroes" are legitimately a thing? If you compress all zeroes is just gonna reset every minimum window size (4k)?

I still want to figure out how to do the rolling addler32 of the top part. I KNOW I worked this out before, my blog says I did it in 2001 and again in 2013 and it would be nice if I'd actually DONE it back then rather than restarting every 10 years.

Of course today's interrupt is updating the filesystem type detection list, which is tricksy because the kernel isn't consistent. I already have one more small patch (basically a repeat of the v850 patch one) to send to lkml, but they'll just ignore it. (I need to reply to Andrew Morton, but "you guys no longer take obvious one line fixes" is hard to say POLITELY.)

[Editorial, April 15th as I'm fixing this up to post and replacing the [LINK] with an actual link... WOW Google search is imploding fast. Googling for "linux landley v850 patch" does not find that patch, nor does adding "elf" before patch. Adding "remove" before v850 finally found one copy of it in mail-archive.org, which is not the kernel's own lore.kernel.org/lkml nor is it the iu.edu one that's been there since 1995, nor did it find a copy in any of the archives in the vger list for linux-kernel. Google search is blind to all of those. I got the above link out of my preferred archive by checking the date on the post in the one copy Google DID eventually find after all those retries, and then going to lkml.iu.edu and manually navigating there from the top down. Remember when it was easier to google for stuff than bookmark things? Not anymore...]

Sitting down to actually implement gzip --rsyncable, I'm hitting the problem that the USE I'm making of the zlib stuff is "pass off an fd and it returns when done", meaning my code doesn't get to read the data and partition it. I could do a wrapper that reads the data and passes it along, and probably will eventually, but that seems kinda silly?

We switched the household slack to Discord. Let's see how that goes. I have so many "send notes to self in my DM-to-me slack channel", which was a scratchpad I could easily access on both my phone and laptop, and now I'm laboriously copying the to-laptop ones by hand now that I've lost access to that in that context. Gotta do that before deleting the slack accounts and uninstalling the app. (Alas, the phone side doesn't have a selection option that can highlight more than one entry. On the web side I could mouse drag and scroll and grab multiple pages of stuff to a test file at once, but the android UI doesn't have anything similar. Press-and-hold to highlight a single entry. No obviout shift-click to highlight another entry without deselecting the first. And so, I laboriously type one entry at a time into the keyboard. I'm back to February...)

Doing gzip --rsyncable kinda implies doing rsync. According to the rsync wikipedia[citation needed] page the checksum in question is adler32, which seems simple enough, although I'm squinting at the modulus: I'm pretty sure that can happen at the END as long as the input length is less than 256? Sigh, wikipedia[citation needed] keeps saying "look at the zlib source code to see a more optimized version" rather than just saying WHAT THE MORE OPTIMIZED VERSION IS. This is a five line algorithm! Obviously it's moving the modulus to the end. Alright, let's do for loop here to see where the overflow is... If the starting input is already ffffffff and you add ff each time you'll overflow a 32 bit counter after... 5552 entries. So page sized inputs are fine. I can add a comment.

Ok, so the theory here is you do a running checksum on the input, and when the bottom X bits are all zero, you reset the deflate stream. (When Hayase Nagotoro invented blockchain there was a lot less originality than I thought: the post proposing --rsyncable for debian came out YEARS earlier.) Since deflate is designed to work on concatenated archives, I don't even really need to communicate with the encoder, this is a "close and reopen, append results together" situation. Probably you want some minimum amount of input before checking the results, and maybe initialize the CRC to something other than 0 so a run of zeroes doesn't leave it zero? (Or does the "minimum amount of input test" cover that case?)

The next question is "how many bits of zeroes" and "what's the mimimum block size", and the original paper isn't even using adler32 like rsync is so I don't want to take its answers? Unfortunately nobody seems to actually document what gzip --rsyncable is actuallly doing here, let alone how many bits it considers worth resetting for in its --rsyncable. I hate looking at gnu crap both for licensing reasons AND because it's always TRULY HORRIBLE CODE, but I'd like to be at least somewhat compatible? And in the absence of ANY sort of documentation, hold my nose and see what's publicly available on github's web view... Looks like they're using 4096. And they're using MODULUS on a POWER OF TWO to check for the zeroes.

That's just sad. I need to step away from the keyboard for a long walk.

The toybox test suite is a bunch of shell scripts in tests/*.test (one for each command name) that get run by scripts/test.sh (which calls scripts/runtest.sh). The actual tests are shell functions that look like:

testing "name" "command line" "expected output" "file input" "stdin input"

I.E. each test has five arguments: 1) the name to print when running the test, 2) the command line to run for the test, 3) what the test is expected to produce on standard output, 4) what to write toa file named "input", and 5) the input to pipe into the command line's stdin.

There's some complexity: each test gets run in an empty directory (generated/testdir/testdir) with the $PATH set up so it's testing the right command(s). Arguments 3, 4, and 5 are run through echo -e to resolve escapes, and if there's a newline at the end you have to explicitly state it. (There's almost always \n on argument 3.) If argument 4 is an empty "" string then no input file is created (and "input" is deleted between tests) so it's not there messing up ls output and so on. If any command fails to produce the expected output, the script exits (unless the environment variable $VERBOSE contains the string "all" somewhere in it) so later tests don't even get run. But that's the basic idea of the test suite.

There's a few more corner cases, such as the checks that conditionally skip tests (shell functions like "toyonly" and "optional" and "skipnot", which set the $SKIP environment variable), and a whole second set of testing aparatus providing the "txpect" function (txpect NAME COMMAND [I/O/E/Xstring]...) that works like 'expect' listing a series of inputs to stdin and expected outputs (on both stdout and stderr) and eventually an expected exit value, ala:

PS1='$ ' txpect 'shell hello' 'bash --norc --noprofile -i' E$'$ ' I$'echo hello\n' O$'hello\n' E$'$ ' I$'exit 3\n' X3

And someday maybe I need to figure out how to hook that up to pty master/slave plumbing (or do they call it dom/sub now? Hey, that's a consensual relationship...) so I can query cursor position in a virtual screen (testing stuff like "top" in an automated fashion is REALLY nonobvious). But implementing "expect" in pure bash was hard enough...

Possibly the most complex part of all this, from my perspective, is that Android doesn't use my scripts/test.sh, it just uses scripts/runtest.sh. All the shell functions are defined in runtest.sh, but the test.sh script is the one "make tests" and "make test_sed" call to set up the generated/testing/testing directory and work out whether we're testing a single toybox command (calling scripts/single.sh to build it and install it into generated/testing if so), all the toybox commands (calling scripts/install.sh to put all of them into generated/testing so they're all in the $PATH at once), or running the tests against the host commands (which is testing the tests themselves, not testing toybox's command implementations: I haven't proved much if I pass tests I wrote but nothing ELSE passes those tests. Alas the host is a moving target and each time I upgrade devuan some tests that used to pass start failing because their output changed, there's some regex fuzzing I can do but it's a red queen's race...) I'm never entirely sure what will and won't break android's testing when I fiddle with my test plumbing, but Elliott pokes me when I do and I can fix it after the fact.

So anyway, I recently added "ls --sort" which needs tests, and first I was converting the existing ls.tests from "testing" to "testcmd", which is another wrapper in scripts/runtest.sh that supplies the name of the command being tested so you don't need to start each command line string with the same command. (Not just to eliminate redundancy, but so it can force testing the _toybox_ command instead of shell builtins and alias trickery by providing absolute path to command as necessary. Otherwise testing "echo" under bash isn't testing toybox, it's testing bash.) I didn't want to change the base "testing" function to do that because sometimes you want to be explicit, ala "VAR=VALUE $COMMAND --blah" or "for i in a b c; do $COMMAND $i; done", and besides: switching them over requires editing the command string to remove the command name, which gives me an excuse to review tests using my standard lazy approach. (Same general idea as the college study advice that taking notes helps because when you write it down you remember it.) But converting entire test files from "testing" to "testcmd" is generallly good because the result is shorter and less redundant, and often avoids wordwrapping. In THEORY a nice low-brain activity I can do when I'm not feeling up to much... (Except that when I do review, I tend to find stuff and go off on tangents. It's basically horror movie logic here, there WILL be something. But I'm getting ahead of myself...)

Another entry in the "there's some complexity" pile above is that 1) the name of the command being tested is prepended to the name of the test, so you don't have to repeat it each time, 2) if the first argument to testing is an empty string, then the second argument gets used as the name of the test. So if I say testing "" "-R" ".." ".." ".." it'll go "PASS: ls -R" in the output. (Or FAIL: or SKIP: depending on what happened. They're all the same number of characters so the output lines up either way. And when going to a tty, it's color-coded.)

I converted "testing" to always prepend the command name because I never want it to NOT do that (or at least couldn't think of any use cases), but when I combined left the first argument of testcmd blank I wound up with output like "PASS: ls ls -R" (or worse, PASS: ls /big/long/path/to/ls -R" because they were BOTH adding it, in a way that was nontrivial to untangle.

So that's where I went off on a tangent and parked the "add ls --sort tests" todo item last time. And this was AFTER my previous excursion into fixing the plumbing ls was using (so the tests actually ran in an EMPTY directory, so ls didn't keep having to duck into a subdirectory to avoid showing debris; this is the downside of me saying "I can always use more tests" when people ask me "how can I help", Divya meant well but left me with some technical debt to shovel out. The real problem is I'm a prefectionist acting like I'm making faberge eggs, but if I'm not doing a BETTER job than what's already there why bother? I mean ok, "licensing", but that's not sufficient reason by itself...)

ANYWAY, with all THAT sorted, I converted the ls tests from "testing" to "testcmd" and now I'm looking at a few of them I noticed are kinda weird. The -N test was actually testing -q, which means back when it went in I didn't review it enough, even though I found one issue right off (which seems obvious: -q isn't the default and -N switches off -b but not -q, so you have to be able to switch -q on first to tell?). And now that I'm going back to try to PROPERLY test it (turns off -b but not -q because gnu/dammit of course) and then I hit:

$ ls --show-control-chars $'hello \rworld'
'hello '$'\r''world'
$ ls --show-control-chars $'hello \rworld' | cat
world

What is this "shell escaping but only to tty" nonsense? Does THAT have a new command line option? I actively do not want to implement it, because it's STUPID. What is the POINT of doing SHELL ESCAPING ONLY TO TERMINAL OUTPUT? And if you're goint to do a $'' wrapper why not just have that be around the whole thing? Why have MORE THAN ONE QUOTE CONTEXT IN THE SAME OUTPUT? What is WRONG WITH THESE PEOPLE?

I miss the days where the gnu/dammit clowns failed to add -j support to tar for 5 years because the gnu development had completely stopped. Everybody just had a standard patch they added and it was all good. That was the period during which the gnu tools became actually popular in Linux, when they WEREN'T CONSTANTLY BREAKING NEW STUFF.

And I rebooted my laptop, losing all my open windows. Not because of any hardware or OS thing this time, but because I was working out test plumbing (to fix gentoo's inability to reliably run the toybox test suite when they build the package, by letting tests request which filesystem they run under), and I did "mount blah.img sub && cd sub && umount -l ." and SOMEHOW instead of unmounting the new loopback filesystem the debian host umount command unmounted my /home partition out from under every running desktop process. So that's nice.

That's a "press the power button and hold it down until it does the unclean override sudden power down" thing. THIS is why I like to test this sort of infrastructure in qemu instances. And of course thunderbird doesn't retain emails in the process of being composed the way kmail did, nor do my 8 gazillion terminal windows+tabs restore their state to show what I was in the middle of...

The design reason I was doing that is a test should be able to go "force_filesystem ext234" at the start and if "stat -fc%T ." says that's not what we're currently using then it should dd if=/dev/zero up an image (because loopback mount can't use sparse files so truncate -s won't work here, although it's transparent to an emulator so qemu can eat one just fine), mke2fs it, loopback mount it in a directory, cd into that directory, delete the loopback file and lazy umount the directory (so both the file and the mount point get freed when the last process using them does; the mount pins the file's inode, and our test process has the mount pinned as its cwd, but no matter _how_ the test exits it can't leave the mount lying around on the host afterwards), and then the rest of the test proceeds as normal and then does a cd out off the directory as part of normal cleanup.

Except of course the gnu/dammit stat command says "ext2/ext3" instead of a proper driver name. Gotta add filters to the parsing because they got "clever" in an actively harmful way. Toddlers "helping" in the kitchen, minus the learning part.

And now that I've rebooted, chromium stopped working with slack, which is now a fullscreen "you can switch to a supported browser or you can install our data mining app but otherwise fuck you" page. I asked on the #devuan channel which says it's a #debian issue because chromium 90.0.4430.212 is the newest version in the "oldstable" category (which devuan beowulf lines up with), so now I'm asking on the #debian channel. There is a "backports" repository, but this package isn't in it.

I've heard that chromium is near-impossible to compile from source, so I'm not TOO surprised that debian can't get it to build in older environments. The standard Google problem of their code both having a zillion dependencies and being completely unportable to the point they care about the specific dot-release of each dependency. Sigh. (And hermetic builds are technically a move to be LESS accepting of variations in build environment. You deploy the one true build environment to target, because building in an emulator on a provided image is slow. I care very much about reproducibility from first principles. This does not appear to be a common viewpoint.)

But needing to do a major OS version update to regain access to my household slack on my laptop? Sigh. I might need to start caring about firefox again. (It's a "household slack" with Fade and Fuzzy, but half of what I use slack for is cut-and-paste of URLs to my phone, and running another OS in a vm to run chromium in there means I'd have to get cut and paste working in kvm, which... enough ratholes for one day, thanks.)

Had to go to the hospital to have a piece of glass professionally removed from my foot. Not my most productive day otherwise.

One of those "spent all day trying to get in the headspace to do productive work, and didn't" days. Cooking and cleaning and generally being Fade's housewife worked out ok. And I did actually invoice the middleman. (Yay!)

(Why does avoidance productivity either put me into DO ALL THE THINGS mode or else completely stop all work, with no middle ground? This wasn't even tax paperwork, it was "resubmit an invoice". Which yes I had a bad bureaucratic experience with 3 months ago, but seriously...)

The mkroot dynamic build (which there's a waiting user for) SEEMS simple, but the current script is using a "cc --print-search-dirs | xargs cp -a $TARGET" approach that winds up populating the target with over a dozen gigabytes of crap which will NEVER fit in a ramfs, and is big enough repeatedly doing that build seems likely to noticeably shorten the life of my laptop's SSD. (And that's after I fixed the "it's copying symlinks" problem that cropped up in an OS version upgrade, so that it wasn't that big, but didn't reliably work either.)

My first stab at cleaning that up was "copy everything to target then sort the hashes, use them to compare files and hardlink together what's identical". Which cuts the space in half but the result is still multiple gigabytes and doesn't reduce the disk thrashing at all (the files still get copied before being discarded).

Now I want to dig up my old "run lld on each file on target to get list of libraries actually in use and copy just those into the new chroot" approach that I had a bash script for even back in the busybox days, code which I recently removed from toybox in theory because mkroot superceded it, and in PRACTICE because I need ldd in the $PATH to make that work, and when I mentioned my desire to add that to toybox Elliott had kittens. (I still don't understand WHY. He doesn't have to enable it for android, but it's a thing I personally have an immediate use case for. Copy this and the library files it needs, and the library files THAT needs. Do it recursively but skip ones already present on target, which prevents endless loops. My first script to do that was in 2001, back when I first put together a tiny boot image with binaries harvested from the distro I was running. I want to say... Red Hat 6?)

Sigh, need to invoice the middleman. I can do it on monday. (It's not EXACTLY rejection sensitive dysporia, "submitted paperwork that got bounced because weird politics, reluctant to do it again" is at least in PART me wondering "do they want to hold on to the money because their finances are dire enough that having it in their bank account reassures THEM, and if so will they actually pass it on once the arbitrary limits they invented are satisfied?" A bank limiting withdrawls to $3000/day is not a healthy bank. Their behavior is NOT A GOOD SIGN, and I'm reluctant to put pressure on the broken thing and see if it hurts... but I'm trying not to invent an unnecessary crisis here either, and "I last got paid in October and we just sent more money to the IRS than I've ever paid for a car" is ticking audibly...)

I suppose the flood of "april fool's" nonsense online shows that people are feeling better? It went away pretty much entirely during the Rump administration because geratric fascists shouting "fake news" to discredit their opposition by loudly and repeatedly asserting that anything they didn't like simply couldn't be true (the previous nazis called this the "big lie") made anybody ELSE not scrupulously telling the truth and fact checking everything they could... kinda inadvisable. It's still really annoying, and seldom even slightly funny. Oh well.

Going through the github requests trying to find simple things to close, but I've just been overcomplicating stuff recently.

For example somebody requested the "shuf" command yesterday, which I added, but I spent a while arguing with myself over whether it should use random(), lrand48(), or getrandom(). In theory the randomest is getrandom() but each calll consumes kernel entropy, which seems overkill for something like this? And it's _wasting_ entropy because it returns whole bytes and I then have to chop it down to just what I need, the common case of which is what, 500 entries?

Initializing a prng from a proper entropy source is a classic middle ground for a reason, and the _easy_ way to chop a randomness source down to a specific integer range is modulus, which will introduce bias unless the modulus is MUCH smaller than the range of the random number (so the uneven coverage of the last wraparound is statistically insignificant).

In the end I just went with srandom(millitime()) and then random()%count which is good enough. (And the trick to make it efficient is lines[ll] = lines[--TT.count] because if you don't care what order the not-yet-used entries are in, swapping the last one down into the hole you just left avoids the memmove() you'd do to close the hole while keeping them in order, or any sort of usage bitmap nonsense.)

What?

FAIL: chmod 750 dir 640 file
echo -ne '' | chmod 750 dir 640 file &&
  ls -ld 640 dir file | cut -d' ' -f 1 | cut -d. -f 1
--- expected	2023-04-01 02:57:10.424197685 -0500
+++ actual	2023-04-01 02:57:10.428197685 -0500
@@ -1,3 +1,3 @@
--rwxr-x---
 drwxr-x---
 -rwxr-x---
+-rwxr-x---

Sigh, I broke ls with the new --sort stuff, because when I reused the -A and -d flags I didn't UNSET them from the base set, so ls -d no longer produces output in the same order. Oops.

Of course I left myself a todo about this: a pet peeve of mine is --longopts without a corresponding sort option are un-unixy, and the new short options I defined for the new --sort types that didn't already have any were -! and -? (except ? is a wildcard and CAN occasionally misbehave, a pet peeve of mine with "qemu-system-mips -M ?" to list available machines is you need to quote the ? if there's a single character file in your current directory; the reason the magic . and .. files don't count here is wildcards won't match hidden files unless the first character is an explicit period... Ahem, anyway maybe -~ would be a better option since tilde is only special to the shell as the _first_ character of an argument, and ~ means approximately anyway so case insensitive shouldn't be TOO hard to remember.)

Using punctuation like that means I'm MUCH less likely to conflict with existing or future gnu nonsense. (The cut -DF support STILL isn't upstream in coreutils, last I checked. I should poke them again...)

So I need to grab the extended argument parsing plumbing I added WAY WAY BACK while working on mkdosfs, which wanted -@ to set the offset and I added a whole mess of lib/args.c and scripts/mkflags.c plumbing to allow that. Which I checked in and tested and everything, but the only user of it is still out of tree in my local pending directory because I got distracted and still haven't finished mkfs.vfat. So, how does it work:

Take your ascii character value (@ is hex 40), set the high bit to turn it into "high ascii", turn that into a good old K&R C octal escape circa 1976, and include the octal escape in the option string: for -@ it's "\300". The FLAG macro you get is FLAG_X followed by two hex digits, in this case FLAG_X40 which means FLAG(X40) should work.

I want to write up a quick mkroot explanation for the qemu guys (who are testing with my mips and misel mkroot images, yay!) but alas, it's not quick and simple. It SHOULD BE, but I'm having Pascal's Apology again. Which is why I need to record a video tutorial for this. (Hearing it out loud helps me get a written version to be concise and intelligible too.

Whole bunch of work, that.

Recovery from travel.

A QEMU thread has me rebuilding all the qemu targets again, which is a bit of a time sink.

It's kind of hilarious that Ubuntu doubling down on Snap and SuSE doing a flatpak distro came out the same day. Snap is Ubuntu's proprietary version of flatpak the same way Ubuntu had the upstart init system, unity desktop, mir 3d compositor thingy... Ubuntu is run by a white male billionaire from south africa, not a whole lot of "listening" or "following" going on there. It's a real pity that their move to replace /bin/sh with the Defective Annoying SHell was swallowed by Debian, but Debian also switched to systemd which was approximately as stupid. Being at the mercenary of a billionaire's whims can't be comfortable. (Debian has an unfortunate history of FSF-adjacency, which means its development got so flamewar constipated the project almost died with many years between "debian stale" releases, and Connonical hired at least one a full-time developer to shovel out the mess on the engineering side of _debian_ (not ubuntu) because the open source project he'd overlaid his proprietary project on going under would have been embarassing. This was also the period where fleeing Debian developers squashed Gentoo, which that distro never really recovered from...)

Travel day. Onna Airplane.

I'm carting two pieces of checked luggage, the first of which is a suitcase inside a suitcase so I can fill one with Japan Loot for the return trip. I've probably missed Milk Seafood Ramen season (to Fuzzy's great disappointment), but I also left a bunch of clothes and books and stuff in the apartment when I left (meaning to return) over the pandemic, and it got packed away into storage and I should reclaim it.

The second piece of luggage is the GIANT CARDBOARD BOX with the oscilloscope Jeff sent me in the middle of the pandemic. Apparently the stuff they made 50 years ago is way better than the stuff you can get today, because nobody makes analogue waveform storage anymore and the digital equivalents are hundreds of thousands of dollars IF you can find something sufficiently high resolution. So when one comes available at a good price (usually because the people who knew how to operate it retired or died, and the inheritors don't value something they can't use), he snatches them up. This sort of thing can record signals for DDR3 and USB3 busses. We've done LPDDR2 and USB2 already because the cheap digital stuff can keep up with that, but anything faster gets expensive rapidly to see what's actually going across the wire.

Analog storage is the same general idea as a mercury delay line: giant capacitor that reproduces the wiggles of the input in its output, then you can loop it back on itself to retain the signal for a while. It is INSANELY high accuracy, calibrated with NASA-style equipment that sadly doesn't exist anymore. The downside of the analog stuff is A) the stored signal only lasts a few minutes, B) there's a hard cap on the SIZE of the capture because the delay between input and output is fixed and you can't record more than that at a time.

(When Scotty stored himself in a transporter pattern buffer for decades, the technobabble description was a bit like this. And in the Dr. Who episode Timelash, the 6th doctor used a McGuffin based on this principle to hit the bad guy with his own zap gun. Modulo "no, it's still totally murder when you're counting down like that while pointing the output at him, you could have just turned it to face the wall; there might be a limit to self defense when the dude is literally begging", but that's the kind of writing Colin Baker suffered under. At least he never had to deal with Yellow Kangs or the Kandy Man.)

Anyway, giant box under my desk in the bedroom became giant box in bedroom closet which is now giant box in Fade's apartement, which I hope to convert to giant box somewhere in japan that is no longer my problem. MAILING it to japan would cost hundreds of dollars (more than Jeff paid for it, that's why the seller only offered US shipping, not even to Canada), but it's just under the weight cap for checked luggage, and they don't charge extra for it being _bulky_. (The weight limit is a health-and-safety thing, maximum weight workers can be expected to individually lift between conveyor belts lots of times per day. Anything heavier than that requires two people to lift for liability reasons, and thus special labeling and handling procedures, and generally gums up the works trying to load and unload the plane quickly.)

No idea when Jeff plans to fly me to Japan, but hanging out with Fade until then. Disposing of Giant Box is a nonzero portion of the reason I agreed to the Japan trip. (I also really LIKE tokyo, and it would be nice if the stuff I worked on for years actually got launched out into the world, although toybox comes first these days thanks to Google.)

Bunch of errands today. Four bus rides.

I've meant to switch credit unions for years now, and as long as I was going to be down at UT after 9AM anyway... I'm seldom still there when I walk because I head back around sunup or I get all hot and sweaty, plus I can't see my phone display in full sunlight. And then I needed to go from UT up to The Domain to close the old credit union account, because that's the closest remaining Amplify location. University of Texas credit union has 2 locations and 2 ATMs within a half hour walk of my house, and two more at the university from a bus that picks up within sight of my driveway. The closest remaining "Amplify used to be the IBM Texas Employees Federal Credit Union but renamed itself" location is eight miles away, a two bus minimum each way or something like a $40 round trip on lyft WITHOUT surge pricing.

Fresh full backup of my laptop to USB drive. This SSD is old enough I'm occasionally checking dmesg to see if it's started to get unhappy about stuff. (Shouldn't, but I can be hard on things...)

Huh, corner case in the toybox test suite. So the general theory of toybox tests is a file full of testing 'name' 'cmdline' 'result' 'infile' 'stdin' lines (each is a call to a bash function) where the first argument's the name of the test to print on the PASS: line, the second argument's what to run, the third is the stdout output to expect, the fourth is data to write into a file named "input" (which only gets created when that's not blank), and the last is what to feed into the command's stdin.

Three complications to this: 1) The 'name' has the name of the command being tested automatically prepended to it so you don't have to repeat it every time, 2) there's a wrapper function testcmd which inserts the name of the command we're testing into the start of the 'cmdline' argument so we don't have to repeat it (and it makes sure we call it out of $PATH instead of a bash builtin by providing the absolute path when necessary), if the 'name' argument is blank it uses 'cmdline'.

The problem is that if you leave 'name' blank in testcmd it prepends the command name TWICE. Once when testing() prints the PASS/FAIL/SKIP line, and once in the testcmd() wrapper.

Got tired of waiting for Jeff to actually schedule a trip, and got a plane ticket to visit Fade up in minneapolis. (If I'm flying to tokyo, it should be from there.)

This means I have SO MUCH TO DO before then. Laundry! Fresh full backup of my laptop! Toybox todo items I should flush up to github... And it means I should NOT walk to the table tonight, because then I won't get anything done during daylight hours tomorrow because sleep schedule. Alas...

Fuzzy's birthday was on the 20th and we ordered her an Oculus 2 so she could play beatsaber, and it does not work. So we're returning it, which means I need to drop it off the return box at the Amazon lockers in Gregory Gym (the building with two first names), and I thought that was my excuse to do my 4 mile nightly walk watching anime on my phone despite the earlier "I shouldn't do that for schedule reasons"... but the building doesn't open until 9am. (I'm currently on a night schedule. The flight tuesday's noon-ish. Gotta impedence match between now and then.)

If I'm planning to be at the university during daylight hours I should get a new credit union account at the UT credit union on Gadalupe. Which means I should also close down my old Amplify account (which used to be IBM Texas Employees Federal Credit Union before they moved entirely out of Austin up into the northern suburbs. The closest location left is in <snootiness>The Domain</snootiness>, which is 8 miles from my house, an hour away by bus or bicycle. All their closer locations closed years ago.)

I deleted the Google Maps app off my phone screen back when it turned into all advertising all the time and stopped showing me black owned businesses (such as the haircut place I regularly go to in Hancock Center) even when I zoom in all the way, but sometimes I still need to see how far it is from point A to point B and what bus to take (and/or when things open, which it's never been quite right about since the pandemic), and when I do that I'm using the web version on my phone. Here's the SERIES of bugs I just hit in Google Maps' web version: enter the two addresses, hit the arrow on the keyboard to actually search and... it doesn't do anything. Plus it's scrolled itself to the right in a way that won't let me scroll back left so I can see the start of what's written on the page. And when I rotate it from landscape to portrait mode in hopes it resets itself... it loses track of the addresses I entered to ask directions about. It loses track of the location I was looking at, and instead reset itself all the way back to zoomed out full city view. That part's trivially reproducible, does it every time. Ask directions, type in the first address, rotate the phone, and the page undergoes a hard reset losing all context. Bravo Google. Your own browser in YOUR PHONE can't handle your website. That's... *chef's kiss*.

Anyway, from UT to The Domain is one bus (the 803). Yay. I should do that. (I don't want to give Patreon and such the banking info for the household account. I'm still paranoid about combining "money" with "internet".)

I got the ls --sort stuff checked in but not properly tested. Confirmed it didn't cause any obvious regressions in the test suite, but then got distracted by the whole Microsoft Github clusterfsckery trying to check it in. Had to delete the man-in-the-middle key four times before it stopped complaining. (IPv6 is not fun.)

Hmmm, tests/ls.test is ugly. Each test is bracketed with "cd lstest && $TEST && cd .." because otherwise the "expected" and "actual" files wind up in the current directory listing, and hence the output of most tests. The first being the output the test is expected to generate (argument 3 to testing()) and the second is the file output is currently redirected to, which are files so we can diff them and naturally get useful labels on the results. There's a fourth file, "input", but these days that's only created when testing() argument 4 isn't blank.

I suppose I could move them up a directory level? Because the action's taking place in generated/testdir/testdir, with the first "testdir" being where temporary binaries we're testing live. Since none of them are called "expected" and "actual" it shouldn't conflict if I use it as a work directory. (Modulo whatever Android's doing to use this test infrastructure, I THINK it should be ok? They use my scripts/runtest.sh but not my scripts/testing.sh which sets this up... Sigh, I should poke Elliott, shouldn't I?)

Walked to the bat bridge instead of UT, 25k steps total instead of just 10k, but my back was killing me when I sat down on the couch in Jester Center and I didn't get anything done. (They're kind of terrible faux leather couches on the second floor, mostly there for show I think, and my lower back's been unhappy since I slept on it wrong a few days ago, like a crick in the neck but older and more decrepit. I _really_ don't want this to become chronic because it wouldn't just suck, it would be CLICHE. The difference between being 15 and being 50 is problems resolving in about 8 minutes vs problems resolving in about 8 days. Lots easier for to fall behind on cumulative wear when it's not clearing itself nearly as fast as it used to.)

Finally dug up an old-style micro-USB cable that WASN'T a charger cable but actualy did data, so I can see the serial output on the turtle board. It works fine once I got a cable, but the linux-kernel I built and released last time does not work at all. (No output to serial once the bootloader hands off to it.) The one the sdcard had on it was linux-5.10 (dunno if I tested something newer since, that's just the reference version I know works), so there's some bisecting to do.

Huh, the musl-cross-make toolchain rebuild I did with gcc 11.3 earlier this month didn't build the sh2eb cross compiler because libgcc/unwind-pe.h had an error: '_Unwind_gnu_Find_got' was not declared in this scope which... I mean clearly it's a gcc bug, but what exactly broke? (It built sh4. Is this a nommu thing?) How do I track that down... What I _want_ to do is bisect it in the git repository, which is tricksy. It's slow to build gcc at the best of times, and mcm with my wrapper script doesn't do partial compiles.

I'm kinda tempted to compare the Linux From Scratch chapter 5+6 build script with musl-cross-make and just do a toolchain build script. If I have to fish out my own patches to make the build work _anyway_... I did that in aboriginal linux, this time it should probably be a proper project all on its own.

That's already 2 nested tangents from what I'm TRYING to do.

Got the LFS chapter 5+6 script building to the end. No idea if the result's actually useful yet, haven't done the chroot and started the second script. For some reason following the current LFS instructions, half the new commands _aren't_ in the /tools directory? They're in the normal paths. What's the point of the airlock step if you do that? I has a CONFUSED...

Alas, my initial naieve attempts to run record-commands to get a log of the host commands called for this build script... did not work. I need to update scripts/record-commands until it works right out of the box even when I haven't looked at it in 6 months and don't remember how I'm "supposed" to use it. (For one thing, it calls scripts/single.sh to build the log wrapper. It should check if "toybox" is already in the $PATH and symlink logwrapper to that if so, and only try to build it if it can't. Otherwise, you can't use it from anywhere OTHER than the toybox directory...)

I also have a github bug request from somebody who did scripts/mkroot.sh and then couldn't "ping" anything because glibc is crap at static linking. Um, yeah. That's why I added a "dynamic" script, but I've updated devuan since the last time I poked at that and now it's copying a bunch of symlinks into the target, including absolute paths outside the chroot. Unfortunately, when I add a -L to the cp -a the result is 1.7 gigabytes of usr/lib space because glibc is an insane pig, so I need to hardlink them back together to get the size down to a dull roar.

Back at the table again (I've missed this), putting together a Linux From Scratch 11.3 build script, so I can do the old trick of substituting in toybox commands one at a time and comparing the output to make sure nothing changed. (I should probably diff the config.log as well. To get consistent results I should do single processor builds, but I'm having the script make -j $(nproc) and then I can just "taskset 1" to force that single threaded later.)

Jeff thinks he might wind up flying me to tokyo on monday, but the hard part is working out hotels. It's cherry blossom viewing season there, which coincided with spring break in the states, and it's the first time in 3 years Japan's been open for tourists. The hotel room shortage has not eased up at all yet. Still a big staff shortage. They've announced plans to allow more foreign workers, but it apparently hasn't manifested results yet...

At the table, with a can of checkerboard tea. It's been a while. (Ok, I'm at one of the tables NEXT to the original one, working on battery because the outlet's blocked off, and ignoring the construction fencing. But still: same porch, same lighting, same comfortable seating.)

Poking at dd.c because I had the tab open, and... ok, that's a kind of painful use of TAGGED_ARRAY. There's nothing BUT the strings and the position indicator for the strings. This makes me sad. There's gotta be a better way to do that. I'm not sure what that better way IS, but this is ugly...

And now distracted by the half finished ls --sort plumbing, which I have now finished and the result compiled and failed the very first test in "make test_ls". Great.

I have done something to my back while sleeping. It's like a crick in my neck, except lower back, and I'm on something like day 3 of this. Reeeeally hoping it doesn't go chronic.

There's an i2c bug report on github that's been... badly explained repeatedly. I think the submitter doesn't have english as their first language, and I have no i2c domain expertise, nor do I have a test environment, which is why I haven't done the normal level of cleanup on this command, which ALSO means I haven't done as much review.

Because writing code is easier than reading code, I tend to rewrite as I go to utilize my far-more-practiced writing code muscles to help with the reading. Yes I know it's a bad habit, and sometimes I throw away the result because it's just marking stuff up in red pen, but that seems a waste with toybox? If I'm gonna clean up the code and thing the result is an improvement, I want to check it in, but I can't test for stupid thinko/typo regressions if I don't have a test environment and ANY change can theoretically introduce a regression. I've borked semicolons or bracket nesting levels in code refactoring before (back in my tinycc fork), and the result compiled but subtly misbehaved. Gotta test. CAN'T test. It's a problem. I've USED i2c tools on various board over the years, but it was all at contracts where I left the hardware behind with the job. My laptop hasn't got it. I don't THINK the turtle board does either but when I just tried to boot it up I didn't get serial console... it uses the old pre-C usb cables and I think this one might just be a charger cable not data? (Why do they DO that?)

I'm poking at qemu to see if that has a good test environment for i2c somewhere, but none of the ones I built did because the kernel hasn't got CONFIG_I2C enabled, and when I switched that on (and CONFIG_I2C_CHARDEV because that's not enabled by the first thing for some reason), then there's DRIVERS: I2C_SCMI, I2C_CBUS_GPIO, I2C_GPIO, I2C_OCORES, I2C_PCA_PLATFORM, I2C_SIMTEC, I2C_XILINX, I2C_MLXCPLD, I2C_VIRTIO... Plus whatever I2C_HELPER_AUTO is for... protocols? I suppose I could just switch it ALL on and see if any of the QEMU board emulations bind to something? Whatever I come up with should probably be added to scripts/root/tests so I can build regression test systems that do this automatically, but first I need to make it work _once_.

You can sing "closing tabs" to "closing time".

Trying to collect old superh patches for Glaubitz (the new arch/sh maintainer in Linux), but... there's so much old debris here and I have no idea what's still relevant. I collected lots of groups of 4 or 5 patches at a time and sent them to Rich when he was nominal maintainer, most of which never got applied, but I didn't exactly archive them again afterwards. (Checking back email in my sent box is one of the avenues of investigation here...)

Hah, scp-ing my blog file and corresponding rss file up to the website takes LESS THAN A SECOND with the new router. The old one took long enough I usually tabbed away and came back.

I keep meaning to find a way to post these bog entries to mastodon so people can reply there. This thing is a text file I edit with vi and periodically rsync, with a python script that generates an rss feed based on the lines that start each entry being regular enough (mostly thanks to cut and paste) that the text parsing to chop stuff out and plonk it into wrappers is pretty simple. But there's no WAY I'm turning that into an activitypub feed any time soon, and

Mastodon can provide an rss feed, but not let you FOLLOW on rss feed. Or easily convert an rss feed into mastodon posts at some known @user@server account. (If you google for it there's dozens of weird little projects on github or websites to do-it-as-a-service that seek to address this, but no real winner emerges and Google's search ranking to indicate which ones to look at first has deteriorated into uselessness over the past few months. My wife regularly complains about google becoming useless and she's not a techie.)

Blah, I need network block device tests, which is fiddly both because it's a client/server thing requiring root access AND kernel support for the /dev nodes, but also because the server and the client test against each other and "make test_nbd_client" would build just the client and then try to grab the server out of the $PATH, which most likely isn't there. As with the tar --xform stuff needing toybox sed, the test is looking at a _combination_ of toybox commands, which... the current test suite isn't really set up to do. (Well, "make tests" that tests ALL of toybox at once can, but not in a more granular fashion.)

I can have the nbd-client test check that nbd-server is there and fail to run if it isn't, but... the tests are mostly the same on both sides? Sigh, what are the tests:

1. nbd-client can mount nbd-server device on loopback, read a 4k block from it, write a 4k block to it, flush, and exit.
2. nbd-server -r: can export a read-only file, client can mount it read only and read from it, client can't mount it read/write. (Or does it fall back to read-only like iso9660?)
3. nbd-client without -b does default to 4k, and -b 1024 is a different block size. (Different ext2 filesystem mounts care about underlying block size for reasons I'm not entirely clear on, but it gives a failure case to check).

Hmmm, so far most (all?) of the toybox servers are inetd style. I should probably find some way to indicate that in the help text? Ok, sntp isn't because that's a UDP protocol, and figuring out when a UDP transaction is _finished_ is AI-complete. By which I mean "C3P0 could do it, but I wouldn't trust chatgpt near it". There's one of those P=NP things going on with this AI nonsense, where closing the gap is likely to take multiple lifetimes if it can be done.

Possibly I need more lib/net.c code to do a server wrap thing that takes a callback function? Except then my httpd and nbd_server need more command line arguments to indicate the server and port to bind to, which is a UI issue. Hmmm, I need to revisit httpd anyway to add the rest of cgi support. And nbd_server already says "ala inetd" which is funky since I don't have an inetd in toybox.

Long ago the samsung guys contributed tcpsvd to pending, which doesn't share any code with netcat. It does do a number of things netcat doesn't: limits on simultaneous connections, sets a bunch of environment variables... it also doesn't support nommu (which netcat server mode does), and combining vfork() with the -h option to look up remote connections (which can take an arbitrarily long time) does NOT sound like fun. Um, wouldn't the -b N thing be rendered irrelevant by kernel syncookie support? It's been YEARS since I've looked at that, where does -b get used... no FLAG_b and it's not TT.b it's... Sigh, count the arguments: TT.bn. Am I going to have to clean this thing up just to properly EVALUATE it? Grumble grumble... I really dislike duplicate infrastructure, but at the same netcat doesn't track multiple children. Plus this one hasn't got the "cat" part, it's always setting up filehandles and leaving the reading and writing of them to a child process.

Hmmm... I suppose I could clean it up and potentially merge them _later_? They have a hand-rolled hash table implementation. It's doing an error_exit() on recvfrom() errors. Is there a UDP packet you can send that DOS the server? (I remember TCP out of band data, but not UDP?) Why is it using sigemptyset/sigsuspend instead of just pause()? Does tcpsvd MEAN to write a trailing nul byte on the message part of -C COUNT:MESSAGE or is this an accident? (What do other implementations do? Is there a spec? Sigh, break down and look what what busybox does: no they do not have a trailing NUL byte, and they use nonblocking send() instead of write, which seems kind of important. Although I could probably fcntl(F_GETFL/F_SETFL) to set O_NONBLOCK, but why when send() exists? I could also check MTU length vs the message, but again... simple thing.)

Oh this looks like a long cleanup. And learning domain expertise. Why am I opening another can of worms when I'm trying to CLOSE TABS again?

Got the new toolchains built with gcc 11.2, the patch worked and I should poke dalias about merging it into musl-cross-make. (It's a backport, this should not be controversial to upstream? But then I felt that about the kernel patches. Oh well, it's on the #musl backscroll, maybe he'll notice...)

Built scripts/mkroot.sh CROSS=allnonstop LINUX=~/linux/github followed by scripts/test_mkroot.sh and everything except sh4 and the "no kernel" targets (armv4l armv7m microblaze mips64) passed, and the sh4 problem is the qemu+kernel clock issue (that emulated board isn't getting a battery backed up clock, and I ran the test without my laptop connected to the net so it can't set the clock from NTP).

So the new toolchain's working as well as the old I guess? More warnings, such as 'sprintf' argument 4 may overlap destination object 'ifs' in sh.c which... Ok, I can see an "even more optimized" version getting that wrong and I should maybe switch that to memmove() but first I should refresh my "what data is in which variable" mental working state which implies I should have a LOT more comments here (and possibly rename some variables) but reading through this code I did a couple quick simplifications but NO I have like FIVE DIRTY VERSIONS OF THIS FILE to collate already (I was working on this at Fade's last month, where was that... I _just_ dirtied the toybox/toybox file which had previously been clean, where's the recent... not in clean, not in kleen... it's in kl2). Ahem: NOT NOW...

Yay, somebody who seems to know this i2c stuff finally piped up on the confusing bug there. I still haven't got a test environment, and "get a raspberry pi working" is not ideal there. (I've been meaning to do that forever: their bootloader needs horrible proprietary blobs to bring the system up, the hdmi+keyboard setup in front of the TV is awkward and the connections buried and I haven't got a hdmi monitor for the desk in the bedroom (been trying to get out to Discount Electronics to buy one for months but they moved 5 miles further away, up near where Fry's used to be), and the only non-broken pi case I have is in use on my turtle board... Sigh, I should sit down and do it anyway. So many tangents.

Speaking of tangents, the recent "cpio -i extra garbage arguments" thing really SHOULD have them be extract filters, and opening cpio I see I have that "cpio skip NUL" test still not passing on the host, and a TODO about hardlink support since that's what the TRAILER!!! entry actually flushes (the cached hardlink detection), which means I really should try to get the other mother to sew buttons onto the hardlinks in a test directory to confirm what the output looks like, and then confirm the kernel's consuming it the same way AND poke the people who were talking adding xattr support to initramfs...

Sigh. Pull a thread in this jenga tower...

One of my early posts to mastdon was reminiscing about how the INTENDED use of a Tardis in Dr. Who seems to be for very long-lived Time Lords to bog off to deep space or some deserted beach for a few years while they catch up on STUFF, and then return 5 minutes after they left actually caught up on all their reading and browser tabs and todo lists without the society around them moving on so they missed anything or accumulated new todo items while they were gone. And the Doctor got in trouble because using one to go to a planet and interact with people was Doing It Wrong.

Yeah, 500 years between regenerations (both because the second doctor said he was about 450 years old and the eleventh lasted about that long in his little exile town), the ability to pause the world for a decade at a time in a nice quiet workshop area with kitchens and libraries and swimming pools and long corridors to walk down... I can definitely see the appeal.

The coreutils guys have got their knickers in a twist about new gcc releases breaking trying to build existing packages again, and rather than go "our code didn't change, yours did, this is your bug", they're capitulating because gnu. And providing horrible emacs examples.

Anyway, I should probably try newer gcc so I'm at least not surprised and can have -fno-stupid-thing workarounds prepared for fresh compiler bugs from C++ loons? The current musl-cross-make git version has gcc 11.2.0 as its newest toolchain... And it broke. Then new version can't even do a canadian cross:

from ../../../../../src_gcc/libstdc++-v3/src/c++17/floating_to_chars.cc:31:
build/i686-linux-musl/i686-linux-musl/obj_gcc/i686-linux-musl/libstdc++-v3/include/fenv.h:58:11: error: 'fenv_t' has not been declared in '::'

The line in question is "using ::fenv_t;" which can't possibly be a good idea.

The fix is to tell the libstdc++ build not to include the standard C++ headers in its search path. No really! Adds a compile flag. (And according to heat on the #musl irc channel, that's what got merged upstream.)

No wonder each new release breaks. It failed to build itself with itself, and THIS SHIPPED.

Weekly call with the J-core engineering team. Still no word about actually going to Tokyo. The tourists are back, it's sakura season. You'd think the overflow of hotel rooms from the olympics would mean they aren't all full, but having plenty of ROOMS does not mean having plenty of STAFF to service those rooms, and everybody got laid off during the pandemic. Japan does not have extra people in general these days (under the age of 60, anyway), and the covid restrictions allow tourists back but not yet foreign workers to run cash registers. It's apparently a problem, but there are worse problems. (The "nobody has any money, everything's going out of business" problem has at least been arrested by the return of the hordes of tourists. Although a lot of individual shops didn't survive.)

The new router arrived, and we eventually got it set up without installing any apps. It is SO much faster than the little white circle from Google (and the signal strength bar is green rather than yellow when my laptop's on the desk in the bedroom), although we haven't gone all "office space printer" and smashed the google circle with a hammer yet because we're giving it a few days.

The fiber connection itself is actually quite nice: the router SUCKED. The _service_ is mixed: why can't we get a static IP for less than twice what we're paying for the connection now? It's LITERALLY THE SAME SERVICE with a trivial config tweak. Wasn't the whole point of IPv6 that even if you can't get a stable IPv4, everyone everywhere could have a stable IPv6? But no, they want to capitalism at us.

Downloaded a fresh LFS book, and the magic all-in-one source tarball which should probably be more well documented, and I should automate another build and then try to get mkroot to do it. I can insert a toybox dir at the start of the $PATH and switch over commands one by one, just like I did with busybox back in the day.

Alas I'm not feeling inspired, because I have too many open tabs. Closing tabs tends to be hard because they're all only still open if I didn't get them closed last time I sat down at it. But starting anything NEW just makes it worse. And it's deep into "if I work on anything specific I'm not doing anything ELSE" territory. Generally a sign I'm still undervolt. (The cedar pollen is not helping.)

Still not feeling great, but I should do stuff.

I reached the point of editing and uploading blog posts where the entire entry for Feb 22 is "Oh god, kernel people." I know exactly what that's about but... really don't WANT to expand it? For the same reason I stopped replying to the kernel threads. Can I just use old kernels? I want them to stop breaking stuff that USED to work.

Sore throat, couldn't sleep. Spent most of the day huddled on the couch.

Tried to watch the "campfire cooking" isekai with Fuzzy, which Did Not Work because of the stupid Google router continuing to die. (I tried associating my phone with Google's router to save bandwidth for like five minutes when I got back, and then undid it again because even when T-mobile is throttling me for going over my 50 megabyte monthly quota it's STILL WAY FASTER THAN THAT STUPID ROUTER.)

This was finally enough for us to break down and get a new router. It was $50 cheaper to overnight a netgear from Amazon than to buy the exact same router at the Best Buy a fifteen minute walk from here.) So far it looks like it needs an app installed on somebody's phone to set it up (the card in the box says what app to install, or gives a URL to talk to a support being; no other instructions), so we haven't actually swapped it in yet, but there SHOULD be a way to talk to it directly...

Sore throat. Kind of lurgy-ish. Trying to figure out if this is allergies or dryness or microorganisms. Possibly it's a team effort. And, of course, I'm old.

Jeff got his contract signed, which means I may be heading back to Tokyo to help him organize the giant archive of stuff we did so it can get spliced together into a new product. Historically speaking, I can do toybox stuff from tokyo MORE easily than from Austin (he hates Apa hotel rooms, I find them just about my platonic ideal of a work environment, with a conbini downstairs for lunch rice balls), so...

We rebooted the Google Fiber router yesterday because it had become unusable again. Today it's already bad enough that reloading the household slack tab (after a "pkill -f renderer" because chrome was taking up too much memory again) did the ?cdn_fallback=1 then then added ?force_cold_boot=1 for the third attempt and then timed out saying it couldn't contact slack.

I don't mind google.com taking 7 seconds to load nearly as much as I mind being completely unable to use some sites, or thunderbird pausing for ~3 seconds between each email it downloads via pop3 (meaning a 400 message download takes over 10 minutes, so downloading my ~1500 daily messages is a background task that takes over half an hour).

Capitalism's really BIG failure is externalities. Engineers should be forced to dogfood their own products. I want THIS router put on the desk of the person who designed it, with all their traffic going through it, and to be forbidden from rebooting it for a week.

And yes, I'm happy to dogfood toybox. The main reason I don't already is I want a feel for what the other versions do so I can make toybox roughly match it. (When your frame of reference is your own output, it's really easy to spiral off into the weeds.)

Much wrangling with cpio, trying to fix three different issues. Got two of them fixed, calling it good enough since the third isn't a regression and nobody's waiting for it. (That's the "TEST_HOST fails, when did that start?" Moving targets...)

Pondering (st.st_mode&S_IFMT) == (mode&S_IFMT) and wondering if the compiler is smart enough to turn that into !((mode1^mode2)&S_IFMT) or if that's even a win. (3 operations vs 3 operations, although ! is only an operation sometimes? It could also go r1 = S_IFMT; r2 &= r1; r3 &= r1; branch-not-equal r1,r2 or some such. The repeated constant is PROBABLY something the compiler can handle for me, I don't need to go "that's redundant, I could rephrase it in a way it's not stated twice" and then ponder whether or not that's actually an improvement.

Ahem: premature optimization. Back away slowly.

Ok, I _think_ for the help fixes: "toybox --help COMMAND" should print Elliott's advertising line and "toybox help command" should not, and "toybox --help" is equivalent to "toybox --help toybox", but "toybox help" is equivalent to "toybox help help".

This is all UI stuff, so there isn't a right answer, but I'm trying to come up with an answer that makes sense for me without obviously disappointing anybody else.

I have 8 zillion accumulated 80/20 patches were I've done most of the work and then hit "does this cover all the cases, what ARE all the cases, and what are all the test cases I need to put this through to prove that" and I can't quite work that part out. I'd very much LIKE to check all this stuff in, but making sure it's _right_ is hard.

The sad part is I keep trying to grab low-hanging fruit, finding out the thing is not low hanging fruit, parking it at a good "almost finished but not feeling up to finishing just now" parking spot, grabbing OTHER presumably low hanging fruit, and then coming back a couple weeks later and having to reconstruct my mental state from scratch.

The external bug reports are actually easier to field because somebody else is waiting for me to finish and I can tell whether or not I've fixed their test case.

I'd like to have a nommu test system that runs under qemu, and "coldfire" (an m68k variant) is the oldest of the lot. The problem I had back under aboriginal linux is none of the nommu board emulations had the complete set of hardware devices I wanted (256 megs RAM, battery backed up clock, serial I/O, two block devices, network card), but most things have a serial console and I can fake the clock with sntp or an environment variable, and if I have a network card I can use network block devices. It's not ideal, but it's _something_. Alas I can't use swap on nommu so a board with only 64 megs ram isn't running modern gcc on anything complicated.

Alas qemu is terrible about labeling its boards (it's getting better, but there's no docs/system/m68k yet), I can go "qemu-system-m68k -M ?" and I THINK the first two boards there are coldfore (as opposed to the with-mmu ones that Linux inexplicably won't let me built a nommu kernel for) are an5206 (Arnewsh 5206) and mcf5208evb, the latter of which is the default board. As far as I can tell (from reading through hw/m68k/ar5206.c and hw/m68k/mcf5206.c) the 5206 has 128 megs ram but no hardware except a serial port? The 5208 has one network card, which is at least something.

So, back to the linux source: arch/m68k/configs has a file m5208evb_defconfig so let's build that and see if I can feed it to qemu-system-m68k -nographic -no-reboot -kernel vmlinux and hey: boot messages! Panicing because no initramfs. And the no-reboot is ignored implying this board doesn't know how to reboot or power off which is... sigh.

Memory goes from 40000000-41ffffff which... echo $((0x1ffffff)) is 32 megs ram. That's a bit squished. And it ignores qemu's -m option to try to give it more, which beats the cortex-m boards that were erroring out when you gave it any -m value other than the default. (QEMU may be undocumented, but at least its behavior is inconsistent.)

What else is in these boot messages: ttyS0 is the "mcfuart" driver. A dozen TCP/IP layer boot messages about hash table initialization and such but no line about the actual network card initializing itself. (Doesn't mean it didn't, which messages happen at which printk verbosity level is kinda potluck in embedded board drivers.) Oooh, mtd probe address, we've got a Memory Technology Device which means flash chip. Data storage onna block device, which QEMU might be able to stick a host file under. /dev/mtdblock0 which the "initramfs didn't work" root= fallback logic tried to mount as ext2... because apparently the default kernel command line (from qemu? built into the kernel?) is root=/dev/mtdblock0 and WHY does it bother saying /dev/ there? Honestly, what's the alternative?

Ok, I got a kernel to boot and spit out messages to serial port, which means I MIGHT be able to get an initramfs to boot to a shell prompt with serial console, even if I don't have any other I/O devices working yet. Assuming I can figure out how to get musl to...

Ah, darn it. I did this a year ago. And why did google not find musl's official web mirror on openwall? Google searches are getting RAPIDLY less useful, it's very annoying. I manually navigated to the right place but for some reason Google can't find that. Do THEY have a borked robots.txt? No, looks sensible. This is just Google increasingly sucking. I hope they recover.

Anyway, yeah, that's why I didn't do this earlier. Puppy eyes at Rich time again, I guess?

Took ADHD meds _and_ a store brand zirtec AND a prophylactic ibuprofen this morning, just for good measure. Actually able to concentrate for once, at least so far.

And Elliott's having build trouble on mac, which... how slow is it to launch executables on mac? Is it just a homebrew thing, or are all mac binaries latency spike city? And yes, I should have realized old version of bash without "wait -n" isn't just a centos thing, it's also a mac thing. So my centos hack is insufficient if you care about the mac build being well-supported, which Elliott does.

Checked in the fixes for the warnings from yesterday.

Grrr, tests/files/* is design-level wrong, but it would take a largeish rewrite to make it right. I need generally better organization for "not the actual toybox source" files: scripts/make.sh and scripts/mcm-buildall.sh and scripts/mkroot.sh and scripts/root are all slightly different categories.

Cycling back to the "help" redo...

Dear compiler loons:

toys/posix/ls.c:393:16: warning: too many arguments for format [-Wformat-extra-args]
printf(" "+FLAG(m), 0); // shut up the stupid compiler

But if I yank it, llvm goes:

toys/posix/ls.c:393:16: error: format string is not a string literal (potentially insecure) [-Werror,-Wformat-security]
printf(" "+FLAG(m));

So one compiler warns if you give it an extra argument, the other warns if you DON'T give it an extra argument, and in NEITHER case is it an ACTUAL PROBLEM. (Sigh, switching it to xputsn() but still. This is unsuppressable false positive noise. Stop it.)

Meanwhile, gcc is also going:

toys/posix/cat.c:31:32: warning: the omitted middle operand in ?: will always be 'true', suggest explicit middle operand [-Wparentheses]
int i, len, size = FLAG(u) ? : sizeof(toybuf);

It's not "true", it's a constant 1. Guaranteed by C99. I WANT it to be a 1. When the flag returns 0, I want to replace it with sizeof(). That's what that code is DOING. (This warning showed up when I added the !! because previously it was (integer&mask) which coincidentally was 1 but gcc wasn't treating "1" and "true" as different because THEY ARE NOT DIFFERENT IN C, THAT IS A C++ THING AND C IS NOT C++.

On the bright side, flag position is less important, so less of a lurking land mine. The cost is that gcc's rapacious stupidity triggers on more irrelevant crap. I'm sad I can't just compile with old toolchain versions from Before The Stupid, but I did that in aboriginal and there was a limit.

How limp did I go after getting back home? I was 3 days behind on reading my webcomics.

Sigh, I was so _amazingly_ spoiled by the speed of Fade's internet connection. I'm back here with Google Fiber and pages are taking 30 seconds to load, and email is downloading at one message every 3 seconds. (In batches of 400. It takes a bit.)

Finally applied Elliott's pending patch. (Saw it in the web archive yesterday but hadn't downloaded enough email to grab a local copy, yesterday I took my laptop to Wendy's and HEB and neither offered net access. Phone tethering drains the laptop battery fast, and the radio signal situation in Hancock center is appalling: it can't see my WIFI access point if I lay the phone on the keyboard, and around the Corpse of Sears (which Wendy's is across the parking lot from) my bluetooth headphones need my 6 inches from my left ear to avoid dropouts. Plus t-mobile did the "you have used 48gb of your 50gb gratuitous metering quota before we throttle the hell out of you" ping in the airport, and doesn't reset until the 5th...

So FLAG(x) now uses !! to force the return value to 0 or 1, which gets optimized away when it's used as a logic value. Audited all the users to remove a bunch of existing VALUE*!!FLAG(x) that are now redundant, and removed several subtle dependencies on a flag having a specific value along the way (some of which were commented, some weren't, including at least one subtle bug introduced by a commit that moved flags). There's still several VALUE*!FLAG(x) which now turns into VALUE*!!!(x&y) but the extra ! also get optimized out.

Whole lot of little style fixes as long as I was doing a review pass, spaces arround the = in assignments, removing inconsistently used parentheses, str = FLAG(x) ? "" : "K" becoming str = "K"+FLAG(x), etc. A few cases of "FLAG(x) ? TT.x : other" becoming "TT.x ? : other" which is actually subtle: sometimes you check the flag to see if it was set because it's an argument that only takes collated arguments, so --blah=abc sets TT.blah to "ABC" but --blah leaves it NULL. But I checked that it wasn't the case, and switched to the "test only one value and hopefully it's still in a register" version. (That said, I _kept_ one in patch.c because TT.p is numeric and could legitimately be -p 0 which is different behavior from not saying -p, so we need to check the flag not just the value.)

Whole lot of other "verification" of VALUE*FLAG(x) was _previously_ the rightmost flag, and not a hidden *4 or something. (The one case where it was had a comment.)

While I was there, I normalized todo and Todo to TODO so it's easier to grep for. (Can't just grep -i because "todo" shows up in comments and at least one local variable name.)

This wasn't (intended as) a micro-optimization to shave a few bytes off the code, this was "remove some conceptual land mines", but I did run bloatcheck a few times in hopes it wasn't making the result noticeably larger.

Oh goddess, this chunk of tar.c:

do {
  TT.warn = 1;
  ii = FLAG(h) ? DIRTREE_SYMFOLLOW : 0;
  if (FLAG(sort)|FLAG(s)) ii |= DIRTREE_BREADTH;
  dirtree_flagread(dl->data, FLAG(h) ? DIRTREE_SYMFOLLOW : 0, add_to_tar);
} while (TT.incl != (dl = dl->next));

Is assigning to ii but not USING it, the argument to dirtree_flagread() recalculates one of the flags and leaves the other zero. How is the passing the test suite? Would fixing it _break_ the tests?

Fixing it does not break the existing test suite. I'm gonna fix it and see who (if anyone) complains? (I think it might only affect sorting at the top level, which might not be a thing since even when the top level is a directory that's one entry. I need to think through it and come up with a test, which I dowanna do now because this is big and I want to get it CHECKED IN.)

I have SO much half-finished crap in my tree I need to FINISH and FLUSH. The recently help plumbing changes aren't quite done yet. My most recent bount of shell work. The lib/passwd.c rewrite. I get a bunch done but don't make it over the hump so it accumulates instead of reducing. Need to CLOSE TABS...

And I think even tar --sort isn't going to sort the command line arguments? Procesed inn the order provided, the CONTENTS get sorted. Which still gives us the stable ordering, which is what they were after... Ahem, NOT FOLLOWING THE TANGENT RIGHT NOW.

Still kinda collapsed. I have pending email from 2 people to reply to, and spent most of the day not doing it. So many tabs to close...

Alright, the design issue with the --help output is when should it have the toybox summary line? Going with my most recent release binary, it looks like "toybox --help ls" prints it but "toybox help ls" does not? I can work with that...

Sigh, there's a lot of THINGY*!!FLAG(x) and Elliott's most recent patch also modified code that assumpes FLAG(x) is producing 1 which is an artifact of position. (There's a comment about making sure the flag is in the right place in the optstr. That's... more brittle than I like.)

Possibly the FLAG() macros should have the !! built in? I should check whether the optimizer is smart enough to produce the same code. (No, I am not going to start using the "boolean" type.) Time to dig out make baseline and make bloatcheck! Which don't quite work here because changing toys.h at the top level doesn't get dependency checked and cause a rebuild, and "make clean" deletes the baseline out of generated/unstripped. Workaround: rm -rf generated/obj before make bloatcheck.It's not _quite_ the same output. (With gcc, anyway.) In do_sha3sum() it's because we care about the flag position, which should be masking instead of using the FLAG() macro anyway. In do_gzip() it's because we're passing the value to a function which does not appear to be being inlined so even though it's only ever being used as a logic value the status doesn't propagate far enough. In cp_main() FLAG(f) and FLAG(n) are being assigned to local variables which are then used as logic values... which shouldn't make a difference to code generation, but does? Ha! And when I yank those local variables and just use the FLAG() macros directly, it shrinks 34 bytes! In touch_main() it's another "we care about flag position" thing saving 3 bytes: I'll live, and cpio_main() is another "flag is 1" with a comment, and also assigning FLAG(t) to a variable which only cares that it's nonzero but the variable's incremented a couple times later (to make it nonzero) so... take the hit. In cksum_main() FLAG(L) is passed as a function argument, so "zero or nonzero" must become "0 or 1" (and crc_init() is in lib/ so I don't expect it to be inlined across compilation units). Still kinda surprised su_main() isn't in pending because that whole subsystem is still unfinished, but reset_env() is taking FLAG(l) as an argument which lives in lib/ so isn't inlined so doesn't see it's being used as true/false. In pidof.c print_pid() is returning FLAG(s) and that function isn't being inlined because the function pointer is passed to names_to_pid(). Ha: nl_main() was doing another "depend on the flag being in position 1" but did NOT have a comment about it... and there's about 5 more of those. Sigh.

Huh, patch -R looks broken: apply_one_hunk() did reverse = FLAG(R) and then part of the "allow fuzz" test was c=="-+"[reverse] which means it depended on FLAG(R) being 1, but when Elliott added -s in commit 6f6b7614e463 I didn't catch that he put it at the end and moved R to 2 meaning in the reverse case it'll be comparing against the NUL terminator instead of the '-'. And we don't have a test for autodetecting fuzz. So adding the !! would actually _fix_ this.

Alright, I think I want to audit all the FLAG() uses in toys/*/*.c because there's a lot of !! I can now remove, and I should be consistent about not parenthesizing VAL*FLAG()|VAL*FLAG() because * is higher priority than |. It's a pity there's no "make test_dmesg" to make sure I didn't break that. I expect this is gonna come up a lot in a treewide audit...

Oh hey, another email in my inbox this morning about what somebody thinks I SHOULD be doing instead of what I am doing. (Watched a good video on "autistic inertia". I've mentioned before that I work based on momentum, and there you go. Having something Looming can either be extremely motivating (avoidance productivity: I will do SO much cat waxing to "virtuously" avoid the Looming Thing), or extremely demotivating (loss of momentum and traction because I can't muster the executive function to address picking up that piece of paper, it just won't budge).

Anyway, ignoring the "linux-kernel community is so broken" pile, the question du jour in my email was:

what is the official toybox opinion on rust being added to toybox?

And "My gut reaction is "Oh goddess not again" and I've been actively ignoring it?" was too short, so Pascal's Apology kicked in, and I replied:

Define "added"? I'm not putting a rust compiler in toybox, if that's what you mean?

If you mean "should I implement some commands in Rust and some in C", having a single simple context everything is done in the same way is part of toybox's design goals? Early in Toybox development the build needed Python, and I cleaned out that build dependency so it's all C and bash, and I'm implementing my own bash compatible shell so toybox builds under toybox. Early on I even had some commands implemented as shell scripts, and I wound up removing them again and doing them in C even though I planned to ship a shell interpreter, because I wanted the whole thing to be a single file with no external dependencies which you could statically link and drop into an empty chroot directory and have it just work.

If you mean "rewrite the whole project from scratch in a different language", long ago I was thinking of rewriting the whole of toybox in Lua but the problem I hit is that Lua doesn't ship with a standard set of posix bindings so I had to install something like 7 different prerequisite packages just to manage things like "wget", let alone implementing "mount" or "ifconfig", and if I had to implement/ship my own new Lua bindings written in C (and cross compile those to every supported target architecture) I might as well just do everything in C. (Which is a pity, Lua was quiet elegant, but their deployment strategy was too minimalist to be usable on its own.)

If you mean how would Rust affect my variant of countering trusting trust then having the project be in multiple languages again kinda defeats the purpose of a minimal installable base capable of reverse engineered binary auditing.

If you mean coming up with a replacement tiny system written in a single language that's both learnable the way minix and xv6 are _and_ scales up to actual load bearing deployment in real world usage (the way Linux 0.95 through about 2.2 did)... I'm still trying to make that work in _C_ (well a non-GPL one, I had it working with busybox but the insane FSF poisoned that well so thoroughly with GPLv3 around 2007 I wound up starting over.) I'm told the Rust compiler is now written in Rust and dunno what its system call binding approach is, but I still await a Rust kernel that actually ships in a product. (Even a vmworks level of kernel: I decided to wait for that when I saw that blonde lady's Rust talk at linuxconf.au in 2017 and I'm still waiting. Heck, even something as silly as Fuchsia, just something somebody somewhere actually used for something in a non-demonstration manner. There's a dozen different My Little Kernel variants people have done, but nobody actually seems to do real work in Rust? It's all either reimplementing stuff that already exists because "Ew Icky C", or "here's how we're going to change the language governing bureaucracy" and "here's how we're going to add yet more complexity to the language" and it's very tiring...)

Show me a serious attempt at a system that rebuilds itself under itself from source code, all written entirely in Rust with no C anywhere, and I might start to care? ADDING Rust on top of existing complexity is just more xkcd standards layering. (Yes, you have garbage collection and bounds checking like Java did in the 1990s. Yes you have native compilation to binaries like Java did with IBM's Java Native Compiler back in the 1990s. Yes you have a Big Marketing push and drive to rewrite everything in this one language like Java did in the 1990s. Yes you have a strong argument that C++ is a terrible language like Java did in the 1990s, which is not the same as C being a terrible language but try telling any C++ developer that. From a safe distance. Bring popcorn.)

If you mean the "Rust is inevitable, the same way Hillary Clinton was in 2008 and again in 2016", I note I've lived through the following:

• 1990: We don't bother to teach C in college, we teach Pascal instead.
• 2000: We don't bother to teach C in college, we teach Java instead.
• 2010: We don't bother to teach C in college, we teach C++ instead.
• 2020: We don't bother to teach C in college, we teach Rust instead.

I learned C in 1989, spent about 1992-1995 doing C++, and then was all in on Java as my main programming language from 1996 until about 2000, caught the Python 1.x->2.x transition and then bowed out again when staying on 2.x actively offended the 3.x developers... The C I learned way back when remains relevant. If I tried to write new code in the ~1995 version of any of those other languages it wouldn't build in modern environments.

I was part of the "rewrite everything everywhere in Java" crowd for about 5 years. My bug report was the reason the ability to truncate a file was added to Java 1.2. I worked on IBM's port of JavaOS to the PowerPC in 1997, taught Java at the local community college in 1998 and 1999, designed a hard realtime garbage collector... It was really exciting. (I wrote a little about how that ended in my blog.)

Any time someone goes "why aren't you using Rust" as an accusation, I treat it the exact same way as the C++ and Java people doing that before them. I had 20 years of Windows people asking why I didn't do windows (smoothly transitioning from rejecting OS/2 to rejecting Linux). I don't care if "everybody's doing it", I've never had a Facebook account either. It's not _my_ job to "be convinced". Lua had "here's cool stuff Lua does better", which appealed to me enough to take a look. I have yet to see arguments in _favor_ of rust, they've all been _against_ C. "C bad, icky and dangerous, we blame you for perpetuating it, you must stop now". No thanks.

A big reason I keep coming back to C is I can stay 10 years behind on the standards without a problem. Heck, I can still compile K&R stuff from 1978 if I really need to. The main deficiency of ANSI C from 1989 is that the first 64 bit processor came out in 1991 so the 64 bit "long long" type was a widely implemented compiler extension that worked its way into the standard later. I only moved toybox from C99 to C11 recently because of like ~3 minor convenience features (typecast array/struct literals, the "has_include" macro, and an alternate "inline" syntax that let us work around an llvm bug that's probably since been fixed).

Rust still hasn't settled down and decided to be nearly that stable: from a distance it looked to me like the first decade or so of the language was just WILD THRASHING leaving the language unrecognizable 5 years later, and now it sort of knows what it is, but still changes?

Has anybody made Rust work on a nommu system? Or only XIP from read only storage with 256k of sram? (Which Linux has been made to do, for example. Good luck pulling that off with garbage collection...) If not, your argument is "we'll still need C, but just less of it, so a smaller pool of people will have less expertise and age out without replacement". That's kind of Tesla's version of the self driving car argument: 99% of the time it'll drive for you just fine, and the remaining 1% it will crash and/or kill pedestrians and we're calling that the driver's fault but the driver won't be paying attention and may be way out of practice assuming they ever knew how to drive in the first place. How this is supposed to be a net improvement, I couldn't tell you.

Is there a Rust version of tinycc? What's the smallest, simplest Rust compiler out there? (Tinycc could happen because the language wasn't a moving target. If I decided to pick it back up and bang on it again the old stuff I did is still theoretically relevant. Is even a 5 year old version of Rust still relevant?)

If you want to implement commands in rust yourself, you can stick them in the $PATH and it should just work. Is there an obvious reason this should have anything to do with toybox? The "start over and rewrite everything in rust" approach like I was poking at doing with Lua would mean getting all four packages written in Rust. And preferably a stable version of Rust where a newbie could grab an existing system deployed 10 years ago and not touched since then, fire up the old build, reproduce it, understand it, and be able to modify it. As far as I can tell, this isn't a thing the Rust community _wants_, let alone is actively trying to achieve.

Sigh, I haven't got anything _against_ Rust, any more than against Ruby or PHP or Lisp or Prolog. I just don't care. Nor was I _offended_ by the people submitting forth and lisp interpreters (yes, plural) to toybox over the years. (In the absence of toysh, people have decided it needs a programming language.) I understand this guy's interest, and would like to politely decline... except I DO have something against projects like systemd that don't give me a graceful option not to participate, and the push to rewrite the linux kernel in rust without forking it is exhausting in the same way the build requiring perl was exhausting.

This guy didn't exactly knock on my door with a rust version of The Watchtower to tell me the good news about our new savior, but... I'm not getting "live and let live" vibes from this community either.

(I have a youtube video bookmarked, which claims to explain Rust in an hour. It's on my giant to-watch heap. I'm not AGAINST Rust. I just... still don't see the point?)

Fiddling with toybox help plumbing. Kinda spiraled.

So "toybox --help toybox" wasn't producing output, because of fallout from changes to prevent "toybox toybox toybox" stacking arbitrarily deep (and blowing the stack now that Linux doesn't necessarily enforce environment size limits even on mmu systems). So I started poking at that, but the show_help() flags API did the old "this argument was a yes/no boolean, then it grew a second bit, then it grew a third bit, and now it needs #defines" thing that I hadn't cleaned up yet. And while I'm there, "help -au" should print the usage lines for all commands, but calling help as a shell builtin does unique filtering so what happens when you "help -u" on the builtin? And the "See:" logic isn't filtering right as a builtin (redundant lines). And this whole "Toybox 0.8.9 multicall binary (see https://landley.net/toybox)" line at the start (which wasn't my idea, but then calling Linux "linux" wasn't Linus's idea either) should only be output SOME of the time and when is that some?

I keep trying to do quick fixes that wind up touching a half-dozen different files and leave off unfinished after hours of work and then it just ADDS TO THE MESS.

Flying back to Austin on tuesday. Not up for programming stuff today. Reading fanfic on AO3 instead.

Some months back I posted an observation about the Tardis to mastodon, which is why I want one. Just catch up on everything and come back when you're feeling up to it.

I wrote up an email reply which is a bit rambling and off topic for the toybox list (see "not up for" above, combined with pascal's apology for writing a long letter, substituting "spoons" for "time") so here it is instead. The context is that Michael Kerrisk, the man-pages maintainer, retired and handed the project off to a new guy, and didn't properly announce it (quietly added a co-maintainer to the git repo and then ghosted everybody), and now that we've finally figured out what HAPPENED we're trying to adjust.

On 2/24/23 11:46, enh wrote:

> > Possibly the new maintainer needs to poke Konstantin to get access to update the
> > directory, and then put stuff under the actual kernel.org page? (Or you could
> > put some under an android.org location? Either way they'd be up to date with the
> > repo instead of a couple years behind...)
>
> yeah, that's one of the options... generate the html and stick it on one of the
> android-specific sites, but that seems a bit odd (people are already confused by
> places where the man pages are actually only talking about glibc; hosting them
> on an android site would only make that worse) and there are already a lot of
> links to man7.org out there in the wild, that it would be
> unfortunate to see go stale. (though if no-one has access to man7.org
> any more, there's nothing we can do about that anyway.)

The downside of depending on individuals is you're inconvenienced when they cycle out. The downside of depending on organizations is they're all just a bunch of individuals who get together and collectively pretend, so things go just as pear shaped when the people actually doing the work well leave without a proper handoff to someone else who will actually do the work well, but you tend not to notice as fast (before _or_ after: see the Linux Foundation's consumption of the Free Standards Group and thus the Linux Standard Base). This lack of warning isn't necessarily an improvement.

Ahem: man7.org was offered as a community resource but is actually Michael Kerrisk's personal page and he is not handing it off to the next guy. (The maintainer of landley.net does not get to throw stones here, although all the toybox.net variants are camped by people who want thousands of dollars.)

The responsibility for the man-pages git repository was handed off (resulting in the repo effectively moving to a new URL which nobody seems to really care about), but not the website or the release announcement email list. (Haven't gotten one since, if it's still having releases?) If it's a good idea for the project to move to more of a "package deal" where there's a repository+website+mailing list that can be passed to a new maintainer as a group, that's sort of a design issue.

Jeff Dionne set up the original uclinux project, which I believe busybox.net was modeled on after linaro ended in the dot-com crash and the kernel parts got merged upstream and Erik Andersen kept the busybox+uclibc subset of uclinux going as a personal project. He handed busybox.net off to me in 2005 by giving me a login to the server (it moved from the DSL line in his basement to osuosl, but Erik still pays for the domain renewals). When buildroot forked off of uclibc, I'm the one who abused my root login on the shared server image to create a new mailing list and kicked the buildroot traffic off to the new list. (Alas, too late to save uClibc.) Buildroot has since separated itself the rest of the way from uclibc (its own VM with its own domain), so it's not inconvenienced by shared infrastructure going down (as has happened a few times what with uclibc being dead and all), which also means that handing over the keys to a new maintainer is a thing that buildroot could potentially do if necessary.

Sigh, somebody should write up a non stream of consciousness "handing over the keys of an open source project to a new maintainer" document. Do you even have a manifest of what all the project's resources ARE for something as big as Android? Not that Google's ever going to hand off Android. I remember when Red Hat set up Fedora and it (it pretended to be independent until Red Hat finally admitted it was just Red Hat Enterprise Rawhide. (So an independent Centos emerged... and Red Hat bought it.) Anyway, the point is when people/management change, the project's gonna wobble no matter what the corporate structure says because it's people who do things and know things and remember things, whatever they corporate structure says.

> > Let's see, how hard is it to produce html output from this git repo... it's got
> > a top level Makefile to do exactly that as its default target, but it wants a
> > package called "man2html". And installing that on my laptop installed apache
> > which LAUNCHED AN INSTANCE ON LOOPBACK. Why on EARTH would... that's just sad.
> >
> > But ok, I can uninstall it again after building... looks like it populated
> > tmp/html with files? No top level index. Let's see, the first file under "man3"
> > is __after_morecore_hook.3.html which seems to be a synonym for malloc_hook (not
> > symlinks or hardlinks, just redundantly generated files). The "Return to main
> > contents" link goes to file:///cgi-bin/man/man2html which does not exist. The
> > #include link goes to file:///usr/include/malloc.h which ain't gonna
> > work on a web server either...
> >
> > Looks like there's the start of something workable here, but it needs a bit of
> > shoveling? (Or at least digging into how to configure it?)
>
> yeah, and one problem with being part of a large bureaucracy is that the docs
> folks and the branding folks will all want a say in making it look "right" if
> it's on an android site!

My first really well-paid consulting gig was working at a dot-com that was managing a rewrite of IBM's mainframe pricing and sales system. Various departments within IBM had wrestled for control of the project so extensively that upper management had outsourced our bit of it so NONE of them had it. Taken the ball away and given it to someone else entirely so they'd stop fighting.

Which meant my job was to be on an 8am conference call with IBM Europe (Boblingen, Germany: initial deployment) and IBM USA (Poughkipsee and Dallas, one did frontend one did backend), a 6pm conference call with IBM USA and IBM Australia (Worldwide Integration and Test, it was _not_ in didjabringabeeralong because that's a Discworld reference but don't ask me what city it WAS in, somewhere that was simultaneously under water and on fire at one point but that came later), and when I needed Australia and Europe to talk to each other that was a 3am call and I slept under my desk AND BILLED FOR THE TIME. (The dot-com manager told me to.) I don't think I authored a line of code (for them) the entire contract, the _technical_ part of my job was matching up defect reports that told us to do one thing and defect reports that told us to do the exact opposite (or explicitly NOT to do that thing) and bring up pairs of them in the meeting.

Somebody eventually explained to me that a specific manager in Dallas (Ken somebody?) had figured out how to get promoted by sabotaging projects: during the design phase he demanded to know why implementation hadn't started yet, then when they started implementing an unfinished design he'd demand to know why it wasn't being tested yet... The answer was always "because we're not ready" but he'd make a stink and get it started and the reputation was Ken Got Things Done. It wasn't happening before he made it happen. It all collapsed into chaos the moment he left, but that just showed how vital he'd been didn't it?

So this project had fundamental design changes coming in regularly, requiring not just complete rewrites multiple couple years into the project, but constant changes to the test plan. ("Why can we never get real database data to test with?" "It's their strictest trade secrets." "What is this system for anyway?" "Pricing 360 mainframes." "How much do those usually cost?" "That's not how it works, the salesman figures out how much the customer is able to pay, and then they produce an invoice that adds up to that amount." "So this whole system is a giant bullshit generator that emits nonsense to produce a predetermined result?" "The invoice has to be reproducible and comply with a bunch of legal and regulatory clearance issues, you have to word things right for the technology to be exportable to various jurisdictions..." "You didn't answer my question." "No I did not.")

Eventually the Australians did a slimy clever political thing to extricate themselves from this cluterfsck death march, by declaring that one of the endless thrashing "release candidates" they'd been given had PASSED THE TESTS and they certified it as deployable, closed out their budget, and scattered to the winds reassigned the testing staff to other projects. Completely ignoring the fact that the testing they were doing was useless (something nobody else could call them on because nobody could, for political reasons, admit it to be true in so many words.) They took a random passing snapshot in time of the vague contradictory specifications they'd been given and ran the red queen's race fast enough to catch up just long enough to call Bingo. They'd been given an impossible job and claimed to have done it, because ignoring the "it's just busy work until we're ready for you" nature and instead declaring victory meant they could STOP DOING IT. Which immediately clogged up the pipeline leading to them, because NOTHING MORE COULD BE TESTED, an existential constipation crisis leading to ALL THE PHONE CALLS.

That's about when my 6 months were up, at which point the consulting company all this had been outsourced to offered me a 50% raise to just STAY AND BE ON THE CALLS... and I just couldn't. I couldn't put into words WHY, this was almost 15 years before David Graeber wrote his first article on "Bullshit Jobs". But at the start of the contract, the existing employee who'd been doing it had used all his accumulated vacation time AND some family emergency under the family and medical leave act to take a solid two months sabbatical, forcing them to reassign the project to ANYONE ELSE BUT HIM. They'd thrown money at a passing junior dev to just Sit In The Chair And Be On The Calls, and he'd left me a pile of useless printouts to "get up to speed" with. There WAS no documentation. The job was babysitting, and the burnout was just insane if you didn't understand that and tried to actually accomplish anything ever. I found myself physically unable to just shut up and take the money longer than I'd already done.

Anyway, tl;dr there are sometimes political advantages to having something live outside an organization.

> > > nope... that's still
> > > https://thephd.dev/c-undefined-behavior-and-the-sledgehammer-guideline
> >
> > I apply the sledgehammer to the compiler. (Push back against the abuser causing
> > the damage, don't make the victims endlessly escalate ever-changing "compliance"
> > that's never good enough. Danegeld encourages the dane.)
>
> read the link (or listen to what i've been telling you for the best part of a
> decade) --- the problem is that the compiler folks don't believe we're their-
> customer. they don't care about "is it useful?", they care about "microbenchmark
> line goes up?". or, in your analogy "the law is currently on the abuser's side".

Oh sure. I know. Doesn't mean I'm going to stop fighting. (There's a reason I was poking at tinycc/qcc.)

Steven Universe's "that's why we can't fight them", "that's why we have to fight them" line works in here somewhere.

Got an email from Andrew Morton which would have been great if it was the first one, but after Thomas Gleixner's repeated replies ignoring the code and talking about bureaucracy (I actually MET him, and later recommended his company to Taylor Simpson at Qualcomm for handling Hexagon's kernel patch review and upstreaming, nice guy back in the day...), and that japanese guy going "we voted on this stupid unnecessary API so adding code that renders it irrelevant would highlight how stupid it had been and embarass us all"...

I'm trying to scrape up the politeness to answer Andrew's questions in a constructive manner. Rather than an honest one. "Who do I expect to merge this?" Nobody. I do not expect the kernel clique to be functional enough in 2023 to merge external contributions from individuals. All of this code was submitted to the list before, and ignored. This is a roundup for people outside the kernel. If people build their own kernels, this can add to their patch stack. If lawyers give me guff I go "look, I submitted it to them, they chose not to merge it for their own reasons". But linux-kernel being a functional place to discuss patches? That's LONG gone.

But I can't just SAY that. It's not USEFUL. I'm not sure what would be, and dealing with them makes me SO TIRED. (Andrew is being polite and constructive! I should do the same! I really should. I'm just out of spoons for kernel "community".)

Many moons ago I was trying to add cortex-m support to mkroot but seem to have lost my notes. (I want a qemu nommu target so I can more easily test nommu support without copying stuff to my turtle board. Yeah, I can tell toybox to enable nommu support anywhere and use the nommu codepaths, but that doesn't prove nothing LEAKED, and that the result actually WORKS on a nommu system.)

My old blog entry from the time just says I was working on it, but doesn't provide useful context like what QEMU board or Linux defconfig I was trying to make work. So we start over.

According to qemu-system-arm -M ? | grep '[-]M' the list of QEMU Cortex-m boards includes "stellaris" (64k sram), bbc microbit (no obvious Linux target), and stm32 has two boards: vldiscovery and netduino, neither of which implement ethernet or block devices. That leaves mps2.

The mps2-an500 and mps2-an511 each have 16 megs DRAM, and qemu's hw/arm/mps2.c has a gratuitous explicit test for an -m trying to increase it and then refusing to do so: if (machine->ram_size != mc->default_ram_size) error_report("Invalid RAM size, should be %s", mc->default_ram_size); (Which seems silly, there's space in the mapping? Oh well...)

Linux has an mps2_defconfig build. I need a kernel config, QEMU board emulation, and compiler that all agree on the target, where "compiler" includes both gcc tuple and musl support. I have a static PIE toolchain for armv7m (I.E. thumb 2 I.E. cortex); I'd like an fdpic toolchain but haven't made that work yet because support hadn't been merged upstream yet last I checked. (gcc, binutils, and linux all need it, not sure about musl?)

Standard ELF has absolute memory addresses hardwired into it, which means you could only run at most one instance of each ELF binary on a nommu system (it's kind of the same problem a.out had with shared libraries: in practice the ELF loader just isn't allowed on nommu). Position Independent Executables (PIE) use relocatable Position Independent Code (releative addresses from a base pointer kept in a register), which is basically building your executables the same way you build your shared libraries, so they can be loaded anywhere in memory. It's slightly less efficient but the security nuts love it because exploit shellcode hasn't got known absolute addresses to use on the target system. FDPIC takes that concept and expands it to make all four of the standard ELF segments (text, data, rodata, bss) independently relocatable, which means your program doesn't require one big contiguous chunk of memory to fit into, but can instead fit into four smaller chunks (which is very useful on nommu systems, where memory tends to get fragmented over time), AND it means the read-only segments can be shared between program instances (five copies of bash can all use the same text and rodata, each one just needs their own data, bss, stack, and heap), but the downside is you need 4 registers to store the 4 base pointers (or have your base pointer point to an array of 4 pointers with an extra dereference on most memory accesses). But that's ALSO something the security guys like because foreign exploit shell code can't even know where rodata is relative to text in a given running binary, it's even fiddlier to exploit.

You'd think the FDPIC loader would be the standard one by now (since it can handle normal ELF binaries just fine: FDPIC is ELF with an extra flag in the header, it has the OPTION too make the segments non-contiguous but not the obligation to do so.) But as with the ext2/ext3/ext4 drivers the kernel guys went "no, fork it and have a completely seperate file that will get out of sync with the other one", and then years later it's a mess...

Oh god, kernel people.

[That's all I wrote for this entry at the time. It's now March 19 and I haven't edited and uploaded past this entry yet because I just don't have the emotional energy to deal with that toxic waste dump, but we're coming up on a month behind, so here goes:]

Thomas Gleixner replied, ignoring the actual code parts and instead having a multi-part exchange entriely about the bureaucracy, where I didn't cc: the right people (I cc'd who get_maintainer.pl said!) and my subject line was wrong and UGH, my DESCRIPTION and also there's something in some thousand line documentation file I missed but he literally won't specify what it was because the onus is on me to FIGURE IT OUT. And he's also arguing that if a dependency is EVER needed then it's ALWAYS needed so my patch to be able to build without objtool is conceptually wrong because SOME configurations need that dependency therefor EVER building without it is a crazy thing to want to do.

Meanwhile, Mashahiro Yamada is literally saying that my patch to try the "cc" name before falling back to "gcc" and thus autodetecting llvm in both native and cross compilers (with no other behavior change I am aware of) can't go in because, and I quote: "In the discussion in the past, we decided to go with LLVM=1 switch rather than 'cc'. We do not need both." (With a link to the previous vote.) This was his REPLY to me pointing out that the name "gcc" is like "gawk" and "gmake" (and "gsed" on macos homebrew) and that just about everything else uses the generic name where possible. What's his logic here, "we voted, therefore the topic cannot be revisited"? I just...

So tired.

Got my patch series posted to linux-kernel. The oldest patch in that series was first submitted over 15 years ago, albeit in a different form then. Another one fixes a minor bug I myself introduced 10 years ago, which nobody else has bothered to fix since even when I pointed it out to them.

If you're wondering why I'm tired of dealing with the kernel clique...

Dear bash, up yours:

$ bash -c $'cat << EOF\nthingy'
bash: line 1: warning: here-document at line 0 delimited by end-of-file (wanted `EOF')
thingy
$ echo $?
0

I'm trying to test ERROR PATHS to make sure they exit gracefully instead of throwing ASAN allocation errors, and you have WARNINGS? The shell has errors and the shell has success, having WARNINGS is new territory. (Still exits with 0... I do not have a syntax_warn() function.) Comparing with the Defective Annoying SHell... that accepts it without a warning and also exits 0. Fine, change the error path to be a... strange sort of success? (Dash does not append a newline to thingy, bash does. Dash doing it strongly argues in favor of NOT doing it, so always newline it is.)

Going through the HERE document parsing logic, I hit a "this can't work" bit (comparing two pointers only one of which gets incremented in the loop), and tried a simple ./sh -c $'cat << EOF\nhello\nEOF\n' test and sure enough it didn't (never recognizes EOF), and tried to find the last place it did and gave up in 2021...

That can't be right. There's a regression test suite. I know it doesn't make it thorugh all the tests, but... I had this working at one point.

Sigh, symptom of swap thrashing: sh is a big command that requires a lot of focus and I've had to do it in small increments with all the other demands. Now that I'm focusing way more on toybox, there's still lots of bug reports that spawn off tangents that SEEM quick but aren't. (I spent a couple days basically SCOPING mkisofs. I need to cycle back to diff. I still haven't set up a test environment to check in the lib/passwd.c rewrite that's actually buildable and testable with the bionic NDK...)

I've fixed a lot of sh bugs that were in front of me, which broke other stuff and I either hadn't made it though the test suite (because of the expected failures from still missing features) or the relevant test isn't in the test suite yet. So I need to grind away at fixing stuff for a sadly large hopefully uninterrupted block off time.

I miss the 36 hour porogramming sessions of my youth. These days I look up after 4 and need a long walk...

Sed bug report came in while I was poking at the shell double free, and of course the sed thing is another object lifetime rule issue, introduced by the sed speedups which added extra cacheing. Got it sorted I think?

The double free is in an exit path, where the cleanup does not match the assumptions. The HERE document logic adds the EOF marker to the end of the ARG list: not a COPY of the marker, the actual pointer to the original string we parsed earlier. The sh_pipeline variables "count" and "here" let us know we're in HERE document accumulation mode so each time parse_line() gets called it moves the marker and discards it when matched, but the cleanup function called by the exit path isn't looking at that.

There's also something I called "bridge segments" where additional commands that do NOT have HERE documents attached to them get parsed before the line continuation logic fetches the body of the HERE document(s), ala:

$ cat<<EOF; echo hello
potato
EOF
potato
hello

In that case the pipeline segment "echo hello" parses into would get marked as a bridge (its ->count set to -1) so the parse_line() entry path knows to back up through it and look for uncompleted HERE document segments. Once they're completed it works its way forward unmarking completed segments until it can either return "we have a complete thought, you can execute it now" or finds another reason to ask for line continuation (being in the middle of a for loop or if statement, for example).

The PROBLEM is that when you DON'T complete the HERE document, that extra entry indicating what EOF string we're looking for shouldn't be freed, because it exists earlier in the pipeline (in whatever statement had the redirect) so if you free it in both places... double free.

Alas while fiddling with this I found MORE wrong cases. For example, if the redirect ISN'T attached to a statement, it gets freed early (when the NOP statement is freed) an thus the HERE document can't be concluded for a different reason, ala bash does:

$ <<EOF; echo hello
potato
EOF
hello

And toysh can't handle that yet because free("EOF") happens after parsing the first line and then the HERE document fetching use-after-frees it.

I think I need to just xstrdup() it. Premature optimization strikes again, "I don't need to copy this, the original's lifetime is longer than HERE document parsing by definition"... "yes I do to be CONSISTENT".

Sometimes "progress" is just adding yet more tests the existing code doesn't pass.

I've been meaning to post my patch stack to linux-kernel for weeks (not because I think they'll merge it but so it's not my fault that they haven't), and hey: Linus did an -rc8 so this isn't the merge window week. Yay extra time, but I sat down to do mkroot builds of 6.2 anyway and...I broke the shell. Darn it. One of those fixes for Eric Roshan-Eisner's fuzzing bugs introduced a strcmp(ex, blah) without a test for ex being null, and running mkroot's init script triggers that codepath and segfaults. Stupid thinko, but have I really not tested mkroot in a month? Sigh.

Oddly enough, I'd already hit this and fixed it up in the shell work I did yesterday, but getting that to a good stopping point so I can check it in is tricksy. (It started with an attempt to add the read builtin and there's a lot of half-finished debris lying around the tree.)

Went to Walgreens early this morning and bought earplugs. Much less painful work experience. (I am not a dog person. Never developed the skillset. If I lock adverb in the bedroom when he's not alone in the apartment, he claws at the door endlessly and will damage it. If I let him out into the center room (combination living room and kitchen), he barks at the front door for maybe ninety seconds every time somebody else in the apartment complex walks through the hall. Fade sits on the bed with her laptop and closes the door, and Adverb thinks that's the correct way to be home and keeps trying to lure me there, but I'm used to a table and chair and this room has better lighting.)

If you're wondering how my day is going, my attempt to add a shell "read" builtin has diverged into reverse engineering my ${variable} expansion code to figure out what all the corner cases are which led to reading the relevant part of the bash man page which led to me restarting the bash man page from the beginning which led to redoing sh_main() flag parsing and adding tests for sh -cs "arg" thingy vs sh -c "arg" thingy which led to me changing the logic so -c "arg" aren't an arg.c colon attachment (because they aren't in bash: it reinterprets the first argument as a command instead of a shell script but sh -c -s "echo hello" prints "hello" instead of trying to run -s and yes I need a test for it) which circled back to me trying to get all the existing tests to run under ASAN which means tracking down why sh -c '<<0;echo hello' was faulting which is because TT.ff->pl = xrealloc(TT.ff->pl) sometimes ALSO needs to update TT.ff>pl->end and now I'm trying to work out when that's true. (The only realloc of an existing pipeline segment is when attaching HERE documents to one, which expands the arg[] array at the end, but I need to update ALL the pointers.) And then once I added a loop to check all the pl->end in the pipeline and update it if necessary (which SHOULD happen before function bodies get moved so it should all be in the one doubly linked list), that revealed a double free error I need to track down.

None of this was what I planned to do next, but with Android in feature freeze it seems like a good time to make a dive back into shell stuff...

Adverb has been barking continuously throughout this. Fade's dog is unhappy when Fade isn't here, and expresses it when he's not alone. (If he barks at the front door long enough, clearly I will bring Fade back. It's worked every day so far, after enough hours. I have headphones, but need earplugs. I have escaped the clingiest cat to visit a neurotic dog.)

Gentoo's "make tests" is failing on du because overlayfs lies. My first instinct was to mount a tmpfs when run as root, ala if [ $(id -u) -eq ]; then mount -t tmpfs tmpfs .; cd "$PWD"; umount -l .; fi (the lazy unmount means it's still there on the current directory while we're in it, but automatically unmounts as soon as we cd out or exit the process).

Unfortunately, the results from tmpfs are very different from the ext4 I developed it on: mkdir allocates a 4k block up front on ext4 but in tmpfs directories are always size zero (because the dentry cache doesn't take up space in the page cache). And I can't convert the tests to what tmpfs produces unless I'm going to _require_ it to run under tmpfs, which you can't do as a normal user. I think I need to do:

  dd if=/dev/zero of=ext2.img bs=1M count=1 status=none
  mke2fs -b4k ext2.img
  mount ext2.img .
  rm ext2.img
  cd "$PWD"
  umount -l .

Which should get me a filesystem that behaves like the one I'm developing on. (How does "dd" manage to get "unix" so wrong? Success is silent so your pipeline isn't full of trash, having to status=none to do that... I'm blaming IBM, they got ebcdic in there somehow. I'd use "truncate -s" but you can't loopback mount a sparse file...)

Attempting to close tabs: the gentoo locale thing should be fixable by having it try C.UTF-8 (which macos hasn't got) before en_US.UTF-8 (which gentoo hasn't got). My readingn of "man 7 locale" says it should try C.utf8 in its search path (feed it the "official" name and it tries four different variants: upper and lowercase, with and without dash)... gentoo still didn't work. I tried to run it under strace to see why, but "emerge strace" doesn't work on last Sunday's LiveCD because /etc/portage/make.profile is a broken symlink. Emailed a "huh?" at Patrick Lauer...

Oh goddess, whatever Horrible Gnome Thing gentoo's livecd is using as its' terminal (or is it a Horrible KDE Thing?) is FLASHING the broken symlink at me. Causing KVM to gratuitously eat CPU doing perpetual screen updates just to the display can cause ADDITIONAL EYESTRAIN. That manages to be counterproductive on multiple levels. (And I haven't dug into figuring out how to make the background be actually black instead of dark grey, because they decided "less contrast, that'll help".) Cleared the terminal and my CPU usage graph no longer looks like a heart monitor.

February 10, 2023

Onna plane. Heading to Minneapolis, visiting Fade until the end of the month. (Flying back on the 28th, which is as far as February goes this year.)

Haven't blogged for the past few days, felt under the weather ever since the ice storm. (It _really_ threw off my sleep schedule.) Made a few notes about "huh, I should blog about that" and then didn't. (Sigh, I should backfill but mostly the things I thought about blogging were when I wasn't in front of the computer, so said notes would be in Austin and I'm onna plane.)

What did I do: aggroed the bash maintainer into a coreutils thread. (Still subscribed because cut -DF still hasn't been merged or rejected.) The arch/sh maintainership transfer is still up in the air. Started researching mkisofs. Did NOT post my kernel patch stack to lkml yet.

On the toysh "read" builtin front, bash's behavior is subtle: read -p hello > /dev/null doesn't work because the prompt is output to stderr not stdout (justs like the $ prompts). If I go read "" it exits with an error immediately (because "" is not a variable name it can assign to), but if I read potato "" it reads a line of data, splits it, assigns the first part to potato, and THEN exits with the error. I don't understand why it only checks the FIRST value for validity before reading input? (Why check it at all before reading if you're not going to check the rest...)

$ read -p % ""
bash: read: `': not a valid identifier
$ read -p % potato ""
%one two three
bash: read: `': not a valid identifier
$ echo $potato
one
$ read -p % potato ""; echo $potato
%blat
bash: read: `': not a valid identifier
blat
$

First time it doesn't even output the prompt, the third read shows it's not a syntax error (just a normal error exit). So that's good to know. I should add tests...

And then of course, after all that bashing my head against input granularity, sitting down to write "read" I'm hitting OUTPUT granularity. Namely: you can list multiple variable names on the read command line and it does IFS splitting to put a word in each argument the way it does for $1 $2 $3 etc for commands and functions... but if there are fewer variables than arguments it STOPS splitting early, and puts the rest of the string into the last argument, not having consumed the remainder's $IFS characters. Meaning read A B <<< "a b c" will preserve a run of multiple spaces or whatever space/tab/space combo was between "b c" when assigning to $B. Which is NOT the "split and glue back together with the first $IFS charachter" logic of "$*" nor the "glue back together with specifically space regardless of what IFS says" behavior I implemented SEMI_IFS for in "eval" and "case"...)

The problem is, my function that does all this work is expand_arg_nobrace() which is already taking six arguments, the last two of which are usually zero. I'm reluctant to add a third "usually zero" argument, especially since the last one that's currently there is "long *measure" which seems like it could be repurposed, but what it currently does is "set it to a character to search for a bit like $IFS but this one's a hard stop where you write the offset at which you found this character into *measure and return early", which is used to reliably find the semicolons in ((math;moremath;evenmoremath)) regardless of quoting and ${thingy#$((blah))} nesting levels. Totally different from "set NO_SPLIT in flags after argument 3".

(I also hate $IFS as a concept, and spent months wrapping my head around the details of what does and doesn't become a separate argument with "" and ""$EMPTY and """$*" when there are no arguments, and how x() { echo $#;}; x """" should print 1 not 2... and looking back through this code I remember that there ARE a bunch of special cases but not WHAT they all were, which is why I made so many tests/sh.test cases for it, and I dowanna touch this forest of nested horror that laboriously jenga-style made them all work, but I have to find exactly the right place to drop in a state change with no state inappropriately crossing the change point... and I dowanna.)

Setting *measure to a negative number is uncomfortably magic.

Adding an IFS flag to change the meaning of *measure would let me avoid changing all the callers to add another zero, but it has a naming problem: the common prefix of almost all the existing flags is NO_ as in NO_SPLIT and NO_IFS to disable something expand_arg() would otherwise be doing. (Which isn't great either, but EXPAND_NO_SPLIT is too long when you're or-ing together five of them). I already violated that with SEMI_IFS and dowanna do so again or I've just got a bunch of random #defines floating around the code.

I made a quick stab at adding an expand_arg_nobrace() wrapper calling expand_arg_nobrace_raw(). After all the original API is expand_arg() which handles ab{c,d} processing and then passes on to expand_arg_nobrace(). But two of the calls ending in double zeroes are recursive calls within expand_arg_nobrace() itelf, and I'd need to provide a function prototype (with seven complex arguments to keep in sync if anything changes) to let those two call each other, which is exactly the kind of nonsense I'm trying to avoid with the ever-widening API on this sucker as I find new corner cases.

February 9, 2023

February 7, 2023