Still walking to UT overnight for the exercise, but while the porch area I was using is "under construction" (minus the activity part) I've been using a nice couch and some hallway student desks on the second floor of Jester Center. I keep trying to stay until the Wendy's opens but if they're not open by 9am it's not happening.

It's sort of random what's open at 3am. Sometimes the "language learning lab" classrooms are locked, sometimes they're open. I miss Metro, the 24 hour coffee shop Guadalupe used to have, before the Cult of Scientology pushed all other tenants out of the building. (Back before the 2008 Mortgage Crisis, there was a 24 hour video arcade next door to a 24 hour coffee shop at the University. My habit of walking to UT at night started in that context. Unfortunately, that environment was created when the democrats were in charge of Texas. But then George H.W. Bush bought governorships for his sons in two of the most populous southern states (Jeb in Florida and Dubyah in Texas), and the launch gantry that put Dubyah in the whitehouse fell on the state an smashed our infrastructure. The number of homeless people ballooned and the number of late-night places they could be indoors dwindled, and the pandemic finished off most of what was left. Now even the 24 hour HEB I bought my house near is only open 6am to 11pm. Austin's not exactly a dying city, but it's definitely not what it was.

Capitalism is the Boomers' religion, and the Boomers are not dying fast enough.

An observation that should probably go in the blog:

This bug I just fixed wasn't a Mac OS kernel issue, it was ifndef O_PATH define O_PATH linuxvalue which on MacOS was already used by O_SYMLINK.

The previous bug wasn't misbehaving glibc thread plumbing, it was a signal handler inappropriately inherited by a child process they tried to use global data the child process had repurposed.

Now I'm glowering at the old bug that only reproduces in a Fedora VM and is happening inside the glibc FILE object. Yes, that's taking a bullet, but from where?

It's hard to feel productive when you're taking multiple days to come up with a one-line fix. (Sure I wrote a thousand lines of scaffolding along the way, but you tear that down again when you have your answer.)

In professional programming, adding a thousand lines of code to avoid having to come up with that one line fix makes everybody's numbers look good. In open source we spend a lot of time looking like we didn't do anything. Oh well...

Back to chipping away at Mt. Backlog, resuming release polishing, I need to do a trawl for dropped plot threads...

Why did git spontaneously decide to pack the "clean" repository I just created (via cp -a then pulling another local repo into it)? And how did it fire off a BACKGROUND process to do this? I had a "git log" blocked on "less" in there, but the PROCESS that's running says it's git pack on the command line. I did not agree to any "peg all 4 processors while my laptop is on battery" prompt? Nothing else I did ate a command prompt? Why would it launch a background job without telling me it was doing so?

There's a reason I have a xfce's "CPU graph" thingy in the toolbar. Usually a runaway battery drain is chrome and I need to kill a few renderer instances, or just "pkill -f renderer". But no, this time top showed git doing something I did NOT ask it to...

My laptop shouldn't be doing weird asynchronous stuff it didn't get from the keyboard because I do not have NET here: my phone tether does not work from the HEB deli's tables. The radio reception around Hancock Center is ABSOLUTELY NUTS, with spots of "my bluetooth headphones need the phone to be 3 inches from the left ear or it goes silent" all the way at the sidewalk on the far corner of Red River and 41st street. Given that I've reported "hey, there's a smell of gas coming from the drains at Red River and Keith Lane (just north of 32nd street) via the city's 311 app more than once and they ignored me, I've never really looked into trying to track down WHY all the unlicensed radio bandwidth in Hancock is supersaturated. (Maybe a really buzzy power transformer somewhere? No idea.)

I used to think the entire parking lot between Sears and I-35 was "bluetooth dead" because of some strange obsolete security system Sears had, but it got worse instead of better after they died. So wifi being unable to associate between my phone and laptop with the two physically in contact with each other in the HEB deli area of the same "used to be an open air shopping mall but then half of it was bulldozed for parking lots when Austin increased its mandatory parking requirements" (no really, that's literally what happened) isn't really that surprising? The wifi list doesn't show _any_ access points here, which is pretty much unique in Austin. The phone gets good reception and if I really need net I can tether with a USB-C cord, but doing so drains my laptop battery even faster because it charges the phone. I get maybe 45 minutes tops here if I do that.

Anyway, dear git: do not do things I did not ask you to do on a unix system. Stoppit. Rolling security upgrades together with this kind of behavioral change is rage inducing.

Luckily killing the process doesn't appear to have damaged the repo. I'm just using the "clean" repo to collect some of my outstanding kernel patches so I can put them in a git branch or something. The bc removal one is waaaaay too big to apply via sed, so I might as well publish a branch somewhere people can pull from. (The bc removal patch would let me yank "bc" from the airlock list: nothing else anywhere uses it, as far as I can tell. Another patch lets me yank "gcc" because then the kernel builds with cc and doesn't have to be explicitly TOLD if it points to gcc or llvm, it just works.)

Ooh, one of the patches I was applying with sed in mkroot.sh went upstream! Because somebody ELSE noticed the bug and pushed it, apparently, but eh. I'll take the fix. An advantage of arm nommu getting serious: they're fixing up some general nommu issues existing embedded developers haven't been able to get the kernel clique to care about.

Nommu systems are to mmu systems as bacterial cells are to mammal cells: add 'em all up and they outweigh the higher lifeforms by a LOT. Yes, the number of phones in use is converging on the population of the planet (which just passed 8 billion, phones had passed 3 billion last I checked but it's been a while), but there are an order of magnitude more nommu systems out there in cars and microwaves and air conditioners and water pumps and elevators and transformers and headphones and fitbits and so on. Linux is overkill for most of them, but a small slice of a moon-sized cheese is still quite a few seats, and I've already heard from people using toybox on qnx. Plus the "linux on nommu" skillset is a good gateway drug to "programming on posix-ish RTOS du jour" and eventually clawing your way down to DSP programming.

Yes, those are likely to stay in C when you're not doing raw assembly. While it's possible to do realtime garbage collection I've never seen anyone actually implement it (or even heard anyone other than me propose it, despite having come up with that design in 1996). Note: if that RT garbage collector is traversing down the stack and the processor running that function returns "above" it, the garbage collector can immediately terminate the stack traversal because all the local variables it hasn't examined yet just got discarded, so anything they point to doesn't matter anymore. So when the realtime garbage collector is traversing a stack it needs to check the running thread's stack pointer before accessing each variable, and stop immediately if it's now past it. If the GC and running function are on two different processors running at the same speed and the garbage collector can't be interrupted, then the stack contents from the returned function won't have time to change between the stack check and the fetch. (Plus if these are object table indexes you can sanity/bounds check the object table entry anyway. Worse case scenario you false positive something as still used and don't free it this pass. Of course I have no idea how rust or go handle this, I did the design against a now very stale version of Java.)

Yes, this implies SMP with the garbage collector running on a different processor, but doing that doesn't even require extra circuitry. Intel was so proud of "hyper-threading" and Qualcomm did the Hexagon but the underlying technique of round-robining mutiple register profiles down the instruction pipeline is called a barrel processor and dates back to the 1960s. Jeff and I worked on a Barrel version of J-core that would fit in an ICE-40 back before the pandemic. (We're already using 512 byte ram blocks to hold the register file in that FPGA family. The ISA has 16 4-byte registers, although it's got a bunch of status registers and the microcode uses a few more internal temporary registers so the grand total is more like 24 registers. Since "32x4=128 bytes" we've got space for 4 register profiles in that 512 byte ram, and can just use the thread ID as the high bits of the address. (There's a few more details to make the rest of the processor happy with this, mostly having to do with registers that for implementation reasons don't live in this RAM, such as the instruction pointer and the multiplier's internal registers, but on the whole it was pretty manageable. All got shelved for the sky130 ASIC work, of course...)

Because it was happening in a signal handler. Of course.

So timeout.c was deferring ALMOST all its signal handler setup until after calling xpopen_setup() which spawns the child process, but it has to set up the SIGCHLD handler _before_ spawning the child or the parent may not get scheduled again before the child exits. (While SIGDFL allows zombies to accumulate, it won't send another SIGCHLD and I hit some race where the process had already exited and it sat there for the full timeout waiting for nothing to happen, I think it was a timeout -i thing where stdin/out/err didn't close until the zombie got reaped so there was no notification it had exited if you missed the SIGCHLD? Setting up the handler before the fork() fixed it.)

The problem is, if the middle process doesn't REMOVE that SIGCHLD handler, but we don't exec() either, then the handler is still there. But in the recursion case the shared global setjump buffer (TT.sj) the handler tries to siglongjmp() to gets memset by the call to toy_exec() for the new command, which SHOULD be fine because toy_exec() never returns (a bit of stack has been consumed but we check how much is used before recursing again). So this leftover signal handler does siglongjmp(NULL, 1) from signal context (with its own dedicated stack these days), and ASAN's attempt to provide a backtrace was not up for that. And if it couldn't provide a backtrace, it wouldn't indentify the function the instruction pointer pointed into at all, just a post-ASLR hex address. What I SHOULD have done is copied /proc/self/maps to a temp file at the start of main() and then manually translated the faulting instruction pointer into the executable's readelf ranges, but hindsight... (Often easier to say "this is how I should havve found it" AFTER you know where it is.)

Toybox (like busybox) is doing the recursion because it's faster than exec(), doesn't depend on $PATH or /proc/self/exe to be populated (so you can run commands in a completely bare or untrusted chroot, which comes up both on embedded systems and when doing system recovery/forensics), and mostly reuses the shell's builtin command plumbing: you can't have "cd" or "export" run in a child process, it won't work on a conceptual level. The shell's variant of this recursion copies the TT contents into a temporary buffer and puts them back again afterwards, while the simpler version in main() just goes "we're never returning" and wipes the old state. Both had the same problem here because _while_ the child is running, timeout's TT.sj field did not have the right function pointer in it, so calling that signal handler at ALL was bad juju.

Ok, the problem happens when a command calls xpopen() with xfork() enabled (so the NON-vfork path, which I admit has gotten less testing because it's theoretically the easy one: the fork+recursion case is the historical baseline config), and then THAT command calls xopen() and the child does an exec and exits. The middle process of the three segfaults in the libc waitpid() call receiving a SIGCHLD and then SIGSEGV shortly afterwards. And depending on the timing, it doesn't even always call the funky rt_sigwhatsis so the glibc thread stuff being stupid may be a red herring?

The hard part is that reproducing this requires three interacting processes, and the tools I have to provide visibility into this aren't good at it. It's not happening IN a syscall so strace is about as helpful as a foghorn for navigation (seriously insufficient granularity, at times actively misleading). ASAN just bailed right up front (NO BACKTRACE at ALL). No standalone sample programs I've written to try to reproduce the failure have experienced any failures. So it's back to the slow, methodical dprintf(2, "%d thingy", getpid()) spray-down, chopping stuff out with if (0) and so on as I go. There are enough moving parts I've debugged down the wrong branch more than once. My mental house of cards keeps collapsing and needing a retry.

Speaking of being frustrated by strace: the fact that dprintf is treating newlines as a magic character to be chopped out and handled as a separate syscall is EXTRA ANNOYING here, because if I add a "\n" to the start of the dprintf() line so it's at the left edge rather than appended to syscall trace line du jour, that turns into multiple syscalls and then the actual output line is STILL OFF TO THE RIGHT SOMEWHERE. There is not an obvious "stop doing that" option in glibc. I suppose I could do it myself with xmprintf(), write(), and free, but that kinda seems overkill to emit debug lines? (Yes, I'm aware of ltrace. No, it's not an improvement over sticking printfs in things, unless scrolling past hundreds of consecutive lines of __ctype_b_loc() is your idea of fun...)

I really don't WANT to be tracking this down, I want to work on other things, but when a bug presents itself you gotta chase it NOW while it's reproducible. (Yes I should go back and do this for the Fedora one, but it's still hard to consider Fedora real. It was dispiriting enough when I thought this failure was happening in libc, which turned out to just be an artifact of strace reporting granularity. The Fedora failure IS happening in libc. Yeah I'm doing something to trigger it, but I have to stomp through libc code to track down why and... Lemme debug THIS one first. And maybe get a release out.)

Ooh, Hades II dropped. Looks awesome. (I am not putting the dots over Melinoe's name. I need a clip of Wesley and Buttercup saying "Supergiant never makes sequels!" "Nonsense, you're only saying that because they never have.") Trying to find more reaction videos of women gamers being excited about the awesome female protagonist, to counterbalance the endless stream of neckbeards posting their reaction. I agree that the trailer 100% needs to be expanded into a full Netflix animated series (and I love her shirt).

I debugged my way through the shell and out the other side. It's not anything toysh is doing, toybox timeout -i 1 find . -name README -exec /bin/echo '{}' ';' does it too. And it happens without the -i in timeout, so it's not some sort of pipe/select weirdness with stdout and an invisible SIGPIPE or something. (Which was at least a recent-ish change that might have perturbed something.)

I made a "testy" command that just does xrun("/bin/echo") and it happens with that instead of find. Adjusted it to xrun(toys.optargs) I.E. call xpopen_both() on the command line arguments with no stdin/stdout redirection, and then xpclose() on the result to wait and get the return code... and the result reproduces the exit code 139 with "toybox timeout testy testy /bin/echo".

I am out of spoons for the day.

I feel I should make a "how I debug stuff" video, but it's not exactly best practice. It's brute force insertion of dprintf(2, "nospoon"); and dprintf(2, "credenza=%s\n", TT.ff->blk->pout); at various places until I've found the point where what the code is doing diverges from my expectations. The names are just random unique indicators that "the code got here", often with a variable value attached. In the case of toysh, each line usually starts with a %d displaying getpid() because it vfork()s and backgrounds a LOT of stuff in the normal course of operations. (Between parentheses and ampersand and pipelines and <() and so on, there's a whole lotta child processing going on.)

Ok:

strace -f ./toybox timeout -i 1 sh -c 'for i in .25 .50 2; do sleep $i; echo hello; done'

The segfault is coming in as a signal to waitpid() in lib/xwrap.c:xwaitpid(). There's a SIGCHLD immediately followed by SIGSEGV before waitpid() can actually return (or at least the next printf() doesn't get called). It DOESN'T happen if the sh -c 'blah' line is called by itself, it needs to be a child of timeout (dunno why). It DOES happen if .50 and 2 in the for loop are removed. It DOESN'T happen if the .25 is replaced with 1. It DOES happen with .9 there. It DOES happen if sleep is replaced with /bin/echo. Ah, and then 1 makes it happen, right the timeout was terminating at 1 second preventing the error codepath, of course. Ok, the entire sh -c payload can become "/bin/echo 1" and it happens. Also with /bin/true.

So it happens with strace -f ./toybox timeout -i 1 sh -c '/bin/true' but it DOESN'T happen with ./toybox timeout -i 1 /bin/true or with ./toybox sh -c /bin/true.

The WEIRD part is strace says:

[pid 26878] +++ exited with 0 +++ [pid 26872] <... wait4 resumed< [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 26878
[pid 26872] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=26878, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 26872] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 26872] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---

The child exits normally, the glibc waitpid() call calls rt_sigprocmask(SIG_SETMASK) with NULL as one of the arguments (???) and then segfaults. It looks like a sigprocmask call made internally by glibc is segfaulting? Why is waitpid() NOT being a raw system call wrapper here? And why is it calling wait4() instead of waitpid() when I specifically asked for waitpid?

Ok, the glibc source (from 2019, newest I have lying around) has sysdeps/unix/sysv/linux/waitpid.c call a SYSCALL_CANCEL() wrapper, which is unnecessarily defined in sysdeps/unix/sysdep.h as "if (SINGLE_THREAD_P) { do the sane thing } else { wrap the call in two LIBC_CANCEL_STUFF() things that are probably inserting the broken rt_sigprocmask thing}". And SINGLE_THREAD_P is a gross hack in sysdeps/unix/sysv/linux/sysdep-cancel.h which is just a nightmare #ifdef staircase of __glibc_likely(__pthread_multiple_threads == 0) or __glibc_likely (THREAD_GETMEM (THREAD_SELF, header.multiple_threads) == 0) or like SIX OTHERS JUST IN THIS FILE ALONE. (There's a bunch of other headers it could be defined in too).

So for some reason, when I vfork() glibc decides I'm using threads (I'm not) and uses uninitialized variables in its thread cancellation logic. Did this show up in an apt-get upgrade, or did I start doing something slightly different to trigger it, or...? No idea.

Urgh, I'm not even sure how to report this. I wonder if it's related to that Fedora libc issue I got distracted from and never finished tracking down?

Sigh. The next step is to chop out unnecessary code until I have a minimal reproduction sequence as a standalone program. Probably even more tedious than it sounds.

The 6.1-rc8 kernel came out, so I'm trying to run the qemu smoketests. Rebuilt the kernel, made the standard mistake of having an exising ".config" in the directory that was defconfig without sh and route (so mkroot.sh didn't generate its own adding them, I REALLY need to get them out of pending) and had to rebuild all the root filesystems.

Fun trick I should probably put in the FAQ: tar czvf root-kernels.tgz root/*/{linux*,run-qemu.sh} grabs all the files generated by adding LINUX=/path to the mkroot build, so if you "make distclean" you can extract that into root/ instead of saying LINUX= again and avoid 95% of the build time. (This is why the cpio.gz file is external and passed to the kernel as an initrd image using qemu's built-in bootloader. If it was static, you'd have to rebuild the kernel image, and mkroot deletes successfully built cp -s source snapshots. It keeps the unsuccessful ones around in root/build/$ARCH-tmp for debugging, but only ever does a "build all" when creating packages, not incremental builds.)

Fiddled with httpd.c just enough to shut up a warning about a variable that's set but never used: yes, it's the text after the ? in a URL which is for CGI stuff. I "used" it to set the environment variable that CGI would expect it in (which nothing then uses yet but the compiler can't tell so stays quiet about), which led me down the path of looking up cgi support, which I last did over a decade ago and could use a refresher on. Wikipedia[citation needed] gives a useless perl expample. (What does having some random perl library parse everything for you accomplish about explaining the format?) Once upon a time I wrote python code calculating login credentials and saving them in cookies, it was just a very long time ago and I really don't want to try to set up a web sever on loopback so I can see what CGI stuff it's sending my test programs.

Sigh. Looked at busybox to see what their httpd exports, and that has a cgi example program (networking/httpd_indexcgi.c) which starts with 90 lines of completely unnecessary CSS stylesheet. Um, no. It reads exactly one environment variable ("REQUEST_URI") and then chops it at "?" and ignores everything after that. So it's not actually demonstrating anything I'm interested in. Urgh, dowanna go back and look at the busybox httpd.c, that was maintained by Vladimir the crazy Russian who communicated through a PC translation program (back in the days before even google translate) and got very angry if anyone ever touched "his" code. That's not just license reluctance, that's bad memories...

Fixed a random new github bug report, which had a little collateral damage in option strings and while I was fixing those up I taught "sleep" to handle multiple arguments (because it wasn't before), and now... an unelated timeout test is exiting with 139, subtract 128 means signal 11 which is a segfault. Run it ASAN=1 and it's dereferencing a NULL pointer... with no stack dump. Why is it not giving me ANY stack dump here? Sigh, that's the only useful thing ASAN does. (The glibc built-in backtrace() stuff is sadly unreliable as well.)

Tracked it down to toysh, which is both surprising and unsurprising. It's surprising that sh is getting called, but I built a toybox with sh and recursion enabled for mkroot and the test is timeout -i 1 sh -c 'for i in .25 .50 2; do sleep $i; echo hello; done' which will run that through toysh in this instance. It doens't look related to the recent change to sleep (it's not making it that far), it looks related to having sh in toybox. So now I'm going down THAT rathole to see what this bug is. ASAN still won't give me a stack dump. (It's not even trying to.) And so I insert All The Printf()s as is the tradition of my people. Hmmm... it's possible this is happening in vfork() context, which might confuse it.)

The 'weird blue flash in the corner of my left eye' which keeps making me think I'm having a migrane-like visual artifact is, in fact, the blinking blue LED on my left headphone reflecting off the corner of my glasses. I keep getting caught by that one and panicing. (Your retina can't sense pain, it can only sense light, so some flashes are the back of your eyeballs going "ow" by emitting the signal they've got. It's something I generally try to avoid; having it happen repeatedly out of the blue makes me nervous. That one gets me about once a month...)

Checked in some FreeBSD fixes, and it more or less builds clean now on my FreeBSD 13 VM. I should write up a FAQ entry about test environments, but I really want to get the tests running under mkroot, which means cycling back to shell work.

Hmmm... possibly I should teach "make defconfig" to detect freebsd and darwin and android and use the appropriate config file? Eh, let's not overthink it just yet.

Sigh. I want to do "make test_command" on freebsd but scripts/single.sh is calling "make" instead of "gmake". The reason I haven't got scripts/portability.sh defining $MAKE to make/gmake the way it's doing for $SED and sed/gsed is scripts/*.sh should not call back into "make" both because the make wrapper is theoretically cosmetic and because the .config file produced by kconfig is consumed read-only by the rest of the plumbing (to avoid "derived work" questions with the licensing).

But scripts/single.sh is the exception to that because it has to put together a new .config for the single build. (These are not the binaries we usually ship, they're only ever really produced for testing. But they're quite convenient for testing.)

It potentially calls make twice: 1) to produce a defconfig if we haven't got one (wrong for freebsd/mac/android anyway, but it'll use an existing .config if it's there), 2) to create an allnoconfig to harvest CONFIG_TOYBOX symbols from for basic system layout.

Hmmm... the allnoconfig step is basically because lots more stuff used to be probed by scripts/genconfig.sh, and it ONLY wanted probed symbols... but when TOYBOX_FLOAT is selectable in menuconfig how do you tell singlemake to do or not do that? To be honest stuff like TOYBOX_LSM_* and TOYBOX_FLOAT might be probeable? (Do you ever NOT want it when your build environment has it?) Hmmm... The "use libcrypto" and "use libz" questions are basically the same question: should we use toybox builtins instead of library versions even when the libraries are available? (Gotta be able to test those codepaths. We can probe the libraries with __has_include(blah) to avoid build breaks.) CONFIG_HELP is a command, it should be ih "other" (the --help and --version support stuff is here, and... do I ever want to turn it off? It saves a little space, but I'm really not micromanaging that? Hmmm...)

Did I mention that the C standards bureaucracy's "double underscore is reserved for implementations, oh every new extension starts with a double underscore... and now we're standardizing widely used double-underscore symbols" is FSCKING STUPID? Gee, who would have predicted that __has_include() would be forced into a stupid name, let's hear it for the three underscore namespace (just wait).

If I trust defconfig to exist (or barf it if doesn't?) then what is the make allnoconfig call accomplishing? I want to say "dependency resolution", but can I harvest dependencies from the config stuff via sed? sed -n 's/^config //;T else;h;:else;/^[ \t]*depends on /{H;g;s/\n//;p;}' toys/*/*.c sort of does it, although instead of *.c that should really be generated/Config.probed generated/Config.in Config.in since I've already GOT plumbing harvesting that from the *.c files... (Sigh, I need to split the header generation plumbing from the build invocation at the end, but the header generation collects the file lists I need to pass to the build. I suppose I could write it out under generated/ but having persistent stuff I can just collect cached in the filesystem is Bad Programming 101: "The two hard problems in computer science are naming things, cache invalidation, and off by one errors.")

Anyway, is it REALLY doing dependency resolution? The problematic case is "command was not switched on and I just added it", but starting from allnoconfig it has to already be handling that? I need to rewrite kconfig from scratch, but I'm trying to CLOSE cans of worms at the moment...

The new table layout at the porch outside UT's Norman Ackerman building makes the raccoons a problem. There's a family of them that patrols the trash cans in sequence each night, and the remaining table positions are basically exactly that circuit. They left me alone in the corner near the outlet (sometimes they'd approach ~10 feet away to see if I had food because these are university animals after all, but they didn't want to get too close). But now every table is either right next to a trash can or between two of them, and they go under the bench I'm sitting on, and sometimes put their paws up on the end of it to see if I brought table snacks. They're adorable, and more well-behaved than the university squirrels in the daytime, but I just HAD a series of rabies injections thanks. Dowanna nother.

Why is there no bswap_16() and friends in posix? Those are the linux names for this. Freebsd has bswap16() with no underscore, and MacOS has OSSwapInt16() because of course it does. I can mask and shift this stuff myself but I'd hoped for an assembly optimized version, especially for the 64 bit ones?

Then again fast fourier transforms need to perform operations in bit-reversed order (it's used as an array index in the algorithm) and nobody seems to have a bit reverse machine instruction? Jeff and I pondered adding one to J-core, but what we really need is for the existing postincrement operation to do a bit reversed increment with range. And the REAL problem there is it's not a 32 bit reverse (unless you're fourier transforming a buffer 4 billion entries long), so you'd need a "bit width" indicator. If you have registers to spare you can increment a normal counter in one register, copy it to a second register, bit reverse it, and shift it by a constant kept in a third register. Only three extra clock cycles, but this is a tight inner loop...

So what DOES build under freebsd? The freebsd_miniconfig isn't really helpful here, lemme try to trim down defconfig myself:

gmake distclean defconfig && gsed -i -E 's/CONFIG_(DMESG|MOUNT|PASSWD|SU|UMOUNT|IFCONFIG|PING|RFKILL|SNTP|TUNCTL|NETSTAT|BLKID|FSTYPE|BLKDISCARD|BLOCKDEV|EJECT|FREE|FSFREEZE|FREERAMDISK|GPIO.*|HWCLOCK|I2C.*|INOTIFYD|INSMOD|IONICE|IORENICE|LOGIN|LOSETUP|LSATTR|CHATTR|MIX|NBD_CLIENT|ONEIT|NSENTER|UNSHARE|OPENVT|CHVT|DEALLOCVT|PARTPROBE|PIVOT_ROOT|READAHEAD|REBOOT|HALT|POWEROFF|RMMOD|SETFATTR|RTCWAKE|SWITCH_ROOT|SWAPON|SWAPOFF|TASKSET|NPROC|UCLAMPSET|UPTIME|VCONFIG|WATCHDOG|CP|MV|INSTALL|GETCONF|PS|TOP|IOTOP|PGREP|PKILL|TAR|ULIMIT|UNAME|ARCH|LINUX32)=y/# CONFIG_\1 is not set/' .config && gmake toybox

Hint: the build dies with a filename it had a problem with (usually failure to #include a header or a missing #define) and while the name of each command file should be the name of the first command in it (you have to #define TT yourself when it isn't, see readlink.c and i2ctools.c and gpiod.c for examples of that), there can be multiple commands in the same file and you have to switch them ALL off to stop it compiling the file. Do grep TOY toys/*/filename.c to get the list of commands the file exports, and once you've seen it you can just throw in wildcards for some of them like I did. :)

It built 149 commands. Not bad. "VERBOSE=allnopass gmake tests" had a failure in chmod +t (chmod 'file' to 1644: Inappropriate file type or format), "file utf-8", "file /dev/zero", "fmt utf8", "split bytes", "split reassembly", and "tee -".

Main lesson from the exercise: maintaining a list of things to switch on is still easier than maintaining a list of things to switch off. (They're both huge, but this way it's quarrantined into one file rather than polluting each config snippet.)

Secondary lesson: I think macos is supporting more stuff than FreeBSD? I remember how FreeBSD hasn't got a /proc so their ps implementation is a nightmare of shared libraries performing magic undocumented ioctls and such under the covers (it's on the mailing list somewhere), but does cp.c not work already on mac? Or tar. Or getconf.

Third lesson: some stuff should probably be fixed. The cp/mv/install and tar problems seem to just be missing xattr stuff. (Are those there on MacOS?) The lib/passwd.c rewrite would probably fix passwd, su, and login. The sntp break is missing adjtime. I thought getconf worked in mac? Doesn't look like uname is too hard to fix. Both blkid and fstype are grabbing filesystem ID constants out of a header, I could probably transplant them but how useful is it to BSD?

Still fiddling with the FreeBSD-13 snapshot.

You'd think you could point git's http:// at an existing repository and have it work, meaning toybox httpd on 127.0.0.1 plus kvm's 10.0.2.2 alias would let me update the vm repo from the host as I checked fixes in without pushing to an external site. And it ALMOST does (if you point it at the .git directory under the repository)... But it's unreliable, and keeps complaining it can't fetch stale pack files. If you do a "git gc" on the repo you're trying to fetch, it'll create a new one and then the error message from subsequent fetches will be failure to fetch the OLD pack file that just got deleted by the gc. (Some sort of stale cacheing? Why does the local one not get confused looking at the EXACT SAME FILES?)

The workaround is to basically set up a cloneable bare repository with "git clone --bare repo baredir; cd baredir; git update-server-info". Which gives an https cloneable snapshot. There's no earthly reason you should NEED to do this, but git is deeply user-hostile in the unix tradition of "ed", so you've gotta know the magic incantations to do anything...

Speaking of vi(m) being stupid: when I edit a file some other vim instance is editing, it tells me the PID of the other process but doesn't give me an obvious "kill the other one" option. I have to exit, type "kill" with the other ID, and then cursor up and re-run the vi.

And no, I will never type "vim" instead of "vi". I will "ln -sf vimrc /etc/vim/vimrc.tiny" on each new install until that stops working, and then start using a different vi implementation. 20 years ago the vi in Red Hat let me cursor around in insert mode just fine, distros that intentionally insert behavioral regressions get adjusted with a chainsaw.

Elliott posted more information about the MacOS realpath failure and... it's a kernel bug? He added some "ls -Rl" calls to show that "dos -> one/two" and "three" in one/two is a world readable directory. The failing call is trying to openat("three") and the only two openat calls in xwrap.c are doing a hardwired ".." and then the one on line 645 where it does openat(dirfd, new->str, O_PATH). The complaint is "not a directory" which lines up with the O_PATH except three IS a directory. It's just about possible that dirfd = 4 points at something other than one/two (since it was opened via symlink), but this exact same code works fine on Linux?

It really looks like I hit a kernel bug here? Not sure where to go from this. If I had direct console access I could more solidly CONFIRM it's a kernel bug, but I'm not sure how that helps? (Maybe I could work around it by inserting an open(".") or something? Probably not...)

Sigh, I should confirm this DOESN'T happen on BSD. Where did I leave my BSD virtual build environment... Nope, Freebsd-11 is way stale. Time to grab a new version...

Explained the freebsd-13 build setup on the list. I should write up a FAQ entry about this, but first... I haven't regression tested BSD in a while, and also need to figure out what similiaries it does and doesn't have with MacOS...

Since cancelling toolchain work with Jeff on the 23rd, he hasn't been available to do any more since. I told him on the 25th I could do some work "tomorrow or sunday" and he replied "ok, lmk", and that's the last I heard from him on signal. I wrote at him on the 26th, 28th, 29th, and today: no reply. Oh well, not my problem I guess...

Walked to the table. Still no outlet, but I need the exercise. Posted to the list (one two three four five) instead of blogging about it.

I should try to get a peertube account. That's what mastodon is using to host videos. It's a thing I heard about long ago (one of the nudist bloggers moved there from youtube when it turned into prudetube). Still not ENTIRELY sure how it works, but if you're not looking to get paid per view by the site spraying advertising everywhere (you can do patreon or get your own sponsorships, which half of prudetube does these days anyway) it seems reasonable? Sigh, it's a pity Google Fiber barely works. (And that my $70/month or whatever it is doesn't get me a static IP. Alas, my little cube computer was turned off on a shelf for a while, and now refuses to turn back on. Dunno what's up with that...)

Although to be honest, if tumblr DOES implement mastodon integration and is a reasonable video hosting solution anyway, then I still might just want to reactivate my tumblr account, post videos there, and reblog them from my mstdn.jp account? Hmmm. (Fuzzy finally got her tumblr account reactivated. She actively pursued it for over a week and finally got a human to look at something.)

Ok, worked it out: there are THREE cases in cp -s. It's not just "absolute path" (which you don't prepend anything to) and "relative path" (which you prepend a calculated number of ../ to but otherwise use verbatim). There's a third case, which debian DOES NOT HANDLE:

$ mkdir one && cd one && touch first second third
$ rm -rf ../two && cp -sr . ../two && ls -l ../two
cp: ../two/first: can make relative symbolic links only in current directory

Because when you ls -l two/first you should get "../one/first" and "one" does not appear anywhere in the command line arguments. When the relative path isn't under the current directory you need more information. (That's what the two xabspath() before relative_path() were accommplishing.)

But the cp.test entry I added was effectively mkdir a; touch a/{one,two}; mkdir b; cd b; cp -rs ../a/. .; readlink one two which with debian produced '../a/./one\n../a/./two\n' but washed through xabspath() those two /./ dropped out.

So to do this right, I need to put the relative_path() call back, but only use it conditionally. Testing with fileunderdir() against xgetcwd() maybe? No, if the path started "../one/.." it still wouldn't work from the destination. Basically if there's any .. in it that doesn't undo a non-period directory it's already specified, then we've left our current context and it's gone funky.

Next issue: if I DO start supporting that, how do I prevent "cp -rs .. ." from recursing endlessly? Traversing up the parent to check inodes only works with stat() and I'm doing lstat()? No, the duplicate detection is testing fstatat() with 0 flags, which dereferences symlinks. Ok, that should work. (Needs a test though.)

Arguing with the cp -rs logic. If I wash the paths through relative_path() letting it xabspath both arguments, I get a correct result for "cp -r www blah" and such, but it doesn't match what debian's cp -s is doing? (I verify against TEST_HOST. Regression testing that the behavior hasn't changed != the behavior is RIGHT.) The debian one seems to be using the supplied source path verbatim, including any ../dir/. nonsense you include. Which might be relevant if you have a symlink in there that could get swapped out to point somewhere else later?

Sigh, tried to head out for a second programming round at the HEB tables in the evening and the place was PACKED. Wound up putting the can of tea back on the shelf and walking out because the cashier lines were like 15 minutes of standing. Didn't feel up to the 2 mile walk to the UT geology building (I've been under the weather since I got back from Minneapolis, not entirely sure why.) Thought about heading to the tables outside In-n-out but the wind is whipping all over the place tonight and those are just "table under umbrella" instead of in a nice protected alcove like the UT ones.

Wound up playing Persona 5 and going to bed. I need to address the "office space" issue...

Oh joy, the realpath stuff broke mac and I don't have a test case for that. Hmmm. (Gotta hijack the github test stuff, but it doesn't run reliably because system load shedding, and is on an asynchronous delay when it _does_. Plus it needs net to debug/test cycle and I'm mostly working out on battery with phone tethering because cat...)

Which of course turned into doing cleanup work on portability.h. If AT_FDCWD went into the kernel in 2006 (commit 5590ff0d5528 on top of 2.6.16-rc1) then why is it still #ifndef in portability.h? It's not old kernel or old libc, that's something like "Mac doesn't have it". But if I yank those old things that are WAY past the 7 year rule, how do I regression test non-Linux build environments? What ARE the build environments I care about regression testing portability.h removals against? I need to collate and triage VM images...

Fixing the shell's $((math)) logic: assignment suppression wasn't being inherited by recursive calls, so $((0 ? (x++) : y)) was incrementing x. I have to evaluate it to parse the equation, but shouldn't write the result back. Plus & is higher priority than && which meant 1&&2 was syntax erroring on "1 & &". Now I'm trying to add a test to prove that || is higher priority than && and... that's a tough one?

Hmmm, the assignment suppression inheritance is still going wrong somehow. And then there's "x=x; echo $((0?x:y))" which should NOT spit out an error about infinite recursion. It's not just "fail silently", the variable expansion should be suppressed because 'x="1?2"; echo $((0 ? x : y))' doesn't error, but swap the 0 for 1 and it complains about a missing ":".

(I don't care if shell plumbing is what I'm SUPPOSED to be working on, it's still a holiday. And I admit I've had a certain amount of writer's block NOT doing The Thing That Needs Doing. Getting the test suite running under mksh requires the shell to handle the test suite! It keeps coming up!)

The Kapo movie was quite good, in an art house sort of way. Pity it can't get distribution. Back when Austin had more in-person art house theatres, I watched stuff like "Mr. Brown" (1997) and Amazing Grace (2006) there which didn't get screen time at Cinemark. You'd think the Alamo Drafthouse would have picked up the slack for less mainstream releases when those closed, but it mostly went with reshowing older movies instead. If I want to see Gremlins on the big screen next week, they've got me covered. (I don't.) You might also think that streaming services would have a solution here, but there's no streaming equivalent of "small press" for books. You either have a big service ignore you, or you self-pub for free somewhere (which includes prudetube and vimeo), which doesn't work so well when you actually had a production budget and want this to qualify as a real movie. (Sort of the difference between "published in an academic journal" and "posted in a blog": one helps you break into the industry and the other does not.)

Just saw a six month old tweet from Ariadne Conill of Alpine Linux who was apparently disturbed by some of my leftover sleep deprivation names in toybox. The more tired I am while coding, the more obscure my variable and function names get. I usually go back and genericize the worst offenders, but I admit I miss a few from time to time.

When enumerating display fields, if I ask myself "what kind of fields are these" the obvious answer is "strawberry fields", and with a structure name like strawberry the obvious loop iterator is "forever" (because of the Beatles song), and after the dozenth reference to "pattern" I'm going to throw in a "logrus" just for balance, but that sort of thing bothers some people. *shrug* Oh well. I regularly do cleanup passes even on mostly finished stuff, going back to try to clarify things, and I often remove stuff I don't want to explain. The sleep dep names are just one category of that.

I've intentionallly left several. I don't expect everyone reading the grep code to be familiar enough with discworld to recognize "struct reg *shoe;" as a reference, but a "shoe" is also a box used for shuffling and dealing cards, and even if you don't... it's a variable name. And I already RESISTED having "struct timespec ts;" be "struct timespec its;" in the millitime() function in lib, which would also be a perfectly acceptable name for what it does). You can't please everyone, and in the case of Alpine I'm not trying.

Alpine doesn't use toybox and I don't expect them to start, because they don't want to. The stated reasons for their decision may change, but I don't expect the decision to. They built their system around busybox years ago, and it's the devil they know. Toybox wasn't ready back when Alpine started, and toybox becoming ready later doesn't matter to Alpine because that part of their system is done, and changing has costs. The relative quality of busybox and toybox isn't why they stay with busybox: they're familiar with their existing codebase and it ain't broke enough to fix.

Looking at toybox to FIND additional reasons to support a decision they've already made makes me grumble a bit, but hey: free code review. Throw it on the todo heap. (Or explicitly confirm I'm not gonna, which is a valid result of my review of their review.)

So I got a bug report that toybox grep is too slow in the presence of literally thousands of patterns to check in parallel. Um, yeah. It would be. Unfortunately the suggestion of gluing them together with | (like I originally did before abandoning that approach as problematic, and which could just as easily be done by the caller making the big -f file) turns out to be even slower.

My first guess here is to grab the "fixed" patterns (I.E. the ones starting with a matchable first character and having no control characters) and check those first using a cache-local loop (memcmp each pattern at same position, so it's hot in L1 cache) and possibly a bitmask of first characters that exist in a pattern (so I can rapidly skip any character that can't match any of those fixed patterns). I could even bucket sort the patterns into an array[256] so I'm only trying the ones that start with this character. If the pattern has no utf-8 escapes in it (nothing >127) then I can even handle case insensitivity cheaply, and the "." wildcard at a place other than the first position isn't too hard to handle either. Plus the -F logic kinda folds into this.

The simple thing to do is then fall back to the actual regex patterns using the old "traverse the whole string for each pattern" regexec() logic, although maybe it's faster to append a ^ match start of line to the ones that don't already have it and try it at each position so it's cache hot? (Especially if it has a non-magic first character for the bitmask check, although x* does NOT count because that's zero-or-more, and having an | anywhere in the pattern is another can of worms.) Depends how much startup overhead each call to regexec() has, how much slower is calling a ^pattern at each position vs calling regexec(pattern) once on the full string to be checked and letting it advance internally. (Does skipping characters it can't start with make it a win, at least in context of the rest of the plumbing?)

Sigh, I should do the simple thing first and see if that speeds stuff up enough for the use cases people are hitting. JUST the patterns that would work the same with -F first. (Adding in "." is easy, but it's also an extra test in an inner loop, and unicode says it should match a character... Speaking of does that mean _printable_ character? What do combining characters do with '.'? Can I get away with ignoring that in Android's use cases?)

According to man 7 regex, \ can escape ^.[$()|*+?{\ and case insensitivity gets weird in the presense of UTF-8 sequences (needing to convert to wide character and then do towupper() so I can't even strncasecmp() because I don't know how long it is, and wcsncasecmp() assumes the string has been converted into the wide character "array-of-int" representation)... anyway, those are "this is not a simple pattern" indicators. Having the . wildcard in it is ok (modulo probably ignoring unicode above), and $ I can also deal with (heck, for -F strings it's an optimization to stop looking before the end although probably not worth it for bulk pattern handling), and ^ might as well stay conventional regex because that HAS to be cheap (it's just checking once at the start of string, not at every position in the string)... But again, moving patterns from the fallback list into the -F-ish array[256] with the bitmask can be done incrementally to speed up more use cases, one pattern feature at a time. (Doing so incrementally also makes regression testing easier.)

Assuming Elliott will give me real use cases, they seem to think the test case with nothing but hex digits is enough. Meaning I only ever care about fully fixed patterns and my mask can fit in a short. Are any of the tests needing acceleration case insensitive? Dunno. Do any of those have unicode? (Do any of them NOT have unicode, which is easier to accelerate grep -i for.) I'm reluctant to speculatively implement codepaths that MIGHT help, I want to know what's NEEDED...

Elliott pointed me at what BSD does, which good to know, but... they're still traversing the string start to finish for each pattern instead of trying a more lex-style width first approach? (Ala the bitmask.) Which means tricks they're doing like figuring out how many charaters they can skip ahead when it doesn't match... don't really apply to what I want to try? Plus they have a whole second codepath for case insensitive unicode support and I'm waiting to see if that's actually necessary or if those can be handled by the fallback regexec() plumbing? (I don't know if the regex plumbing in musl or bionic currently handle unicode matches anyway, but if they _don't_ it's not my problem to _fix_ it. :)

Here's a section I edited out from reply to the debian guy, right after the observation that _POSIX_PATH_MAX and _XOPEN_PATH_MAX are even more meaingless than PATH_MAX, because "They don't describe anything about the current environment you're running in."

Toybox is sometimes clever _within_ a command, but tries not to be clever _outside_ of a command, if that makes sense? There's a certain amount of "I don't know this, I can't know this, I'm not going to TRY to know this" in the external interfaces. I can't predict the weird. They regularly invent new weird after I ship any given version.

The linux VFS lets filenames be 255 bytes long containing any character except NUL and '/' (with utf8 parsing left as an exercise for the reader: here are your bytes back), and the limit on directory depth is inode exhaustion. That's been true for years, and is likely to remain so.

You can always wind up with loops due to bind mounts (or mv while you're traversing, or probably other shenanigans) so the dirtree plumbing has to be prepared to traverse back up your ->parent pointers to detect same_file().

An individual filesystem doing something other than that is "notice system call failure and print errno while exiting" territory. (Hurd or BSD or MacOS may also violate the Linux VFS assumptions, but they're unlikely to _increase_ any of those constraints.) Some mangle case and some truncate filenames and selinux is probably turing complete and other processes can asyncronously delete CWD out from under you (hence openat() needing to be able to recover in a directory traversal implementation, that recorded tree down to where we think we are is useful for getting back as far as we can and going "ok, this branch is gone now, continue from last parent"...)

P.S. no lseek() variant on readdir() is ever reliable, modern filesystems return stuff in hash order which gets perturbed by large flowers and brightly colored wallpaper, so a reliable deep traverse that wants to back up and recover has to read an entire directory level and THEN descend into it, because otherwise you can't recover back up to where you were unless you re-read the directory from the start having recorded each one you saw and filtering it out to avoid processing it again. Presumably less of an issue with rm but if you want rsync or tar to handle arbitrary depth you have to close the parent filehandles to avoid filehandle exhaustion, and you kinda need to read all their data before doing so or it's just a mess. And I need to add tests to make sure they can recover from a directory subtree vanishing during traversal or being mv-ed out from under the old parent, and that's mkroot territory...

I have a tendency to write up long technical... "explanations" is a strong word but "this is what I'm thinking" doesn't quite have a snappy explanatory phrase. Anyway, I then cut them out of the thingy as a digression, but sometimes want to refer back to them. I sometimes think maybe that's what I should post to youtube, just aimless technical rants. That's more podcast stuff, really... Anyway, sticking it here is a compromise.

Got a little extra sleep, and had the first reasonably productive day (by my standards) in a while.

I wanted to see why glibc was pulling in linux/limits.h, and I don't like editing my global system state as root (never quite sure I've restored it all and I do NOT want to track down those bugs a month from now), so I did "cp /usr/include/linux/limits.h limits.h; vi limits.h; sudo mount --bind limits.h /usr/include/linux/limits.h" which works just fine... as long as you edit the local limits.h before --bind mounting it. And the changes show up fine in the standard headers included by your test build. But if you then "vi limits.h" again after the bind mount, vi deletes the limits.h file and writes a new one to break hardlinks, which also orphans the --bind mounted inode. (It's like still having an open filehandle to a deleted file, it gets deleted when the last reference goes away. The old contents are still there at the bind mount, not the updated contents vi wrote.)

Knowing WHY it didn't work doesn't make it better.

I'm running a bit behind editing and posting blog entries (as usual), and I just got my exasperated rant at gcc up yesterday, and woke up to an email response:

...from a sequencing point of view, I think this line is equivalent to:

> ss = ss + sprintf(ss = toybuf, "%d", h1->pos);

... which has a sequencing problem because the ss read isn't sequenced with respect to the inner ss assignment.

To which I replied:

Yes, I know. I'm saying that's nuts for multiple reasons.

The semantics of what SHOULD happen look very clear to me. Entirely possible the standard is what's crazy here, but it IS crazy. They can warn about this but can't fix it? If ss was a global that was modified within the function call they wouldn't be able to warn about this, so breaking it down like that is extremely dubious anyway, just perform the load last before the assignment and treat += as atomic with regard to the optimizer (not threading or smp, but you need locking and cmpxchg and stuff there already). The target of += has to be an lvalue, it CAN'T be that complicated to fetch. Sure the address-of-data could be an outright function call with multiple array dereferences and structure member offsets, but once you've GOT the final pointer to primitive type, don't dereference it until it's needed because yes it can change.

They're also saying that a register load from a local variable can persist across a function call. The function call MUST happen before the addition, they're requiring support for an order of operations that MAKES NO SENSE just because some optimizers might be crazy.

If it was ss = 1+(2+3) then the 2+3 has to be resolved before it gets added to 1, and there isn't an actual value stack on real hardware, what it does is reorder the operations so 2+3 happens before +1 in the resulting stream of generated assembly. So from THAT viewpoint there's no reason ss = ss + (ss = 7); isn't clear when the assignment is inside parentheses, we're SAYING it has to happen first (increasing its priority). And a function call should not be weaker than parentheses.

Maybe my viewpoint is affected by having maintained a compiler fork for 3 years. They could trivially have specified this behavior as stable and CHOSE not to, and I think that choice was really really stupid. (If sticking "volatile" on ss would have made the warning go away, then the optimizer was given too much freedom.)

I'm not saying "the compiler isn't like this". I'm saying it SHOULDN'T BE. Reality is that this particular compiler bug is at the design level, possibly at the standards level. But it's still a bug.

I occasionally engage with the posix committee, because I'm writing toybox and what they say... eh, doesn't really affect me since SUSv4/Posix-2008 came out 14 years ago and is still current barring typos (rumors of Issue 8 abound but until it goes on pubs.opengroup.org it's not a thing yet), plus I just try to DOCUMENT "devations from posix" rather than not have them. But it would be nice if my delta was kept small and even diminished in places.

But I've never bothered to engage with the C committee, both because I haven't maintained my own compiler version in 14 years, and I just RECENTLY moved from the 23 year old release of their standard to the 11 year old release. And also because the C++ compiler developers actively trying to sabotage C very much WANT to make this worse (so C++ looks less bad in comparison), and I would be defending "the way it should be" rather than "the way it is", which is seldom a strong philosophical position. If I was working on qcc it would fscking get this RIGHT... except I'm not and don't expect to have time to in the forseeable future.

But "optimizers affect the otherwise obvious language semantics" is a pet peeve of mine. No, it means your optimizer is broken. If you can warn about it, you can fix it. While I CAN dance around the user-visible sharp edges of your insane over-optimizer, I should not have to. "But other compilers might not..." then those other compilers are broken. At the very least give me more -fno-strict-aliasing variants to MAKE IT STOP.

Some squishiness is historical. Heck sizeof(int) was 2 on DOS and 4 on Linux, but the march TOWARDS things like LP64 has been an improvement. Yes historically "x = 0; printf("%d %d\n", ++x, ++x);" varied its output back under Turbo C depending on whether you selected C or Pascal function call semantics (and thus pushed arguments on the stack left to right or right to left, which affected the order in which it resolved them), and that got baked into the standard (the same way it refused to acknowledge "char is a byte" for the longest time). But that's childhood trauma, not a virtue.

If I know "int x = 0; x = printf("%d\n", x++) + printf("%d\n", x++); printf("%d\n", x);" is GOING to perform those first two printf() calls in order (as visible in their output), then why would it NOT perform the two x++ in order? (Which it DOES. Even with -O2 gcc gives the right output: 0, 1, and x=4. And yet it warns that some OTHER compiler might not.

Optimizers are absolutely FULL of logic to figure out what it can move without changing the visible result. X has to happen before Y, therefore that constrains what can shuffle around. If "a[ss] = ss++;" has an unclear order of operations, then DEFINE it. Pick one, specify, agree what should happen, document it, publish it, and now it's no longer unclear. This is not fundamentally different than 1+2*3 doing the 2*3 before the 1, we have operator precedence tables for a REASON.

When there's ALREADY nothing semantically unclear about what should happen, it's just a question of what people have chosen to defend. (Grumble grumble mark every variable "volatile" by default grumble.)

Optimizers need a MUCH HIGHER BAR for breaking code than compiler writers want to give them, we all should push back on this more. THAT is my complaint here.

Alright, so what should a sed protocol for tar --xform look like?

Launching an instance of sed per filename was always a hack, and the Android guys have noticed it's really slow. But a persistent sed process can get out of sync with tar, either by not returning any output for a name or returning what parses as two outputs (not only can you stick \n or \0 in the output via substitution, but filenames can have \n in them already). Doing a big read() doesn't guarantee we got all the data (PATH_MAX went byebye and if sed isn't doing a single atomic write() I can't guaranted all output will have traversed the pipe yet anyway), and no fiddling with timeouts to see if more data is coming isn't faster than forking a bunch of sed instances (which at least exit promptly).

I was thinking I could add a new --longopt to wrap a protocol around each transaction providing an explicit length (ascii decimal number plus newline before the data) on input and output. The downside is you'd HAVE to use toybox sed, the MacOS or gnu/dammit ones wouldn't have the longopt. (At which point I guess it could still fall back to the curent fork an instance each time behavior, but maintaining two codepaths to do the same thing is not ideal.)

I started trying to make it symmetric, but the existing plumbing is NOT set up to do that: it uses do_lines() out of lib/ which converts the fd it's fed into a local FILE * and then loops calling getdelim() and the provided function pointer, so when the callback function gets control (with the number line it can atoi() so it knows how much to read), it hasn't got access to that FILE * to fread() more data from. No, reading from the fd doesn't work because line reads don't know ahead of time where the delimiter is so they grab a block of data and store the extra until next time, the FILE * almost certainly has the start of the data we need stored in the structure. (This is glossing over sed's "read one line ahead to detect end of file for $ matching" complication, which is merely annoying in context.)

Probably what I want to do is just use -z for input: a filename can't have NUL in it (forward slash and NUL are the two characters prohibited in filenames by the Linux VFS) and tar is providing the input so won't play games with it: the user-provided sed pattern is the tricksy source of breakage.

At which point I'm tempted to just say "sed -z" and if somebody's stupid enough to "d" the string or insert a \0 call it pilot error? I doubt macos sed has -z, but we don't compile with macos sed either: macos sed is trash. The bigger problem is this SMELLS security exploitable. I don't know how, but this is exactly the kind of sharp edge that gets buried three layers deep and then combined with unforseen weirdness, and suddenly you've got a tarball validator stomping system files. Except... you could already do that with ../../.. couldn't you? Hmmm... (Note to self: test that transform happens BEFORE checks for "under this directory" and so on.)

Wendy's! Actually hanging out at the booth with a 4 for 4 and my laptop for the first time in I have no idea how long.

Oh goddess, github has ACHIEVEMENTS now? Apparently I've contributed code to a mars explorer (via linux-kernel), to an arctic vault (also via linux-kernel), and have "starstruck" level 3. Did this really need gamification?

Got a bunch of email answered and cleaned up after the recent test suite API change, and had serious momentum going when my laptop battery ran down enough it suspended. As I suspected, the fact Wendy's no longer has public bathrooms is not the limiting factor to how long I can work there...

This confirms that I definitely need better office space to get work done. I miss the hotel rooms in Japan. (And even the Hello Office.) My house's central air is one-directional: air comes OUT of vents but the only air INTAKE is one big one in the ceiling of the little hallway off the living room (I.E. right under the air conditioner itself, where we change the filter). So if I close any of the rooms and stay in them for hours, they get noticeably warm and stuffy. More of an issue with the small front room that used to be an office before we turned it into storage, but even the big master bedroom isn't _ideal_. (I can set the bathroom exhaust fan going there though.)

Alas using your bedroom as an office is a recipe for insomnia, and the kitchen table I normally use means Fuzzy and Peejee constantly interrupt me (no door I can close to keep them out, which is also why the air conditoning works well in there).

Reading a page which says "China's sudden lurch from Third World basketcase to dynamic modern economy... In the 1960s, sixty million people died of famine in the Chinese countryside; by the 2010s, that same countryside was criss-crossed with the world's most advanced high-speed rail network..." and those things are orthogonal?

It's like saying "millions died in Pol Pot's massacres, but a few decades later they had cell phones". The one did not cause the other. The mass deaths in china were because Mao was the poster child for Dunning-Kruger and kept doing stupid crap like the "great leap forward" where he forced everyone to melt down their farming equipment into ingots for display purposes, and the "four pests campaign" where he trashed the ecosystem. Nixon going to China and Deng Xaioping pulling a 180 on private industry were a repudiation of Maoism, and emperer Xi pivoting back to Mao is doing huge damage to his country.

China see-sawing rapidly between flood and drought today is partly because of climate change, but MOSTLY because they dammed every river in the country multiple times turning the natural water distribution network into a series of evaporation ponds (and then stuck the south-to-north water transfer project on top of it with peverse financial incentives). Here in the USA, Atlanta Georgia was already a deadly heat island decades ago because they replaced enough trees with black asphalt parking lots to affect the weather patterns for miles around. The CCP did that sort of thing on a much larger scale, and every time things got worse they doubled down.

Mao also _prevented_ more chinese people than he killed with his "one child policy", which got worse every year it was left in place (kind of like "stack ranking" does in silicon valley) to the point where they have two consecutive "only child" generations and have now lost the cultural norm of HAVING children (or the economies of scale that make raising kids an economically viable life choice). This means China's army is completely hollow because they can't afford any personnel losses; every dead soldier has two parents and four grandparents who have no remaining descendants, have seen their line end, and have nothing left to lose.

The USA has its own problems here, of course. Thirty five years ago we had "latchkey kids" all over the place, and older siblings babysitting younger ones. Now our broken healthcare system attaches a $30,000 hospital bill to childbirth, day care went from "hanging out at a neighbor's house" to licensed specialists with college degrees and regular inspections (the closest of which is a 4 mile round trip drive with a 2 year waiting list and 4 figure per-child monthly cost), and children are so rare and precious that the slightest hiccup (such as an unescorted nine year old seen walking a block from home) makes Child Protective Services come take them away, meaning children are now risky and stressful to have. Raising two kids to adulthood is more expensive than paying off the mortgage on a house big enough to raise two kids in, and two is still below replacement rate...

Once again, the solution starts with "every single Boomer dies". How the unsustainable social norm is them removed is an open question: taking the society it's attached to down with it is the most obvious way. Ghost towns become ghost cities, sped up by hurricanes, tornadoes, golf ball sized hail, 120 degree days with rolling blackouts... Hopefully we find a softer landing than that, but we can't START while christian dominionists are actively trying to bring about the apocalypse because they think they'll be raptured to cloud nine and get to meet Santa Jesus.

I am reminded of Neil Gaiman's observation (in his "make good art" commencement speech) that one day he realized he had become someone who professionally responded to email, and wrote on the side.

I'm mildly exasperated that I keep writing up largeish responses to _private_ messages. Three different such threads in progress at the moment, including bash maintainer Chet Ramey replying to something off-list. It's not that I don't want to have these discussions, it just seems non-optimal to research stuff and write it up, and then not be able to publicly refer to back to it. (I like to feel I'm at least contributing to a public body of knowledge. It doesn't have to be GOOD to form part of somebody else's research slush pile, if all else fails learning what NOT to do by example.)

And I KNOW I'm not doing youtube videos fast enough when other people wind up basically doing them for me. (Which I'd point people at more prominently if he wasn't using the old abandoned github fork that says at the top of the README "it moved into toybox", as visible in the start of his video. I'm simulaneously proud and embarassed. He spent the second half of the video trying and failing to get the dropbear build to work. It works in the current one. I checked last release!)

Mixed feelings about one's work are normal. (Neil Gaiman defines a novel as "a long piece of prose with something wrong with it", although he attributed the quote to Randall Jarrell.) I remember when I stumbled across an old discussion of toybox that seemed at the time like it was dominated by Gavin Howard going "he hasn't updated my bc code, therefore the project is trash". I was strongly tempted to just delete that bc and maybe write a new one from scratch at some point in the future. What stopped me from doing that was, ironically, being unsure if bc is even still a thing? All I've seen anybody use bc for in ages is to run Peter Anvin's kernel script to calculate the clock tick granularity. (Which I posted both C and shell replacements for before he checked it in, it was Peter's spanner in the works for the perl removal patch series.) Sure bc is in posix, but so are compress, uucp, sccs, ed, fort77, mesg, qdel, pax... All the kernel _really_ needs bc to do is perform the equivalent of kernel/time/timeconst.bc, last modified in 2018. (There's one other use in tools/edid/Makefile where it basically performs printf %x $((100-$NUM%100)) for some reason? I mean, seems a bit overkill? But the tools/edid directory was moved from Documentation/ and isn't listed in tools/Makefile so I can probably safely ignore it.

If I did want to write a new bc, I finally got the $((math)) logic in the shell implemented, and I know the _theory_ of arbitrary length math and calculating trig functions by formula. But... I have literally zero use cases for it? The timeconst.bc plumbing is just multiplication, division, and modulus (remainder from division). It LOOKS like it's using exponents but 2^b is actually a bit shift. Everything it does fits comfortably in 64 bit math.

That says to me we really don't NEED bc. Linux From Scratch and gentoo didn't bother to add it to the base OS until it showed up as a kernel build dependency.

Circling back around to that bug I need Fedora to reproduce, I downloaded a new Fedora .iso that runs a version of glibc that deigns to compile with a current-ish gcc version, confirmed the issue does reproduce there, built glibc, confirmed "LD_LIBRARY_PATH=../glibc/build toybox" gave me the command list... but then trying to reproduce the issue under (as yet unmodified git build of) glibc went:

Fatal glibc error: ../sysdeps/nptl/fork.h:83 (reclaim_stacks): assertion failed: l->next->prev == elem

Before it even GOT to the failing part I could (presumably now) stick printfs in. Dunno if this is git commit du jour having an issue (continuous integration sucks), or subtle version skew between whatever libraries it found in the glibc build directory vs ones it loaded out of /lib, or if it's detecting subtle memory corruption that ASAN, musl, bionic, and debian's version of glibc didn't...

Ok, I'm pretty sure it's not that last one because:

$ LD_LIBRARY_PATH=../glibc/build ldd toybox
Fatal glibc error: ../sysdeps/nptl/fork.h:83 (reclaim_stacks): assertion failed: l->next->prev == elem
ldd: exited with unknown exit code (134)

Yeah, it didn't even enter toybox main() and already bailed. Presumably in the dynamic linker code. Except ldd without all that says:

linux-vdso.so.1
libcrypt.so.s
libm.so.7
libc.so.6
/lib64/ld-linux-x86-64.so.2

The first of which is kernel magic, the middle three I gave it symlinks for (and I ran ldd on each of those three to make sure they didn't pull in any OTHER libraries than the ones in the above list)... is there version skew with the dynamic linker?

The infrastructure is just _spectacularly_ brittle.

Ok, let's try rolling the version backward... and Fedora doesn't say what the version it's using IS. Almost every other shared library is a symlink to a filename with the full version number, but /lib64/libc.so.6 is a file, not a symlink. Great, let's ask "yum info libc6"... network is unreachable. Fedora no longer has a local package repository? It has to talk to the server AGAIN after I already installed strace using yum so it's already initialized whatever there is to initialize...

Sigh. My phone still has a web browser, and Google says Fedora 36 should be using 2.35, so check out glibc-2.35 from git and... yay? I think that worked?

For a while I've wanted to change the toybox test plumbing so:

testing "cat1" "cat" "hello\n" "" "hello\n"
testing "cat2" "cat infile" "hello\n" "hello\n" ""

Is instead:

testcmd 'cat1' '' $'hello\n' '' $'hello\n'
testcmd 'cat2' 'infile' $'hello\n' $'hello\n' ''

The testing->testcmd switch is ongoing, but the move to $'' would avoid gratuitous use of echo and in theory makes it a little more obvious what fields are processed (although 4 of the 5 are, so it's only the first one that ISN'T). It also makes NOT processing them easier: when I have to pass through backslashes or escape sequences to a command it can take me several attempts to get the amount of extra escaping right.

The downside is A) this is needed commonly enough that just doing it for you all the time made sense -- most input and output ends with \n, B) mixing environment variables with this becomes a bit awkward because $"blah" vs $'blah' aren't the same as "blah" vs 'blah', the double quote version is some completely unrelated (and useless) internationalization thing. (I don't remember whether to be annoyed at bash or posix, and am not looking it up right now.)

Part of the reason I've wanted to change this is the txpect plumbing I implemented (yes, I wrote a version of "expect" in shell, which sh.test uses quite a bit) already does this: when you want a newline, you $'\n'. And I can do "$VAR"$'\n' when I need to, it's a LITTLE awkward but not THAT awkward?

But the friction of changing every existing test, and the relatively minor benefit... It's one of those "In academia the fighting is so vicious because the stakes are so small" things. (I did, however, check to make sure mksh can do this too, and it can...)

Aha! Ok, the REASON not to do that is shell variables can't contain embedded NUL bytes, so echo -en 'a\0b' can feed the whole string with the NUL byte in it to a command's stdin, but $'a\0b' fed as an argument into the testing() function gets truncated at the NUL byte (because that's end of string using C's efficient but limited in-band signaling).

I should do an explainer video on C vs non-C string handling. (Back in the day "Pascal" was the big comparison because Turbo Pascal and Turbo Basic were the big competing languages under DOS, and C and Pascal could share .o files as long as you told the C compiler to --use-pascal function calling conventions, hence the "functions evaluated in reverse order" user-visible behavior change.)

The tl;dr is the alternative to an in-band signalling string terminator is explicit length tracking in a second variable, which means each pascal string is a struct from C's point of view. If your length tracker is a single byte, your strings can't be longer than 255 bytes long. If it's a short, you're limited to 64k which still comes up sometimes. If it's a 32 bit length then each string takes an extra 4 bytes to store, which adds up and imposes alignment constraints.

But the bigger problem was you wind up doing a lot of unnecessary copying. A pascal (or Java, or Python) string object points to the start of a string, so if I want to remove the first character I memmove() the whole string (or add _another_ variable to the struct to track starting offset, which has its own death-by-a-thousand-cuts problem set). In C, I can just ptr++ as long as I keep track of what I need to free later. Grabbing data out of a big mmap() or something is easy in C, but in Python you allocate new memory and copy it even for trivial things. Since string processing is (more often than not) the most expensive thing a program DOES (and fundamentally cache unfriendly), this was kind of an important thing to optimize for. Now after a zillion iterations of Moore's Law people care less than they used to, but C (and its Bizarro world version C++) still dominates because "runs twice as fast" still means "eats half as much battery doing so".

The downside, of course, being that in-band signaling can always spuriously trigger and treat data as metadata without rigorous (and inherently recursive) escaping. Which brings us back to "unix process arguments can't include a NUL byte".

Another fun corner case with diff -B is that the return value is 0 if all the matching hunks were suppressed, meaning the FLAG(q) logic has to go _after_ collating the match lines and determining -B suppression. One more test for the test suite...

I started downloading the new Fedora 36 ISO before testing something (partly in hopes it would fix the version skew issue I was seeing trying to swap out the glibc version in the Fedora 34 ISO), and it said it was going to take 2 hours to download because Google Fiber is slower than my phone tether. But my phone tether has a maximum monthly tethering bandwidth before it slows WAY down, and a gratuitous extra 2 gigs is noticeable especially at the end of the month, so I would LIKE to make the fiber do this.

And then while watching a youtube video on my phone as it downloaded... the video hung. For a full minute. Because my phone was associated with the Google router instead of T-mobile (again, save metered bandwidth at the end of the month), and it can't manage to watch a video on Google's servers using a Google phone now? Great.

So I called Fade to see if she could contact Google Fiber tech support yet again, and they wanted me to install the Google Home app on my phone since she's back up in minneapolis, but when she couldn't make it add me to the thing remotely as a "guest", she eventually gave me her login info for the "Google Home" app...

At which point Google added 77 contacts to my phone. I now have the phone number of a bunch of people I barely know mixed in with my phone numbers. I did not ask it to do this. I actively do not want this. There is no undo button. I uninstalled the "google home" app again but apparently the damage is done. (This sort of unwanted "integration" is one of many reasons I've avoided Microsoft products.)

I deleted a test contact off my phone and it deleted it off of Fade's phone in Minnesota. I was afraid of that. It's got her credentials stored somewhere, probably under system settings... yup, there's a google accounts tab, remove her account... it gave me a big warning about how that would remove all her contacts and I went "yes exactly what I want"... and then it didn't. The pop-up lied, they're all still there in my contact list.

Now that I've removed her account, let's try rebooting the phone... Nope, they're still there, but now Signal has notified me that 17 people I don't personally know are all on Signal. Which means Google has inappropriately notified THEM that I'm on signal (NOT UNDER MY REAL NUMBER, it's some voip thing Jeff set up that's since expired I think, I just installed this app for the j-core engineering chat channel when they decided to move there from Line). The cross-contamination ripples out without asking, propagating misinformation, and I can't seem to stop it.

Of course this isn't exactly a high water mark for Google doing that to people, which is why I consider uploading anything to Google cloud services to be equivalent to posting it to usenet. Any information put into a Google Doc is sent to advertisers (Jeff's complained that he gets ads related to things he uploads into the private corporate workspace he pays hundreds a month for; that's information about upcoming patents and trade secrets that's STILL going to advertisers), which also means it's purchased by all the state actors and law enforcement agencies and so on, since those all have have front companies buying all the advertiser data alongside the corporations doing so. Not that Faceboot and Amazon and Palantir and Blackwater being able to buy my data is any better. Every time I look at an amazon product on my phone or laptop it's in spanish these days, and I can't change the language preference without logging in, which means creating an account. Plausible deniability, of course. As with twitter demanding my phone number, it just means I can't use that service anymore.

Can't complain: I'm the product not the customer. It would SEEM like we could complain since we pay for Prime, so I AM the customer. And Jeff pays for the Google Workspace that bases ads on proprietary business plans. But being the customer doesn't mean you're not ALSO the product.

And without selling all our data, how else do we expect the GOP to figure out who to arrest for buying birth control? Preventing them from arresting people would deprive them of their religious freedom to burn witches. Long christian tradition, Cain killed Abel, Moses had 3000 of his followers killed for disobedience after escaping egypt (in the movie Charlton Heston threw tablets at the golden calf and it exploded, but in the book he ordered his followers to stab them: Exodus 32, 26-28, see also Matthew 10, 34-37), Abraham was told to sacrifice Isaac, king David had Uriah killed so he could marry Bathsheba (king Solomon was their son). An especially godly thing to do is murder the kids of parents you don't like, such as David's first child with Bathsheba (after which David apologized for the murder, which made it ok), the firstborn of Egypt, Job's wife and kids... (But that's ok because God gave Job replacements later. No the original wife and kids stayed dead, but they're apparently fungible? The important thing is that a beardy man had an experience. Anyway, that's why ICE putting kids in cages is very christian.)

At least now that I've removed Fade's login from my phone's "list of Google logins" (again, didn't ask it to add her there, I was trying to log into just one app to poke the router, not change my phone's global state and send my wife's contact list out through signal)... Anyway, at least now that connection's broken, I've confirmed I can delete Fade's contacts that have spilled over mine without deleting them off her phone as well. If I can figure out which ones are hers and which ones are mine that I'm just not recognizing out of context...

And of course the Google Fiber support drone once again couldn't fix the router, just like all the other times. This time they wanted us to move the router out of the kitchen, which... look, aside from signal strength NOT BEING THE PROBLEM (the "2 hour download" was while my laptop was on the kitchen table about eight feet from the router), the kitchen is where Google's installer drilled the hole in the wall to install the fiber jack, and the ethernet cable molded to that jack is maybe 5 feet long and would go over a tile floor if we did try to run it anywhere. No, I can't feasibly move it from where the installer put it. The fundamental problem is they made the curb cut on the wrong side of the property, so they couldn't go up the same side of the concrete driveway the cable TV cable went up but instead had to go up the other side, where the concrete driveway slab continues around the side of the house to become the concrete back porch: they drilled through the first wall that's not behind multiple feet of surface concrete, and on that side of the house it's the kitchen. All that concrete was poured long before we bought the house, and easily visible on Google Maps' own satellite photos. That said, it's not a big house and the wifi signal reaches it all reasonably well from there.

The guy who was here to swap out the fiber jack's lens months ago explained exactly what was wrong with the little white circle router, basically it's too "smart" and once it gets this kind of confused it needs to be replaced. (He did NOT say "probably because of all the data collection spyware", he said something like "adaptive routing tables", but a gigabit connection to a single next hop at a fixed IP does not need to make any real routing decisions? There only IS one choice of where the packets go from here. That guy was ready to replace the router while he was here, but couldn't without the associated app and Fade was in Minneapolis at the time. Ever since, the Google Fiber phone drones don't understand the system anywhere near as well as that tech did. No the ethernet cable is physically attached to the fiber jack so it can't be unplugged, no it's only a few feet long, yes rebooting the router specifically (not the fiber jack) is what makes it fast again for hours at a time, which means it can't be a cable problem or a wifi signal problem. And even when the router's slowed way down two devices talking to each other within the house are still fast, and that goes through the router because of the way wifi access points work. Phone Drone does not understand any of that, they're always reading from a script titled "how to avoid spending money on service calls".

I'm trying to deal with this now because during the several months Fade was here over the summer, Google Fiber support refused to send the original tech BACK out to replace the router like he'd offered to. Every time we contacted them since it's been excuse du jour.

Anyway, I rebooted the router again, so it's fast again for the evening, and I downloaded that "2 hour" Fedora ISO in 4 minutes 58 seconds. (The FIBER part's been fine ever since the support guy replaced the lens, it's Google's crappy ROUTER that's broken and which they've refused to acknowledge, let alone fix. And it is so very clearly a software problem, not a hardware one...)

Now to spend far too long cleaning unwanted contacts out of my phone. Is this context-free half-remembered name one of my wife's friends I've heard mentioned but never met, or someone I know but haven't talked to since before the pandemic? Is 'Dad' her father or mine? A fun evening to look forward to.

(Yes, this is my normal experience with technology. I break everything. Too-smart automation did something I didn't ask it to, and now I have to get a mop. Ironically, it happened while I was trying to fix OTHER too-smart automation that keeps doing things I didn't ask it to. I get angry at Google because I still expect better from them, but I'm slowly learning to treat them as a Curate's Egg.)

What a nice way to start the morning. From a guy telling me that the developer behind an attempt to revive Java's "write once run anywhere" promise by genericizing Apple's old universal binary concept to support every OS on every type of hardware simultaneously... It's ok because that developer used to work for the company that removed support for all but 2 architectures from Android, which makes them a domain expert on portability. Ok. You do you. Of course they feel I need to read the new attempt's code to be allowed not to care, the same way each new attempt at making microkernels work is sure I need to read THEIR code to be qualified to continue not to care about microkernels.

This is fundamentally the same issue as "proof of my particular god" du jour. A theologian who has studied every Star Trek novel word by word and gone through every movie and TV show frame by frame to PROVE klingons are real, refusing to talk to anyone who hasn't even seen the last season of Deep Space Nine. Uh-huh. Let me know when whatever you're excited about actually arrives. "Coming Soon Now, You'll See" is a promise I've heard before. If it's the kind of thing where ignoring _one_ person being excited about it means I'm in serious danger of never hearing about it again... how is that an argument AGAINST my position? I heard about llvm repeatedly for something like 6 years before I tried it: it was still there when I got around to it.

So yeah, that's today's new tangent somebody strongly feels I should be following instead of closing tabs I already have open in the giant todo heap.

Got my new diff implementation producing a reasonable diff for Ray Gardner's test case, which is a step up from segfaulting. It's still wrong in two places (the @@ line count of a hunk is off by one, and the three trailing match lines are repeated in the hunk and the greedy matching algorithm is eating them early, so it ends with three - lines. Which is technically correct, that IS a valid view of the change that happened... but unified diffs should only do that at EOF, which this isn't). Not sure what's up with the first problem, need to stick a bunch of printfs into it to see where it's going off the rails. (Also, "mine doesn't match theirs" doesn't always mean WRONG, but I'm not sure how that would be the case here? Do we NOT have the same number of lines in the hunk?)

The easy way to fix the second problem is to compare and peel off the trailing lines before evaluating matches and stick them back on after, which is ugly but conceptually simple. A _better_ way is probably to have the algorithm burn the candle at both ends (have the loop alternate pairing off matches from the start of file and the end of file), but that's a bigger change and I want this DONE. (Sigh, I also want it _right_...)

Meanwhile, I'm not entirely certain what the definition of diff -B is supposed to be, because:

$ diff -Bu <(echo $'one\n\ntwo\n\nthree') <(echo $'one\n\ntwo\nthree')
$ diff -Bu <(echo $'one\ntwo\n\nthree') <(echo $'one\n\ntwo\nthree')
--- /dev/fd/63
+++ /dev/fd/62
@@ -1,4 +1,4 @@
 one
-two
 
+two
 three

Technically, BOTH hunks only differ by blank lines. But debian's diff is not suppressing the second hunk with -B because it decides to treat "two" as having moved instead of the blank line having moved. Which is a coin toss. (Again, "why is the diff algorithm making these decisions" seems like a thing I should understand. I see WHAT it's doing. Should it be?)

My laptop battery's been increasingly fiddly over the past few weeks: whatever communication with the microcontrollerr lets it start charging doesn't happen unless I hold it at exactly the right angle for a moment, then I can put it back down and it's fine up to 100%. The other day I plugged it in and went to bed and hadn't noticed it hadn't started charging, and then no matter what I tried it wouldn't _START_ charging (the battery was at 90% so I could have gone off and used it... but that's the old Voyage of the Dawn Treader when to blow the horn question, isn't it? Besides, if it loses power completely all my open terminal windows with half-finished things where screen backscroll and command line history act as notes-to-self about what I was in the middle of, and all my half-finished thunderbird email reply windows go poof with no record of them because thunderbird has normal Linux levels of usability.

(Thunderbird's "repeat every recognized email address twice" thing is annoying enough, some sort of html reply code triggering in text mode. And when I'm composing in text mode reply lines still act weird for editing reasons and sometimes hitting backspace at the start of the line deletes the end of the previous line instead of collating. And then it wordwraps the sent mail at something like 76 characters, which combines badly with dreamhost's web archive when viewed on my phone's browser which isn't wide enough to show 76 characters even in landscape mode and of course pinch-to-zoom doesn't re-wordwrap but instead puts text off the edge of the screen and makes me scroll to see it, or zooms out to only take up half the screen but doesn't change how it wordwrapped it... but I KNOW the phone is doing the wordwrapping because my laptop browser doesn't do that. Yes that's thunderbird, mailman, and android chrome combining into a captain planet of bad presentation, with each somehow making it worse.)

Anyway, a replacement E6230 battery (6 cells, nobody can quite agree how many coulomb-fornights it should have but somewhere AROUND 65 probably) is $30 through amazon and like $12 through ebay. (Fade went through amazon because she didn't trust ebay, devil you know I suppose.) And it arrived this morning and I left it on the couch rather than fiddle with it because after sufficient tilting and tapping, I got the old one charging again! (The CAPACITY seems reasonable, still well over half of factory specs. There's just something corroded or cat-haired or dendrited about the connector that makes it not want to START charging, but once it starts it stays started. Again, devil you know...)

And of course when I got to the table, it was powered off and I'd lost all my open windows. Fired up again claiming to have 100% charge though! Sigh.

Still trying to debug what's wrong with tar --xform as Elliott's seeing it but I'm not. I reproduced the problem on a Fedora image and it's somewhere in glibc's getdelim(). It looks like the "stdin" object is getting corrupted, although I have no idea how, and would really like to stick printfs into libc.

If you build glibc from source out of git and do an LD_LIBRARY_PATH=. ./toybox with libc.so.6 in $PWD, it says "version `GLIBC_2.33' not found (required by /lib64/libtinfo.so.6)" which I did not link it against but Fedora is a taste sensation dependency explosion.

But if I "git checkout glibc-2.3.3" ../configure then dies with:

checking version of as... 2.31.1, bad

Because:

$ as --version
GNU assembler (GNU Binutils for Debian) 2.31.1
Copyright (C) 2018 Free Software Foundation, Inc.

But configure has:

echo $ECHO_N "checking version of $AS... $ECHO_C" >&6
  ac_prog_version=`$AS --version 2>&1 | sed -n 's/^.*GNU assembler.* \([0-9]*\.[0-9.]*\).*$/\1/p'`
  case $ac_prog_version in
    '') ac_prog_version="v. ?.??, bad"; ac_verc_fail=yes;;
    2.1[3-9]*)
       ac_prog_version="$ac_prog_version, ok"; ac_verc_fail=no;;
    *) ac_prog_version="$ac_prog_version, bad"; ac_verc_fail=yes;;

Yes, the as version I'm building with is TOO NEW to build glibc 2.3.3, which will only accept 2.13 through 2.19. Same for most of the other tools. Yes, the "copyright 2018" tool is too new to build the libc in a Fedora version that has a March 2021 date on ls -l /bin/ls.

The _levels_ of brittle they've built into this thing are just... sigh.

Ray Gardner wrote up his own explanation of the classic diff algorithm, and I'm trying to read it to see if it makes more sense to me than the other writeups did.

Diff and longest common subsequence

The algorithm is trying to collect/mark all the lines that will be kept (leading space instead of leading - or +). I got that part. We want to keep the most possible unchanged lines, but there may be more than one way to do that and some are less legible than others.

Saying that a subsequence is zero or more symbols and that it's different than a common subsequence didn't really help my understanding, because it's spending time defining terminology in a way different than what someone who has used the output of diff and the input of patch for many years already calls these things, instead of bringing the mountain to McDonald's as it were.

The paper by Hunt and McIlroy describes HM better than I can

Then why...?

Sigh. He KNOWS I already found and read that paper, and I said it did not explain to me "why are they doing it that way?" (I can see WHAT they're doing. That was never the problem. Some PDP-11 programmers back in the 1970's decided loading the entire file into memory at once and hashing every line was the best approach: Why did they do that? What are the actual advantages of this approach, what are the downsides, what are the corner cases and pathological inputs, what are the design tradeoffs and alternatives? The McIlroy paper SAID that doing it right is N^2 and the approach they've taken is cheaper but imperfect, but didn't really go into HOW. Sorting has quick sort and heap sort and shell sort and so on, with explanations of the tradeoffs and videos showing them in action, some by dance troupes. I am not finding anything like that here.)

Aking me to read a paper which literally says I should just read the other paper again, presumably over and over until I "get it"... Great.

Let's look at his python code and see if it's illuminating:

# Here if returning actual LCS elements:
# Omit step 9. It weeds out jackpots, where the LCS contains matches
# between values that hashed the same but are not actually equal
# due to hash collisions. Instead just return LCS elements from B.

If a "jackpot" is a spurious hash collision, why not say spurious hash collision instead of inventing more terminology? Is omitting step 9 what weeds out "jackpots", or would step 9 weed out jackpots and we're not bothering to do that? (If so wouldn't NOT checking for hash collisions have consequences? If our hunk says lines are equal that aren't actually equal, that seems problematic...)

Sigh, I can sit and chew through this too, but it doesn't seem much more productive than the first five such writeups? There have been at least two variants since "patience diff" all built on top of the same base, and git had performance issues due to changing hash algorithms, and everybody keeps tweaking diff but nobody has bothered to explain WHY the unquestioned base they're all building on is like that.

The real problem is I've reached the point where I'm tired of this problem space and don't really WANT to implement a "diff" at this point. It is not fun. Especially since a while back I threw up my hands and went "I'll write a new one from scratch that I _do_ undersand!" and did most of that work and people keep going "how stupid are you not to understand the old one, here, it's easy, stop what you're doing and look at what I wrote.... which literally says "read the math paper from the 1970's again".

Long ago I wrote a "patch" implementation that did not work like other implementations, and it turned out to be ok. I wanted to explore doing that in reverse for diff, but I didn't get to because I got an external submission dictating how it should go (which wasn't even patience diff, which I'd asked about at the time), and my questions about it are met with incredulity. No "why" question has received a "why" answer so far, and I am tired.

August 26, 2022

August 25, 2022

$ Z='x=1;x<5;x++'; for(($Z)); do echo $x; done
bash: syntax error: arithmetic expression required
bash: syntax error: `(($Z))'

June 13, 2022

June 12, 2022

Yay, I got a new patreon supporter! Welcome Vikram! (I'd send a hello message but I have to fire up a VM while connected to internet for that, and I'm out at a table on battery reading the email I downloaded before heading out.)

I also need to re-enable every gmail user on the mailing list YET AGAIN because its out of control spam filter sometimes rejects delivery instead of putting false positives into the spam folder where its mistake can be fixed. Youtube also kept getting a true/false test wrong so turned it into multiple categories EACH of which it gets wrong. (Why does gmail sort stuff into folders AND sometimes reject delivery?) Complicating things is not always the right fix. Youtube at least has the excuse of being under constant attack from nazis trying to game its recommendation algorithms with the backing of multiple state actors (China, Russia, Saudi Arabia...) plus the plutocratic moralizing of payment processors who have literally cornered the market on "spending money" without anyone seeming to object and are happy to bow to right-wing loons who think we should all wear hairshirts and flagellate ourselves (it's amazing how much random anti-sex bias everything in the USA gets sprayed down with that you don't find out about outside of funny anecdotes; as with all the racism and misogyny it's just cultural background noise). That's on TOP of the insane IP regime and capitalism's insatiable demands for growth (squeezing blood from a stone until whatever it is collapses). Youtube is doing a terrible job under extreme duress. What's Gmail's excuse?

Our oven's heating coil blew out last month, and the new replacement stove we ordered just arrived. This one's induction, and literally NONE of our pans (except the two cast iron frying pans) work with it. Several are magnetic on the sides, but not the bottom? (Really?) Oh well, Fuzzy bought one induction-friendly pot for now...

June 11, 2022

June 10, 2022

June 9, 2022

June 8, 2022

June 7, 2022

I need to make and upload more youtube videos, but every time I check in youtube continues to implode. Why even have age restricted content if you randomly delete people's videos or entire channels under the same capricious guidelines? Sure, Youtube is not the only online service that's gone nuts against its own userbase, but since I've never bought or sold anything through ebay I have the luxury of not personally caring about its decline.

(Speaking of which: dear The Economist, saying "When the driver shortage normalizes..." about Uber/Lyft's problems is another way of insisting we'll inevitably return to a pre-pandemic "normal". That's an "if" at BEST. A third of the Boomers haven't even retired yet and the trucking industry is already trying to recruit them for its own driver shortage. The average number of children in the USA is down to 1.7 (replacement rate is 2.1 since their kids-per-adult only counts adults, and not everybody survives to adulthood). We've historically made up for that with immigrants but that's cratering too (gee, wonder why). In addition to the Boomers not getting any younger, a significant contributing factor to "the great resignation" was workers dying of covid (especially people who interacted with the public), and an order of magnitude more got long covid and no longer had the energy for a second or third "side hustle". So Uber's dependence on all those schoolteachers driving for them after class isn't necessarily replicable going forward.)

Youtube has always had a creator burnout problem, but at this point half the things wandering by on my youtube recommendations are reuploads of old videos that got some retroactive stupid strike months or even years after they went up. Channels where the creator died inevitably become motheaten as videos in series go down. And no it's not "algorithms", the prudish sexism is coming from the top, on a site where the CEO gave herself a "freedom of expression" award (and yes the annoy people into subscribing guy reported to her).

*shrug* I'm happy to let Youtube die, I just need to figure out where to host my own videos if it's no longer viable, vimeo doesn't want to be in that business anymore, dreamhost is too cheap to be particularly load bearing, and Google Fiber... even if they weren't doing the "it would inexplicably cost 4x as much to get a static IP" scam, their router drops to dialup speeds if I don't unplug it and plug it back in every few days. (The guy came out and replaced the little lens in the box on the side of the house, but for some reason the white circle Google left here keeps getting slower and slower the longer it's on. We have to power cycle it every time we want to watch anything on crunchyroll.)

June 6, 2022

June 5, 2022

I still haven't got adb hooked back up to my phone: the version installed in debian, the version in the AOSP checkout, and what my phone expects are three different things and the first thing ADB does is launch a server daemon out of some sort of search path, which requires installing a magic debian package (android-sdk-platform-tools-common) so it can supply /lib/udev/rules.d/51-android.rules so the adb daemon can find my phone when I ask adb "what phones are plugged in". And THAT is why I can't use the vesion from AOSP and need to use the version from Debian, unless I'm copying random bits of AOSP checkout du jour into my host system's config files by hand...

Anyway, the EASY way to copy files onto my phone was to just throw them into a directory on my website, download them one at a time through the browser, and then create a new directory with the "Files" app and move them to it. Tedious, but straightforwardish.

Oh, and Android has a passable "Music Player" app. It has a bunch of things that cause dropouts in the playback (such as turning the screen on and off), and one of my aac files drives it NUTS to the point it plays random tones and then hangs and pops up the "would you like to report this" window. And for some reason Rock Sugar's Reimaginator album (an ogg file) always has to be played twice. The first time exits immediately, the second plays fine. *shrug*

Between that, audiobooks, podcasts, and the four video streaming services I already PAY for (netflix, hulu, crunchycrunch, and prime), I'm getting over the loss of Youtube. (And no, I haven't reinstalled the Google Maps app either. I pull up the web version a couple times a month, but only when absolutely necessary. When a service goes beyond the Advertising Event Horizon, I wander off. I STILL can't get Google Maps to show me the Great Clips in Hancock Center. I zoom ALL THE WAY IN and it's not there, unless I search for it by name in which case suddenly there's a pin that tells me the phone number and hours they're open. Because they didn't pay Google for the privilege of being listed. Why would I use a service that's not going to show me places that haven't paid to be listed? Applying the "relevance" filter to local geography shows the bias quite clearly, and they won't let me opt out of the filtering because money.)

June 4, 2022

June 3, 2022

June 2, 2022

June 1, 2022

May 31, 2022

May 30, 2022

May 29, 2022

[Still sick. Pascal's apology for this blog entry, stream of consciousness is a failure mode when I haven't got the focus to filter.]

May 28, 2022

May 27, 2022

Money arrived! In the actual bank account where Fade can see it! I'm sort of collapsing into a heap with relief, although still being sick is a contributing factor. (Not AS sick, but a nonzero amount of sick. Which was in the "not sleeping well" phase last night, and in the "any exertion that would raise my breathing rate turns straight into nausea" stage today, but other than that I feel pretty decent except for the coughing and sweating.)

Sigh. I let the youtube app upgrade on my phone (I KNEW better, never let the google apps update or the UI randomly changes in bad ways and loses non-paid capabilities or inserts ads) and now the "drag the slider to the end of the video and hit replay" trick to kill the midroll ads no longer works. I don't mind ads before the video and after the video (I don't _watch_ them, but that's true of all ads) but randomly stopping the video to replay a completely unrelated video is unusable levels of malfunction, which I've always treated as "the video suddenly ended, youtube doesn't have the rest, such a shame". TV shows were _created_ to have ad breaks, the algorithmic insertion sticks them in the middle of a sentence, partway through songs (this cord change is brought to you by some random capitalist thing I would actively refuse to buy if I was paying enough attention to remember who they were instead of treating it as a pure malfunction; the phone ringing with "Scam Likely" and thus pausing the video contains exactly as much information).

So my phone no longer has working youtube support. Oh well. I need to set ADB back up and copy mp3 files into the thing again.

Walked to the table last night (very slowly) and managed to get a little work done, or possibly it counts as research? I'm trying to run scripts/test.sh under toysh, and it barfs on the ;; on line 221. The REASON is it sees that the last block is type 3, which doesn't logically terminate with ";;". That's because there's an implicit terminator, notice how these two lines work the same:

$ a=X; case $a in X) echo $a ;; esac
X
$ a=X; case $a in X) echo $a ; ;; esac
X

I need to teach it this bit of the parsing to not just look at the last block group but dig backwards through symmetrical block groups to go "ok, what block am I really terminating?", and I think I made a function for that already? (It's not hard, but it is one of those for loops requiring its own temporary variable that's easier to make a function out of just for the varible.)

But the bigger issue is I'm stale on this code and need to re-absorb the context of which block group means what, and I really SHOULD have documented all this somewhere but remember that it all still in flux last time I tried?

Ok, so the general toysh data flow is that something calls do_source() on a FILE * (having already created at least one TT.ff function context), which calls get_next_line() in a loop (display prompt and do command history/editing as necessary, return a malloc()ed string), and then passes that to parse_line() which populates a linked list of struct sh_pipeline (its second argument) and returns 0 if it finished a complete thought, 1 if it needs another line of input to finish the thought (that $PS1 ">" line continuation prompt), or -1 if there was a syntax error and the caller should just discard everything up to that point. When it returns 0, the caller can call run_lines() to run the parsed lines (the struct sh_pipeline list), and afterwards free the sh_pipeline list. Which do_source() hits EOF on the FILE *, it returns. (When you need to run an arbitrary string via -c or eval or something, fmemopen() can give you a FILE *.)

The third agument to parse_line() is the "expect stack", which is a standard struct double_list of block terminators. It's temporary storage and when the list is empty we're not in a flow control block. (That's not the only reason we may need to request more line(s) of input, an unterminated "quoted string" or trailing || doesn't require us to be in a flow control block, but we're can't start running the next thing until we're NOT in a flow control block either.

Each sh_pipeline struct has an integer "type" field, which indicates flow control block type. The main block types are:

• 0) normal executable pipeline statement (run as a command)
• 1) start of block (if/while)
• 2) "gearshift" (then/do) switching from the head of the block to the body of the block
• 3) end of block (fi/done)

Some blocks, such as { } or ( ) have type 1 and 3, but no gearshift, but every type 3 has a corresponding type 1 and vice versa. The "end" field in sh_pipeline points from type 1 (and type 2) to its type 3. (At runtime there's a "struct sh_blockstack" with start and middle pointers letting us get back to our type 1 and type 2 blocks, but since those go past as we're evaluating them we don't need to record them in the pipeline structure.)

Parsing "if true; then echo hello; fi > potato" gives us a type 1 "if" block, a type 0 "true" block, a type 2 "then" block, a type 0 "echo hello;" block, and a type 3 "fi" block (with the > potato redirect attaching to the type 3 block). Running that pipeline means hit the type 1 if, allocate a new struct sh_blockstack pointing to this if statement's pipeline segment and push it onto the stack, check its type 3 (using the end pointer) for redirects (and save THEM in the sh_blockstack to undo when we pop it), and advance to the type 0 pipeline segment to run the test. Keep going through the test code until we hit our type 2 gearshift (there could be multiple statements separated by && or something, thus multiple type 0 or even nested "if if true; then true; fi; then echo ha; fi" -- anywhere a type 0 can go can also have nested flow control blocks, that's what the "struct sh_blockstack" tracks. We know when we've hit OUR type 2 by looking at the blockstack.) Then since this type 2 goes with an "if" statement, evaluate the return code of our test to see whether or not we should run the body (jump to the type 0 "echo hello;" segment after the type 2 "then"). If not, jump straight to the "end" segment (using the if segment's end pointer, which our sh_blockstack points to). Evaluating the end pops the sh_blockstack and runs its cleanup (freeing memory and undoing any redirects).

Loops add some jumps back ot the start and instead of a type 0 block between type 1 and type 2, they can have a type 's' block which is a list of strings instead of a command (for i in one two three; do echo $i; done). It's still a pipeline segement with the strings held in an argv list, and yes it has to terminate with ; instead of || or | because it's a syntax error otherwise.

Functions add type 'f' blocks right before a type 1. For some reason functions in bash must be followed by a compound statement, but you can have abc() if true; then echo "$@"; fi and they're fine with it?) When a function finishes parsing it can convert into a type 'F' block where the function gets exported into the function name table and the whole span of struct sh_pipeline segments after it gets moved into the type 'F' sh_pipeline segment's first argv pointer. Why "first"? because the argv in sh_pipeline is an array of arrays: its first entry is the command line arguments and its second and later entries are the HERE documents fed into this command. A struct can only cleanly have one variable length array in it, at the very end: if you malloc() extra space then indexing off the end of that array uses the extra space. So it reallocs() and resizes as necessary to handle an unlimited number of HERE documents. Because you can have multiple HERE documents read in sequence, and HERE documents redirected to diffferent filehandles, and so on:

$ echo one <<EOF two three
> ha
> EOF
one two three

That example didn't USE the substituted input, but it did supply command line arguments after starting the HERE document. Yes you can attach HERE documents to function definitions, and you need to be able to rewind and replay them each time you run the function:

$ a() { cat; } <<< potatosalad
$ a
potatosalad
$ a
potatosalad

I honestly don't remember when a function definition is and isn't exported, although I vaguely recall it's one of those scope things like local variables and I have a pending bug report about popping the wrong thing when a function definition goes out of block scope. MY headache left over from poking at that was working out the lifetime rules for the string allocations in the pipeline segments. (These pipeline segments need to survive past do_source() returning. Functions are maybe 2/3 of the way implemented?)

The next bug in the test suite is that pipeline segments that should be run in their own process, I.E. implicit ( subshell ), aren't being and thus the parent shell hangs generating an infinite amount of data that it would then infinitely read if it proceeded to the next pipeline segment (and then the whole mess fed to "head" which chops out the part it wants and lets the rest know to exit by closing the filehandle so their output errors when they write to it). Except you can't just subshell EVERYTHING because what (if any) of it should be able to work in the parent's context so variable assignments and cd and stuff persist? (Of COURSE posix doesn't say.)

And then there's case/esac, which was added after the rest (including after functions), and has introduces multiple gearshifts (type 2) within the same block. Quite possibly those shouldn't be type 2. This is the part I need to wrap my head around now to fix the next bug that's in front of me to get the test suite to run under toysh, and this is also the general area that's going wrong trying to run debian's /usr/bin/ldd script.

I still have the cold, I'm just tired of waiting until I can think clearly to go back through this stuff. To be honest I've had covid brain for months, and other stress-related problems before that (my startup collapsing during the 2016 election was... not good). I don't get to handle this by being "smart enough", I have to be pesistent with duct tape and nail file until the result looks acceptable to me dorodango style.

Even more sick. All the sore throat and coughing and unfocusedness. Bug reports and fixes are going by on github and I'm trying to keep up, but I do not have a lot of spare brain right now.

I figured out what's going wrong with the darn ctrl-c issue in toysh though: it's waking up multiple instances of the shell which are then taking turns reading from the command line. I chroot in with /bin/sh, then run /init manually which pauses that first shell and gives me a new shell (conceptually a child of the "pid 1" that ran it), but the new shell does NOT take over the tty and create a new session ID and such (calls to setsid() and tcsetpgrp()) which means the ctrl-c is sent to the original chroot shell as well, and it wakes up and starts reading from the tty, but the child shell ALSO caught the signal and didn't exit, so they're both reading from the terminal meaning they grab alternating lines, and that's how $PWD and getcwd() look like they get out of sync because one shell eats the "cd subdir" and the next eats the "cd .." and then when I query $PWD in one and getcwd() in the next I get different answers...

I am not wording well today. I hope that last paragraph was coherent.

Anyway, the fix is to implement proper terminal control stuff. Which I think I have partway done in one of these fork directories already. I did a lot of stuff on job control that needs finishing and integrating...

Bad things hapened yesterday. I'm trying to keep this blog mostly technical these days, but Uvalde is something like 150 miles from Austin: head to San Antonio and continue west when you get there for about as far again. The gun nuttery, and cops being useless drama queens with extra useless (and even actively harmful and lying about it, and yes Uvalde spends 40% of its budget on police, which have no legal obligation to actually do anything), and general refusal to lift a finger against white men no matter what they're doing, is common to both places. Austin may be the dot of yin in the giant yang that is Texas, but that just makes it slightly less concentrated here. We're all just waiting for the last Boomer to die.

Still sick. Fuzzy had this for a week. Wheee.

I've reproduced the toysh confusion where "ls -l /proc/$$/cwd" and "echo $PWD" disagree, and it definitely seems something to do with interrupting a process via ctrl-C. The shell is in a slightly unhappy state after that. Unfortunately, as a reproduction sequence goes "run something that takes 3 seconds and ctrl-c it partway through" is a bit... squishy. Not exactly easy to add to the regression test suite even if I do work out what's going on, and not exactly easy to run exactly the same test multiple times in a row with slightly different printf()s.

But it does NOT happen if the process is allowed to run to completion, only when it's interrupted. Hmmm...

Yay! The middleman said the domestic swift transaction has finally errored out, and the whole thing should unwind over the next few days. Eventually the transaction in the middleman's website should go from "paid" to "error", then I get to resubmit it with fixed bank info, and then maybe actually see money 3 business days after that? That would be nice.

Various bug reports today highlighting the lack of mkroot-based testing, and I'm trying to poke at that (despite being sick and really out of it), but I've hit a WEIRD bug where $PWD and /proc/self/cwd do not match. When I run a child process, it's using $PWD. When I run a builtin, it's using cwd. Except it's weirder than that, because "ls" and "ls dirname" are behaving differently and they should be the same builtin-vs-child state?

This is the standard "freeze, do not step on the broken glass, examine the crime scene" problem where I have to STOP AND FIGURE THIS OUT rather than lose the test case, except slightly creepier because it LOOKS like filesystem corruption, but it's in a chroot not a vm, which means it would be HOST filesystem corruption. (I have a full backup from earlier this month, but still.) Nothing in dmesg. 99% likely it's my code and not the host code, but... WEIRD.

Alas I don't currently have the focus to mentally model anything complicated right now. Trying, but... (Youtube used to have a clip from the Phantom Tollbooth where the Terrible Trivium convinced the Humbug to dig a hole through a cliff with a needle, but of course late stage capitalism has weaponized intellectual property law so short clips from a largely forgotten 50 year old movie must be torn down.)

I was waiting for "minimum electrical bills" to punish rooftop solar panels undermining the fossil fuel industry. That's why people in Austin have been speccing enough batteries to unplug from the grid entirely. (Yes, downtown. The Texas electrical grid came very close to collapsing again a couple weeks ago, and it's spring. Enron was the energy trading arm of Houston Light and Power, the whole "deregulation, free market" libertarian nonsense poisoning the infrastructure here has rendered it basically unsalvageable, and people are opting out.)

And gmail unsubscrubed almost 1/3 of the list again. (Including me this time, although all I had to do for that was open the confirmation link it emailed me. At least gmail's feral spam filter didn't eat the confirmation link. This time. It really does NOT like my replies to scsijon. Dunno why. And yes, it unsubscribed me by refusing delivery of one of those emails it would then put in "All Mail" but not in "Inbox" where pop3 could fetch it so I wouldn't get broken threads in the toybox folder. Gmail has achieved nested failures.) On the bright side, Google has apparently backed off on the forced migration to a pay service. Still haven't got the domain admin login, but it gives me a few more weeks to deal with the migration.

Checked in tar --xform, which left me with various loose ends: 1) --long-name-truncation (ambiguous, possibilities) and 2) dirtree without eating filehandles (and rm -rf infinite depth test), 3) testing under mkroot (mount had tortoise() swapped for xrunread() but no regression tests because root, df test failed on kvm ubuntu image because unionfs returned 0 blocks used). I've got a bunch more: The pending readlink relative path stuff (also hits cp -s), Elliott's todo list... heck, I should read the last couple years of blog and collate todo items mentioned there. (And collate my various todo.txt files.)

Poking at the --longopt-truncation stuff now because while implementing tar --xform I could never remember if it was tar --show-transform or tar --show-transformed and realized I'd been using both, and according to the man page it's actually tar -show-transformed-names which is being automatically recognized as any unambiguous subset:

$ tar --show
tar: option '--show' is ambiguous; possibilities: '--show-defaults' '--show-snapshot-field-ranges' '--show-omitted-dirs' '--show-transformed-names' '--show-stored-names'

This is easy enough for me to implement in lib/args.c, but the question is WHEN should it do this? What, new leading punctuation type to say longopts are abbreviatable? I could easily make EVERYTHING auto-abbreviate, which would encourage scripts to abbreviate longopts in a way that might break with other implementations, and thus break compatbility with the gnu/stuff if it DOESN'T do that. Hmmm...

"sed p --null" works (it's --null-data), base64 --de works (it's --decode), cat --squeeze works (--squeeze-blank, ala -s)... Ok, it's looking like Debian handles any unambigous truncation of --longopts by default. So lib/args.c should just do that for any --longopt.

Debugging the tar --xform stuff, it looked like the new xrunread() was leaking filehandles, but it just turns out that was dirtree recursing down directories and keeping one filehandle open per level. (Which I have a todo item to deal with already, needed for rm -r to handle infinite depth but I'm never comfortable TESTING weird rm -r corner cases so that's in the "do it when I've got mkroot running the test suite" bucket.)

Hmmm, maybe I don't need to spawn a sed instance per filename for tar --xform? If I added sed --tarhack I could force NULL terminator mode and prefix each input with a type (for s///rRsShH), and then on output filter out any interstitial NULL bytes (from somebody being tricksy with \0).

The problem is toybox tries never to DEPEND on calling toybox commands out of the $PATH. Toybox tar should work with debian's sed or gzip, toybox mount should work with debian's losetup, etc. But if I _don't_ depend on being able to modify sed, how do I get it to handle the input type filters at all? I'd have to implement a parallel tar command parser to find the "r" in "/abc/,2{y/s/q/;s/a[s/q]/\1/gr;}"? (Almost certainly that's what debian's tar DID. Having five implementations of the same functionality across your command suite is very gnu.)

Another toybox plumbing rule is that commands shouldn't call other commands' functions directly, because that way lies build breaks. The shared infrastructure lives in lib, or sh.c has NOFORK plumbing (but again, the commands that aren't in the same command.c file drop out of the list cleanly when deselected, which a call to command_main() would not. It's a design layering violation.) I don't want to make sed a nofork because it doesn't guarantee that all its memory is freed and its filehandles closed when a random error_exit() calls xexit(). The nofork plumbing can intercept that to longjmp() back to the caller, but unless the command is sanitized for nofork the error paths will leak resources. And for most of them the normal exit path leaks because the OS cleans all that up for us anyway so calling close() and free() right before main returns is extra code accomplishing nothing.

The "clean" way to do it is just call xexec() to run the other command, and if we can recurse we do so, and when we've hit our recursion stack limit we fork and exec out of the $PATH. (And the Android guys enabled TOYBOX_NORECURSE in the .config for some reason, so it NEVER does that for them and always forks and calls whatever's in the $PATH. I don't know why, by haven't asked.) The standalone builds always call out of the $PATH unless you build the other command in as a NOFORK (which is currently special cased for toysh and nobody else).

I posted my ruminations to the list about what exactly is tar --xform actually DOING with the s///S and such indicators, and the answer is apparently "not much". That's just sad. Hopefully I can just NOT IMPLEMENT THEM and we'll be ok? Let's see who complains...

Sigh, the new xrunread() function replaces tortoise() in mount.c that shelled out to a command and returned its output, and that A) could output an error message with the command's return code (although the command itself presumably already complained to stderr, or else the failure to exec was spat to stderr by the forked child process), B) read into toybuf so didn't have to free stuff. Which meant it was length limited, not that losetup -f is going to return over 4k of "/dev/loop1" string. But the new thing is returning a malloced block of memory, which means it needs to free it to avoid leaking, and... I really didn't want to bother because it's a tiny ammount of memory and how long is the world's longest /etc/fstab anyway? (How many times can this function be called in one run?) But I felt the need to write a comment, then rephrased that comment so many times I gave up and added strlen(s)>=sizeof(toybuf) to the error_exit() case and then strcopied it to toybuf and freed the returned string. (It wasn't WRONG, just... untidy.)

Another very SLIGHT rough edge that calling out to sed will add a newline to the end of the output and my chomp() implementation strips as many trailing /r /n as the file has. If they sed a file to end with multiple /r/n then MAYBE the gnu version wouldn't strip more than one (I haven't checked). I could fix this by calling sed -z, but the tiny possibility of maybe calling the macos sed that hasn't got -z (I dunno?) balanced vs the tiny possibility that somebody would try to create filenames ending in newline/linefeed characters...

Ha! Wait, my sed implementation WON'T add a newline if its input didn't end with a newline. So I don't have to care about that, and neither does the gnu/dammit version... but once again, I have no idea what mac would do, so I should probably chomp() it anyway? Grrr. Now I'm not sure what the right thing to do is here, the common case is two sed implementations (no, three: I wrote the busybox one back in the day and it gets this right also) that DON'T need the input mangled with rules of thumb (not that utf-8 is ever going to care, but who knows what weird binary crap people have in filenames?) vs the tiny possibility someone will dig up an actually nominally posix compliant sed from the 1980s that adds a newline to the last line when there wasn't previously one there, and thus transformed names all have newlines appended to the filename.

It would be easier if my brain felt like coming up with things I could actually TEST today, but that's not the kind of day I'm having. (Speaking of which: tar --exclude is checked before tar --xform, and xform happens before tar --strip. Trying to figure out if I should add that to the tests because normally my policy is to add every test I needed to run when implementing, but if the order changes in future does TEST_HOST failing really MATTER? Sigh. Does anybody use these in combination? I'm not implementing the file type limiters in the sed expression, and what I AM doing is passing through to sed so you can use full sed syntax (multiple of each if you like, it'll do "sed -a abc -e def -e ghi" in order). Already deviating a bit, no standard, I matched the behavior and added comments to the source, and WHAT I'm testing is a weird corner case...)

Sigh: why is tar --strip only on extract? Is there a reason? ("tar c --xform" is done after stripping leading ../ and once again I CAN add a test but am unsure if I SHOULD...)

Garrett Kajmowicz (coworker of mine back at Timesys, who once upon a time wrote uClibc++) is visiting. He's taking advantage of work-from-home to go on a road trip. We had a cookout last night. He's asking me to "show him Austin" and I honestly haven't been out into it much recently? Well, lots of walking around late at night but not while it's open. He's not gonna go downtown to visit a courthouse or the state LLC issuer or the DMV. We have a bunch of really good restaurants... most of which are national chains? Various places to shop, but I'm not big into stuff. Took him to Rudy's. It's still good, but suffered a bit during the pandemic. We should hit the Alamo Drafthouse I guess? Except they're not doing anything interesting until this weekend, and he's only here a couple days midweek. Sample problem with the conference center and theatres and music venues and stuff... all that stuff requires you to camp the spawn. (The lesson here is I'm not really a good guide for this sort of thing.)

Here's a good example of how ontogeny recapitualtes phylogeny within Fortune 500 corporations. "I want simple." "Sure, I'll add that on top of the pile." (Every new C++ release wraps another layer of simplification around what's already there.) Completed debugged code is an asset with a dollar value attached, therefore deleting it costs the company money. After the people who did it left, reverse engineering something that already works is a waste of salary hours: everything becomes a black box you must wrap your changes around, encapsulating to form a pearl around the speck of random trash at the center. (Of course that's not how simple works. But it is how yocto works. Rather than cleaning the floor, install a new floor over it.)

Poking at tar --xform raises some funky sequencing issues: do you filter (--exclude) on the transformed name or the original name? And does --strip happen before or after --xform? I should add tests, but right now I'm just implementing SOMETHING and I can tune it later...

Sigh, and I have a TODO note that tar implements --exclude but not --include and that's just silly? (Well, I guess you use find for that, but you can use find with grep -v too? If you already did most of the work, why NOT wire it up? Sigh...)

Grumble grumble one of my sleep deprivation variable names slipped through (int ala = mode;) and it's a bit intrusive to change it...

Grrr... I think I have to spawn an instance of "sed" for each filename. I was thinking I could use -z for null delimiters, but what if s///p produces two output lines (thus getting us out of sync even with NULL delimiters)? And if I'm not strongly policing what sed -e they send (which I'd rather not, why can't they y/// if they want to? Or a whole sed script with however much complexity they need to get the job done?) then there's no guarantee it's gonna produce ANY output for a given input line. (Which I assume translates to "skip this file".) I can delimit the "this input produced this output, possibly none" by closing the input pipe and waiting for the output pipe to close because process lifetime. This spins the PID but eh, shell scripts do that all the time. (Making sed run as a nofork could be done, but isn't trivial. And so far all the nofork plumbing lives in sh.c, moving THAT isn't trivial either, although I need to do some subset of it to properly integrate Moritz Weber's upcoming git implementation..)

Of course sed itself can't parse the s///rRsShH filters on file type. (Regular, symlink, hardlink... why no dD for directory? Is the filter on the whole path or just the last member name? Because if you apply a filter to the path down to a filename but not to the directory entry that created that directory, it's not gonna match up right. (And so I need to run MORE tests against debian's tar to see what it does in order to work out a spec to implement...)

Dear bash: why does a trailing || pull in the next line, but a trailing | does not? (Yes I know what the standard says, but WHY?)

Still ripping the generated/flags.h plumbing a new one. What I'd LIKE to do is turn something like the "mkfifo" command's optstring "<1"USE_MKFIFO_Z("Z:")"m:" into:

#define FLAG_Z ((USE_MKFIFO_Z(1)+FORCED_FLAG)<<1)
#define FLAG_m (1<<0)

And then generated/flags.h doesn't need to be regenerated when .config changes.

I also kind of want it to be generated via bash script instead of C (tricky), and I also want to collapse together the two FORCED_FLAG macros and just have it always be 1LL. Unfortunately, it turns out gcc's optimizer does "long long" math in 64 bit mode and then truncates the result to 32 bits even when it CAN'T MATTER, ala:

#include 
#include 
int main(int argc, char *argv[])
{
  unsigned long ul = 1LL<<atoi(argv[1]);
  printf("%lu\n", ul);
}

Compiling that with 1LL vs 1 before the shift and then diff -u <(objdump -d one) <(objdump -d two) says that it produces different code, even though the assignment to ul will truncate at 32 bits so the left shift might as well be 32 bits, the conversion-to-int can move one step up in the math because shift left can't have the bottom 32 bits influenced by anything in the top 32 bits. Even on 64 bit platforms it's an extra instruction (because then the truncate becomes explicit). I'm guessing on 32 bit platforms it's calling the library functions to do 64 bit FOIL algebra instead of the simple 32 bit math it has CPU instructions for.

Sigh, that's why I had the two macros in the first place (and switched to using the big one only when it was shifted by enough to matter). Hmmm. And with llvm it's an even bigger difference between the two assembly outputs. Sigh, you'd think with all the work they put into the optimizer, gcc or llvm would pick this up, but apparently not...

Right now, mkflags.c takes two different optstr inputs, one for allyesconfig and one for the current config; it resolved the macros with gcc -E but did it twice and then diffs the strings. If I switch to preserving the USE_X() macro inside the FLAG_X(macro) then it only needs the one (unpreprocessed) input, which more or less means rewriting mkflags.c, and if I'm going to rewrite it anyway I might as well do it as shell instead of C. Except C string syntax is kind of nontrivial, ala "abc\"USE_"USE_POTATO("abc")"def" and yes that is an escaped literal quote and USE_ is legitimate optstr payload although I think I can guarantee it'll never actually occur in that order in a string... I mean it COULD show up in a longopt, but I'm willing to bet it won't and fix it up if it does?

Anyway, trying to process this in bash turns out to be kind of unpleasant. I made a sed invocation to give me the command name and option string for each NEWTOY:

sed -ne ':a;s/^USE_[^(]*(\(.*\))$/\1/;ta;/^NEWTOY(/!d;s/NEWTOY(\([^,]*\),/\1/' \
    -e 's/, *[^"]*) *$//;/"/p' toys/*/*.c | while read NAME ARGS; do
  echo NAME="$NAME"
  echo ARGS="$ARGS"
done

But taking the result and emitting flag macros for it... thats easier to do in C than in bash? Which means I'm writing a new C scripts/mkflags.c which is not ideal, but still... could be worse?

Except... The reason I have the low-ascii skip handling in lib/args.c is so when it hasn't got -Z the parser will error on -Z instead of silently accepting but ignoring it. Even if I chop the optstr contents out with macros instead of replacing them with spacers, the flag positions change so the FLAG macros have to be regenerated, which means rewriting generated/flags.h anyway.

Sigh, I WANT to simplify this infrastructure, but it's all got reasons for what it's doing. Needs more design thought.

In the meantime, I started down this path because tar --transform and --xform were synonyms without a short option to collate them (which matters because they take arguments and put them in two different buckets which don't maintain order relative to each other), and I should implement a solution to that in the current plumbing if this isn't a good near-term excuse to redo said plumbing.

Gmail is back on its bullshit. So many "bounce action notifications", refusing delivery of a message gets retried by dreamhost until the number of attempts exceeds the unsubscribe threshold. I can't get an actual human at either side to fix the design assumption conflict (can you just discard the one MESSAGE and not unsubscribe the USER), just like I can't get gmail and thunderbird to talk to each other about their conflicting imap design assumptions (both the "All Mail" vs "Inbox" folder problem that screws up threading in mailing lists, and the problem "you can only permanently delete out of the 'Trash" folder but thunderbird implements imap "move mail" as "copy" then "delete" so trying to move messages to the trash before deleting them DOESN'T HELP.)

I went through and fixed all the mailing list accounts up by hand in the web interface again. Something like 1/4 of the list users are gmail. Nobody else got kicked into "hold:B" state, just gmail, as usual.

And I still need to move OWN email off gmail by the end of the month due to gmail's forced migration to a pay service of my old account type (which I don't have the admin login info for and thus couldn't migrate anyway even if I DID want to pay for it, which I don't). But I still can't start the migration yet because I want this "getting paid by the middleman" thing sorted out first.

Speaking of which, the middleman emailed me this morning with the reply they got from their outsourced payment processor:

So it looks like we need to ensure transaction is rejected at the bank and the money is refunded first before we can proceed. It should normally take a few business days but I hope to have all of this sorted for you this week.

Please let me know if you have any questions.

Oh do I have questions, most of which I didn't ask. I DID ask if "sorted out this week" involved just reversing the first transaction, or if actually getting money into MY account might happen this week, because of the whole "email needs to change ISPs at the end of the month" and me just assuming it will go as wrong as this because it often does for me. I break everything. It SHOULD take maybe 24 hours for the DNS to update. I EXPECT to be unable to receive email at landley.net through maybe the end of June? No idea.

Questions I DIDN'T ask include whether they could elaborate on the word "ensure": is there a strong possibility of the money NOT being refunded? If the money went into a random account that didn't have my name on it and somebody got dollar signs in their eyes and absconded with it, is there some sort of insurance here? (I have zero visibility into this process. I've heard of large windfalls winding up in people's accounts due to bank error and them getting in trouble for spending it, but they still spent it.)

And of course, is the transaction being rejected because we asked them to reverse it, or because the transaction that "successfully completed" couldn't have successfully completed, both because you can't do a domestic transfer via swift and because there's no destination account with that name/number pair at that bank? (The name "Landley" was made up during World War I because a german immigrant named "Landecker" wanted a less german-sounding name fleeing a war to the side that WOULDN'T draft him; there's not a lot of us. Google says a few people not related to me have taken it as a stage name, usually as their first name.) If "rejected" means rejected rather than cancelled, then how did it "complete successfully" first? What was the three day waiting period for if "destination account exists" was not part of the success criteria?

I had more questions, which I decided to edit out of the email reply as "not helpful", so I saved the longer version as a draft in thunderbird, edited it down, sent it, and watched thunderbird delete from the drafts folder, without backup or undo option. Thank you thunderbird. You wrote a copy of the short version in the sent mail folder, but completely deleted the long version I explicitly saved. Bra fscking vo.

Yes, this is a normal weekday for me, why do you ask? I break everything, all the time. Nothing is foolproof because fools are so ingenious, and whatever soup of neuroatypicality I have combines with knowing just enough to be dangerous to near Newton Pulsifer levels at times. I am trying very hard not to have to migrate my email until the Q2 bank deposit is sorted, meaning I'm on a deadline. Wheee.

And yes, I make an EXCELLENT TESTER because of this. Doing it for other people hasn't been how I prefer to spend my time, in part because Murphy's Law only fails when you try to demonstrate it so RELYING on being able to break things seems like it'd break, but if you wonder why the software I make is like it is, this is what works for me. Achieving basic functionality (not gonna say reliability) by process of elimination. The Linux USB maintainers do not use dodgy cables nearly often enough, or they'd have fixed whatever locking problem causes ports get tied up by a device that's been unplugged. (My laptop has 3 of them, but I've run out before. Luckily software suspend unloads and reloads the USB subsystem as part of its power management shenanigans, which seems to fix the issue.)

People worry about fork bombs, I worry about "tr '\0' a /dev/zero | sort" or "while true; do mkdir a; cd a; done". (Has anyone ever actually TESTED inode exhaustion in most modern filesystems? Running out of SPACE, sure, but ext4 has fixed-size inode tables, if you touch a zillion files but leave them zero length, you run out of inodes without running out of disk space. What do OTHER filesystems do, and when were those error paths really last exercised?)

I have so much practice at debugging because "how did I break it THIS time?" is a question other people have consistently been unable to answer. Still dunno what was wrong with my phone's volume down button. It's happy again now. Could go out again at any time. Same as it ever was...

Last time I sat down to do "tar --transform" the problem was "tar --xform" is a synonym, but there's no short option. Short options can have as many --longopts as you want as synonyms, and they all get collated which means when they can collect a single * argument list and all goes into the same variable. So if "tar -Z" was a synonym the optstr "Z(transform)(xform)*" would let the command line "tar --transform s/abc/def/ --xform s/ghi/jkl/ --transform s/mno/pqr/" put all three in the same list in the original order. But without a short option to collate them under, leading "bare" longopts don't collate, so each would collect into its own arg_list and processing them after the fact would have lost which order they arrived in relative to each other.

It's not a BIG flaw, and this problem has come up before: sed doesn't remember what order it saw -e vs -f in when both are on the same command line. A certain amount of detail being lost is inherent to having generic code do the option parsing before calling main(): the neatly collated data can miss some subtleties of the original. But this time it bothered me enough I shelved it, and now I want to fiddle with the option parsing infrastucture to provide a way to at least collate bare longopts.

Except... $HOSTCC scripts/mkflags.c running on the host to generate a header file at all is a bit of a wart in the design. MOST of generated/*.h is created with a shell script calling the existing toybox commands. I've slowly been deprecating scripts/config2help.c so maybe at some point I could yank it (possibly replacing some of what it does with USE_COMMAND() macros in the help text?) and it really SEEMS like what mkflags.c does can be done with bash/sed. Especially if the result was more like:

#define FLAG_q ((USE_THINGY(1LL)|FORCED_FLAGS)<<42)

I.E. when a string snippet is in a macro (INSIDE the string, I don't care about the guard macro around the NEWTOY() line here), I could just put that macro in the flag definition, and this avoids the whole dance where I curently run the arguments through the cc -E preprocessor twice to produce different option strings and then have mkflags.c compare them. (Which is a BIG part of the reason the headers have to be regenerated whenever the config changes!) It can be done in ONE pass through the data, and the result would be the same no matter what .config said.

Except flag macros can theoretically nest, and teaching sed to push and pop them... would not be pretty. Hmmm.

ANYWAY, what I should do is add an "invisible short option" that lets me collate longopts but doesn't result in a new short option, and teach the existing mkflags.c to handle that. What I DID was a fairly extensive cleanup pass on the build plumbing that doesn't address this at all, but makes the output a little more readable and makes explaining it in a video easier. :)

If a Pixel 3a's "volume down" button sticks, the phone is completely bricked. Specifically, when you reboot it, the bootloader menu comes up and then can't be dismissed while the "down" button is held, so it will never reboot back into android again. (And you can't power it OFF either, holding down the power button only works when it's not also holding volume down. I have to wait for the battery to die.)

The phone repair place in Hancock center didn't survive the pandemic, neither did the one that used to be in Fiesta across the street. The closest phone repair place I can find is the "1up" phone repair place on Guadalupe at UT (which I've walked past late at night and it specifically says they repair Android devices on the window). Dug around to find a paper book to read on the bus. (No bluetooth headphones. No podcasts or audiobooks. No e-books. No phone games...)

The 1up phone repair said a Pixel 3a is unserviceable because the case is designed not to come apart (and wouldn't go back together again if it did), so even if it doesn't need a replacement part (which they DEFINITELY couldn't get), they can't get it open to service anything inside without destroying the phone. A stuck button is the most trivial POSSIBLE hardware problem (other than maybe the usb port filling up with pocket lint, which you can fix with a needle or a sim card ejector tool), but the phone is a brick now. Because of the contact switch under a button.

The thing is, I bought this phone through Google instead of tmobile because I specifically wanted an unlockable one. (Even though the bootloader says I never actually bothered to unlock it.) And mail ordering another one would take a week. I could presumably pop the sim card out and put it in an old phone, but I stopped using my Nexus 5 because it got wet (and was never quite happy afterwards), and besides it was 32 bit so stopped getting security updates. I expected THIS phone to become my backup phone someday, and possibly be a thing I'd unlock and reimage and test software on...

Really not impressed with Google's hardware policy right now. I've had this thing for less than 3 years but it is now completely unserviceable. (I haven't even scratched the display, the problem is OBVIOUSLY TRIVIAL, but no. It's a brick. Thanks Google.)

Sigh, what was that cheap Motorola phone Louis Rossman liked? (Can't google it right now because I'm still out at UT and have no phone tethering because brick. Ah, UT guest wifi. It's the moto g. Which is $190 new and has an sd card slot, something the pixel did not. Its processor is cortex-a53, 64 bit, 8 cores maxing out at 2.3 ghz. In theory that's faster than my laptop. Only 4 gigs ram instead of 16, of course, but THIS phone accepts a microsd card and I've got a half-terrabyte one of those somewhere...) Still multiple days to mail-order it, but there's a T-mobile store in Hancock center. Back to the bus...

HA! The guy at the t-mobile place was able to unstick the button! Even he's not quite sure how he did it (I took it out of the case and pressed, scraped with the SIM ejector pin, and whacked it against tables and such for like an hour) but it's working again! For the moment, anyway. Limp by until money comes in again, at least.

As Sam Vimes said, you do the job that is in front of you. Right, back on the horse.

Finished editing and uploading the second video (first was how to download and use binaries, second is how to build from source, including basic cross compiling if you have cross compilers lying around). I need to start fiddling with playlists...

The dropbear problem is because dropbear's autoconf is stupid. Zlib built for arm64 just fine, but then dropbear did:

checking for aarch64-linux-musleabi-gcc... no
checking for gcc... gcc

Because IT'S NOT -GCC, it's -CC. So dropbear has the same hardwired headache Linux does if you don't hit it with a rock. (Why did the absolute path version bypass that but the search-in-path version didn't? Autoconf! Why is it NATIVELY COMPILING WHEN IT CAN'T FIND THE CROSS COMPILER DESPITE THAT NOT BEING WHAT IT WAS ASKED TO DO? Because gnu/autoconf has "gnu" in it, so it doesn't care what it was ASKED to do, it thinks it knows better than any unwashed masses outside its cathedral (it was the original one) and will impose its will upon them for their own good, "this hurts me more than it hurts you", "look what you made me do", etc. Of course there's a middle aged white man at the helm, you can't be properly patronizing without "pater", the latin word for "father", as in The Fatherland. And of course and OF COURSE. Why did Matt decide to use autoconf? Sigh. I object to the design philosophy behind this tool. Oh well, usual whack-a-mole solution...)

And moving the package builds before the toybox build (because the new scripts/root/dynamic needs to run before toybox builds if toybox is to be dynamically linked on the target) revealed the bug that my dropbear build script isn't returning to the top level menu. (The package builds are sourced, not run as child processes, because dropbear has to set an environment variable that writes extra stuff into the qemu-*.sh launch script. But that ALSO means each build script has to clean up after itself for things like $PWD. The problem in this case is that dropbear builds two packages, zlib and dropbear, and the lifetime of those package builds overlap because the zlib build creates a static target library dropbear links against but which is not installed to the target (no, I didn't modify it to be aware of the new dynamic build option), and setupfor/cleanup does not nest so "setupfor zlib; cd ..; setupfor dropbear; cleanup" puts us back in the directory "cd .." got us to...)

Hmmm. I kind of want a debug function that compares "env" output before and after, and $PWD, but... eh, that's not SIMPLE. Defeats the purpose. Really, the package builds should work as child processes. Maybe the QEMU_MORE stuff should live a file somehow. Hmmm...

Anyway, fix up what's currently there. For the directory I can just pushd . before the source and popd afterwards. Bit of a band-aid but it works.

And while I'm at it have there been new releases of zlib or dropbear? Yes to both. Except zlib 1.2.12 isn't on the sourceforge mirrors. The previous release (from 2017) is there, but the curent release (from 2 months ago) is not. Despite https://zlib.net still linking to them as "mirrors". Time to email the maintainer. (If I link to the copy on zlib.net, the build will break as soon as he releases a new version because he deletes the old ones when he does that. "Because building against it would be insecure", he says. That's not how this works for any other package. Ok, it took him 5 years to get this release out because it's a very stable project that does not do "heartbeat" releases to show the maintainer hasn't quietly died without anybody noticing. But still. Back in aboriginal linux I had my own mirror page I put the tarballs on, but manually maintaining that and other people depending on me to do that going forward isn't SIMPLE.)

Eh, I can update the dropbear version, anyway...

Sigh, miscommunication within the houshold resulted in the mortgage check being sent out on time (so it wouldn't be late), and bouncing today because the money wasn't in the account yet. (Fade then got a cash advance against her credit card to put enough money in the account if they resubmit it. If I knew the money would take this long to get here she could have done that last week, or I could have taken another contract after the last one, or sold stock, or tapped a Roth IRA... Two recruiters have called me this week to ask if "things are finally panning out with Google" and I replied that I honestly don't know. Xeno's invoice.)

The money SHOULD have been in the account because the dot-com company that the middleman outsourced the wire transfer to said it completed at 8:15 this morning, but the money wasn't in our account. Fade called the bank people, and they said there was no trace of a transaction. Cue hours of digging...

So of course it turns out I filled out the bank deposit information wrong. I typed in the info it asked for accurately: the account number was right, my name was right, my mistake was when it asked for a swift code I gave it one. The bank guy Fade talked to assured her you can't use a swift code for a domestic transfer, it's ONLY for international transfers. (And yet the transfer went through...? "Yes, but what happens if you DO?" "Oh. Huh. Ummm..." Ah, that old refrain. Story of my life.)

The middleman company's web form defaulted to "swift" instead of "ABA routing number" (and thus there was no place to type a routing number since I didn't notice you could click on and switch the type), and their system did not check that it was trying to do a domestic transfer to a US address via swift code. Instead it did the transfer to "Wells Fargo Bank International" (which apparently isn't the same as the domestic Wells Fargo Bank?). The account number was right... for the domestic bank. There apparently WAS an account with that number to receive the money, since IT WENT THROUGH. Not minding that my name was not on that account?

So now I've asked the middleman company to ask their outsourced wire transfer company to reverse the transaction, although this took hours to do because the person I was emailing last time turns out to be in New Zealand and didn't get into work until around 4pm my time, so I'd sent four emails and created an account on the middleman's slack by the time I got a reply.

They say I should wait for the transaction to reverse (3 business days), then change the deposit info and resubmit (3 more business days). So hopefully sometime next week I might actually see money. And I shouldn't fiddle with anything while it's reversing which rules out a parallel invoice with the remaining money parked at the middleman. (I only invoiced myself for 3 months because I did not trust this process. A few weeks back I explained to Elliott why Fade handles all our online purchases, and he tried to reassure me as to their safety. Did I mention that as far as I can tell, none of my computerized medical records still exist? You literally CAN'T get rid of that stuff, and yet. Welcome to my world...)

That ate my energy for the day. Curled up into a ball for the rest of it and finished reading Harbinger... which ended on a cliffhanger. Book 4 came out in 2014, EIGHT YEARS AGO. Sigh. I really hope it doesn't take her another 8 years to resolve the cliffhanger, she turns 60 next year. (I'd subscribe to her patreon except for the stupid screen blanking bug every time I visit her profile, no of COURSE patreon hasn't fixed that. I should do that on my phone, but "my phone being authorized to spend money, even indirectly" is a line I have not crossed. The orwellian tracking device with cameras, GPS, and whole-room microphone that's on and with me 24/7 does not get to spend money. Yes I am contributing signifcant chunks of software to its operating system, but that's one lump in the katamari. They try very very hard to secure it, and state level actors and criminal syndicates (venn diagram's not QUITE a circle there but between trump/putin/xi it's pretty darn close at times) push back the other way with a literal body count. The hardware was made in china and I'm supposed to TRUST it? My main defense here is being largely uninteresting and blending in with the crowd.)

I got a bug report about disabling logging changing the cross compiler behavior in a way that broke the dropbear build. It looks like an absolute path to the cross compiler prefix (thus any tool name you append becomes an absolute path to that prefixed tool) becomes an unpathed prefix with the $PATH adjusted to have the set of tools in it. In THEORY both should behave the same, in practice the dropbear build started using gnu/autoconf back in 2009 and thus only ever works by coincidence.

But when I tried to reproduce this problem in a clean directory, I got:

scripts/root/plumbing: line 17: root_download/zlib-1.2.11.tar.gz: No such file or directory
wget: unsupported protocol: https://downloads.sf.net/libpng/zlib-1.2.11.tar.gz

Oops. I promoted wget and updated the airlock list to replace host wget with my wget, but the "defconfig" build doesn't include openssl (and thus https support). And I really don't want mkroot to DEPEND on the host having openssl installed. Long-term I want to do what busybox did and implement my own https plumbing. (Denys did it from the RFC, it's not an unmanageable lift. I'd REALLY like to make puppy eyes at him and use his code under 0BSD but he's been incommunicado for a bit.)

In the meantime, possibly I should modify the dropbear build to use /usr/bin/wget for https downloads. (Or at least fall back to it on a second attempt...?)

My problem with $(echo||) being unable to return an error is that pipe_subshell() returns an fd, but not the PID, without which we can't wait() to get the exit code. That's fine, the shell won't accumulate zombies because wait_job() will eventually harvest the child data (and discard it as not in the jobs table). But it does mean that the return code from the subshell doesn't get propagated anywhere.

Which is actually what bash does: echo -n $(false); echo $? prints 0. What barfs is a syntax error, which bash notices early because it's the host shell recursively parsing the shell syntax within the $() and thus noticing the unbalanced statements before the end of the parenthetical block. Which means this isn't JUST an obscure issue of using case statements in shell substitutions (which I've never actually seen done), this is easily user visible syntax error checking. Hmmm.

Darn it, I have to make $() parsing catch syntax errors at parse time. Which means parsing has to be recursive, which it's REALLY not set up to do. Grrr.

Probably I should finish the rewrite to let while true; do echo hello; done | head -n 3 work properly first. (The first pipe segment needs an implicit subshell, the while can't run to completion in the host shell context before launching the next pipe segment. Also, head closing its stdin needs to terminate the while loop. Right now if you add explicit parentheses around the first pipeline segment yourself, "echo" is noticing the output is gone but the while true isn't and thus doesn't terminate, so the pipeline hangs eating CPU after outputting its three lines.)

All of this is tangled into terminal and job control, of course. I did large plumbing changes for this and got pulled away before they were finished, and need to reverse engineer my own code so I can complete it. Which is a well I haven't dived back into yet because I need to pick off some of the items on Elliott's list first: at least dd, tr, readlink, and tar --transform since all those are ALSO open cans of worms already...

Darn it, it's been long enough since I've edited videos I'm not rembering the key bindings. I need to make myself a cheat sheet. Let's see...

I did blender setup and saved defaults so it starts in video editing mode with a good panel layout, so I don't need to do anything with that, but I should dig up and bookmark the video I used to do the setup in case I need to install it on a new machine.

Plug the USB headphones in first, click the xfce volume icon and select "audio mixer" at the bottom of the drop down, go to "output devices" and click the green checkbox, go to "input devices" and click the green checkbox, THEN run blender (because you won't be able to move what it's outputting to otherwise and for some reason plugging in the headphones doesn't make them the default input/output device). This is more important for recording than editing, but it's still nice to have headphones when editing.

Dismiss the stupid advertising pop-up it always starts with, then at the bottom click "add->movie" (in view/select/marker/add/frame/strip). navigate to the first mp4 file from simplescreenrecorder and select it. Mouse over the display window and hit "home" to zoom it, mouse over the edit strip at the bottom and hit "home" to zoom that. In the upper left (properties) area scroll to the top and update the "end frame" field to the number of frames in the clip. (That has to be manually updated every time material is added, "home" doesn't change it. The default of 250 is insanely short.)

Now down at the bottom press "play". If it loops back to the start you've gone past the "end frame" and need to adjust. If you pause and advance the frame number with the arrows it plays just one frame, and at 5/sec it's reasonable to navigate by that. The vertical green line is the current frame number. Playing forward is more coherent than playing backward. It cuts at the current frame it's the START of the current frame, so if you advance to where a sound plays or a screen change happens, your ending frame is ON that change (not the frame before it). The starting frame for a cut is AFTER the last frame you want to keep. Left clicking also moves the line (selecting a strip is right click, yes it's crazy/backwards). Or hit "b" to batch select, hold down the left mouse button to drag the rectangle over both strips (audio is on top, video on the bottom), and release the mouse button. No there isn't a reliable visual indicator of what is and isn't selected, it sort of changes a bit but not in an easily confirmable way.

Hit shift-k to hard cut at the vertical green line. (Only applies to selected strips, and you have to re-select after every move. Non-shifted k is a "soft cut" which has never worked right for me.) Right click the little deleted bit, hit x to delete, confirm in the pop-up menu (sigh, it's got ctrl-Z to undo with a VERY deep stack but no, it needs a confirmation pop-up every time), select the remaining strips with "b" again, hit "g" to move, and move them left to fill in the gap (cursor left, then enter when done. If it overlaps the existing one it'll go red, but it bumps it back right to not overlap when you release so you don't have to be too precise. If there's a black dropout, you accidentally left a gap so move it again.)

Video editing is basically repeating that over and over to chop out the 1/5 second intervals that shouldn't be there until done. Add the image trailer at the end, then in resolution select "100%" in scale for render resolution (it keeps going back to 50%, maybe I need to update the save thingy?), and then at the top render->render animation. This writes out the file, very very slowly, to a random-ish name in whatever directory it started in (it doesn't prompt you for an output file name, and when you navigate to another directory to read the filename, it doesn't care).

Watch the result in vlc to make sure it's acceptable, then upload it to prudetube.

Slowly getting pulled back into shell headspace. I have toysh to-test notes for things like echo $(if true) and echo $(true &&) that should produce syntax errors (and thus an exit code of 1 despite inability to run the command, same general idea as ${unset?complaint} erroring out in the command setup before having something to run). Making that work requires some poking at sh.c. It also raises that problem that if it just produced error output without changing the return code (luckily not the case here), I'd have no idea how to make the test suite TEST that properly, with TEST_HOST also passing. I kind of need a regex match on the produced output. Or "did it produce any output"? Hmmm... I could probably do it with txpect.

The next fun todo item in that same list is echo $(case a in a) echo hello;; esac) which toysh can't parse because $() contents just matches up parentheses, it doesn't do full recursive parsing (I'm waiting for somebody to complain who actually NEEDs it) because it would have to either discard the parse results and re-parse it again later anyway, or else result in a nasty branching tree structure that would be awkward to free() again. Right now I parse shell script into a circular doubly linked list with each node containing an array of argv[]. (Ok, it's an array-of-arrays with the first being the command line and the later ones being HERE documents; reality is ALREADY pretty complicated.) Having each element of that argv[] potentially have MORE THAN ONE entire subsidiary shell script circular-linked-list-containing-array-of-argv[] is just... really dowanna? That's why I'm keeping $(substitutions) as a string until it's time to execute them. You've still gotta get the sequencing right:

$ X='$((1+2))'; echo $($X)
bash: $((1+2)): command not found

But that's why I have/need so many tests...

I got the release out, after the usual shenanigans.

Back to poking at dd again, and hating dd conceptually. This keyword=value thing was really not a good interface (posix never even says what conv= is short for), and the DEFAULTS are all wrong: Why is there no seek=end? Without conv=notrunc dd defaults to truncating the file before writing to it, but oflag=append does not imply notrunc? (The toybox one probably should; that's a deviation from what upstream does but it's a NOP otherwise! Or undefined behavior when two oflag=append stomp some of each other's initial data and then interlace the output.)

Right, nose to the grindstone. Pickaxe to the coalface. Read the spec again, and the man page. Oh, speaking of which, I'm still subscribed to the coreutils mailing list (camping the spawn of "cut -DF") and I saw a MUCH nicer syntax for something wander by... here. iseek and oseek instead of remembering which one's seek and which one's skip. I definitely want toybox dd to support that. Although the adjacent commit is damn awkward to implement because toybox passes all this through to "atolx()" which is case insensitive, and "b" is already 512 byte blocks... Wait, no, it was originally bytes? Why did that change... because "most things" wanted b to be 512. Which things? Apparently dd! Probably posix says that. And indeed, dd if=/dev/zero count=1 bs=1b | wc gives me 512 bytes. So they added a new capital letter that means something different than the lower case letter. Bra filesystem checking vo, gnu/dammit guys.

Over on github Elliott wrote a thing more or less in favor of assert() to check for "should never happen" errors. Personally, I lean towards "never test for an error condition you don't know how to handle".

The problem is half the time a "can't fail" error is benign. I've seen spurious -EINTR leak through because the process was suspended and resumed, or -ENOENT when a device got hot-unplugged (which should have been treated as EOF but wasn't; I.E. same general reason for nerfing SIGPIPE). In a lot of cases the correct thing to do is ignore it and continue, because it doesn't actually indicate a problem. Exiting in such a case just makes the program more brittle.

An assert() is fundamentally about being afraid to do the wrong thing, thus exiting. But exiting can BE the wrong thing. (Especially when your caller doesn't notice and carries on with the truncated data, but simple denial of service is enough to be wrong when it would otherwise have worked.) In trying not to do the wrong thing, you do the wrong thing. In biology, this is the histamine problem. We deploy antihistamines at everything because our bodies respond to a WIDE RANGE of stuff with swelling, which is often a bigger problem than the original issue... but not always. Exiting is not always the wrong thing: I've got buckets of xfunctions() in lib/xwrap.c that exit when they detect something going wrong. But I've also removed a LOT of spurious asserts from code over the years where it was complaining about a non-problem (and thus the proper fix _was_ to remove the assert).

"What is the best thing to do here" is a DECISION. I do a lot of staring and pondering at cases, trying to figure out possible inputs and their ramifications. But "if the unexpected happens, do this"... having a stock response to the unexpected is a contradiction in terms. When I don't know what the correct response is, I lean towards not making it worse. If I don't know how to respond, I do not prepare a response. (And "suppose the kernel is compromised and sends you malicious attack data when you're reading /proc/filesystems"... I can't fix that. I'll suppose a future kernel produces more than 4k of input, sure, but a filesystem name of "../../../../etc/shadow" is not an attack vector I'm willing to add code to defend against: in that circumstance they can probably already ptrace me. I'm out.)

What I really want is a proper test suite that triggers every possible error path, but that turns out to be really hard to do. An error handler that is never tested can BE the problem. Case in point just now: I got the order of operations wrong because that error path has never triggered. How do I add a test to trigger it? I don't remember how you can fail to create a new variable, but I remember there was a way to do it? I need to read through setvar_found() to see why it can fail... the two goto bad; cases are because VAR_READONLY, or += on VAR_INT failing calculate(). Except (vv = addvar(s, ff))->flags = VAR_NOFREE; does not set either of those flags? Oh what was this case... something about dangling symlinks with VAR_NAMEREF? Have I not implemented the variant that can fail yet?)

I gotta set aside a block of time to get back into this headspace. If I can't come up with a test case, then the error condition can't occur here and I don't need a handler for it. (And that's not a "should never happen" because a function I wrote is calling another function I wrote. When you change the function, you check the callers before committing it. And then you add tests for the externally visible behavior your new code causes/allows/prevents/changes.)

Sigh. It is entirely possible I'm wrong, but I don't think going EITHER way is always right. Elliott leans one way, I lean the other, but it's an individual judgement call in each case what to do.

Today was the day that money from google was supposed to make it to the middleman organization. (45 day payment terms in invoices, and Fortune 500 companies always wait until the last day to pay so they can eke out tiny little bits of interest on payables. That's late stage capitalism for you.) The website balance is still reading $0.00, but I sat down and figured out how to do the paperwork anyway (with Fade handholding via phone). I invoiced myself for Q2 (13 weeks, at exactly halfway between the hourly rate the last 2 gigs were paying me), approved my own invoice, entered my banking info for deposit, filled out the third party form they use (docusign is neither secure nor particularly legal, and yet...) to hand over my identity theft number so they can do tax reporting (although not witholding)... and now we wait. Their documentation says they process approved invoices twice a week. I'm assuming the invoice existing before money comes in won't cause a problem (it'd just get bumped until next time)...

Almost got the toybox release out. I actually uploaded a version of my release checklist which isn't entirely up to date but gives a general idea. Wrote up the release notes, and edited them into parseable HTML. Built the mkroot images with the right version tag and uploaded them. (I cheated slightly and undid the tag, committed two more things, and redid the tag after the build, but both changes were just to documentation and wouldn't affect the binary). The toybox static binaries come from mkroot these days, out of root/$ARCH/fs/bin/toybox in each target build. Yeah it pulls in two things out of $PENDING but I'm ok with that.

I spent April shoveling through open tags and backlog of half finished stuff, trying to close down for a release, but what I really need to do now is get serious about the dev environment stuff. Google gave me a list of stuff they want for AOSP to hermetically build itself, which is the priority to get done first, that's only part of the picture.

I want to run the toybox test suite under mkroot, which is MOSTLY a question of toysh being good enough. Right now, when I run "./sh scripts/test.sh" it goes "source: syntax error 'scripts/runtest.sh'@221: ;;" and yes I had to upgrade the shell to beat that much detail out of the error message. (It was initially just saying "source: syntax error ;;" which... isn't helpful.) This is the same part of the code (case/esac parsing) that's preventing /usr/bin/ldd from running under toysh. (Gnu ldd involves a large shell script, which starts with #!/bin/bash so once again ubuntu's decision to switch /bin/sh to point to dash remains stupid 15 years later. You can't NOT have bash on the system. You can TOTALLY not have dash.) Once I've got the tests running under mkroot, I have a lot MORE tests I need to add that run as root and/or require a known environment containing things like the ethernet "dummy" driver.

I also want to build toybox under mkroot with a native.sqf compiler loopback mounted. This immediately dies due to "make" not being there, but assuming I harvest static binaries from an old aboriginal linux build and stick them after toybox in the $PATH, I can presumably attempt that with toybox providing what it can, and both verify that the toybox commands are load-bearing in this context, and get a list of what's missing via scripts/record-commands.

I eventually want to do a new automated Linux From Scratch build under mkroot. I'm tempted to take my old LFS 6.3 build I automated years ago and run that (because it was known working at one point), except more than one of those old packages have #if/#else staircases ending in "#error what is musl libc". This was the case even when I was trying to use musl in my old "last gplv2 release of everything" toolchain under aboriginal (and was the main thing delaying me from switching that project over to musl: I'd lose the ability to build LFS natively unde the result). Using an even newer toolchain with the old packages will not have improved this.

This sort of version skew problem happens a lot. On monday SJ Hill emailed me asking about initramfs console issues in a 2.6 kernel, and I couldn't build a 2.6.39 kernel from a git checkout to test my answers with the current debian toolchain because apparently newer gcc versions default to producing pic code now? And then when I applied that fix, the 2.6 build tried to #include linux/compiler-gcc8.h which doesn't exist. It really really should not CARE, but does. He said he's building it under an old debian version running in a VM, which was more effort than I wanted to go to just answering email related to documentation I wrote in 2005. Anyway, we got his problem fixed and he wandered off to do his work. The point is, old packages not working with new build environments isn't always a problem with the new build environment, but with the old package making stupid assumptions. When "gnu" is involved, you can just about guarantee it.

A google developer is sending me fixes for toysh. I am simultaneously happy somebody's paying attention to it and embarassed it's not ready yet. It's like a realtor coming to show somebody the house when you haven't tidied up. I'm suppressing the urge to preemptively apologize. They're reviewing an unfinished draft, but I've been working on this for over 2 years now so it's my fault it's not entirely there yet. Speaking of which....

I recorded two large chunks of what should be the next video, on building toybox from source. Now I need to edit it into... at least 2 videos, it's 20 minutes of raw footage. I need to remember to type "reset" and clear my terminal window more often, because my editing decisions are a lot more constrained when I don't. (The backscroll in the terminal window jumping around would be distracting.)

The GOP was especially evil today. Too stressed to focus much. And yes, this is the same supreme court that ruled in favor of Nestle in the recent child slavery case. The Boomers are not dying fast enough.

I'm trying to remind myself that youtube becoming prudetube isn't JUST a sign of US culture being co-opted by witch-burning puritan Boomers clutching their pearl rosaries as they sink beneath the waves, it's mostly about youtube being incompetent and self-destructing by slowly driving away its creators. (Note how in this great resignation howto video at 3:13 the word "onlyfans" is bleeped out, in a list of patreon/etsy sites people use to make money online.) And that Amazon becoming an SEO hellscape being called out by celebrities is about them having bait-and-switched from customer-driven search to bid-for-placement while even screwing over sellers with its own copycat products that get the top five search spaces for free.

All platforms have a lifespan, and if Youtube is going the way of Vine, Twitter going the way of Livejournal, and Amazon going the way of Woolworth's that's not unexpected. More like inevitable. Similarly, I now go entire weeks without remembering microsoft EXISTS, which did not used to be the case. They're still around, but after 30 years of unassilable dominance they have no presence in the phone space, can no longer exclude macs and chromebooks from the PC world, and have their own version of "Win-OS/2" (which meant back in 1993 anybody deciding whether to ship an OS/2 binary or a windows binary could just ship a windows binary and cover both systems, quickly ending most native OS/2 development).

There's plenty of places to buy stuff other than amazon. Even if I'm not personally active these days, plenty of people are still promoting nudism, and holding events with reasonable turnout. (I'm told Star Ranch's naked 5k run had record attendance this year. I wanted to go but it was too close to my return flight from Minneapolis.)

Trying to update various toybox documentation, and it's HARD. I very carefully phrased was was there, over a large number of attempts, and now I need to change what it's saying in a way that resequences stuff, reorders categories, changes category divisions... Hmmm.

Xeno's release process continues as usual. Half way to halfway... Meanwhile the bug reports continue to come in. *shrug* The tree's in pretty good shape, I'd just like to nail it down and SNAPSHOT it already. But that involves a LOT of documenting, which is a special kind of turning over rocks to see what's underneath. (Honestly, I'm not really that good at this. I'm just persistent!)

My wife is getting a degree in classics, doing her dissertation on "Social interactions between (the group of marginalized people composed of) slaves and sex workers in Plautus". She's currently taking a class on sex work in the ancient world.

A recent thread from Dr. Sarah Taber on how men took over the egg trade from women (replacing "egg money" with factory farming and Salmonella) neatly explained why prositution is illegal. The relevant observation is "Men force women into the worst-paying parts of the... system. But once those trades start making [money], the women have to go." Here's a 1983 paper how men pushed women out of computer science as soon as there was money in it. Centuries ago a woman invented kabuki dancing and later only men were allowed to do it.

Prostitution doesn't cause any actual PROBLEMS, it's just that women always make more money at it than men do, thus it can't be allowed. Yet it's been legal all my life in Nevada and Amsterdam which have no problem with it. The current war on camgirls is related to the blowback against "the great resignation": young women achieving financial independence working from home cannot be allowed by a patriarchal dominance hierarchy. Everything else is just an excuse.

Centuries of home weaving and sewing were forced to give way to the Triangle Shirtwaist Fire, an example of human trafficing that had nothing to do with sex work. Most human trafficing today is in agriculture. Blaming sex is the usual pearl clutching diversion.

Fade's tumblr is full of references to, among other things, "our stronger nudity taboo", and as a (largely lapsed) nudist, I am sad about said stronger nudity taboo. Made recently much, much stronger by senile lead-poisoned boomers.

Fade, Fuzzy and myself were recently trying to watch Thermae Romae on Netflix, a remake of an anime about a roman bath designer time traveling to modern japan to steal ideas. It's TV-MA... for no apparent reason. They never show a single dick. Awkward camera angles, shiny water, and anachronistic towels. (In the first episode they put all the bathing romans in towels, and then a few episodes later one of the great discoveries the roman bath architect time-travels to modern Japan to appropriate is... towels.)

Seriously? What is this nonsense? People have bodies. I showered with my younger brother until I was ten. Parents taking naked pictures of their children was entirely normal before the Boomers took over; nudity is not the same thing as sex, and this A) drawings, B) already a pay service, C) an entire story about people having baths.

Daryl Hannah's naked body in the 1984 movie Splash was PG, but the Disney+ version recently deepfaked dog hair over her butt. (Oddly enough Tubi had the original unedited version last month, because an ad-supported service was less scared of itself than a billion dollar conglomerate that regularly rewrites the law.) The video for Wrecking Ball is still on youtube, 1.1 billion views over the past 8 years, which somehow has not led to the collapse of civilization or even the singer (despite the Disney child star curse).

As for full frontal male nudity, Richard Donner's 1978 Superman showed a naked little boy's dick (when Kal-El lands in Kansas and stands up) in a PG movie. In R-rated films nonsexual male full frontal nudity was everywhere from The Terminator to Life of Brian, as recently as 2009 Doctor Manhattan spent most of Watchmen naked.

But Netflix decided to rate something TV-MA:nudity (no other reason given) and then NOT show anything in ANIMATION (drawings, not even pictures of real people) where everybody is repeatedly naked for UNAVOIDABLE PLOT REASONS? Seriously? Either show the nudity, or don't make it TV-MA:nudity. Pick one!

The senile lead poisoned Boomers (no really, they are uniquely measurably stupid) are not dying fast enough. From climate change to fascism (which always includes racism, sexism, and homophobia in the definition of "fatherland" because everyone must perform their assigned societal roles full time no exceptions), it's a race between enough Boomers dying and the collapse of civilization. I'm aware this is a rounding error compared to that, but it's personally annoying.

Oh good, Denys is posting to the busybox mailing list again. That means I can stop paying attention. (I noticed in the first place because there was a guy emailing me about a bug disclosure that he thought might be a significant security vulnerability, and I was trying to help him. He's posted it publicly now. Yes, I still get emailed about busybox stuff, and try to at least gracefully hand off...)

I've copied the toybox release note slush pile into the start of news.html, grabbed a new hitchhiker's guide quote for the top, and am trying to beat it into some semblance of order to cut a release. There's still a zillion trailing loose ends but it's been FIVE MONTHS since the last release and I'd really rather not make it six.

Blah, what is my todo list: Release notes. Youtube channel for the videos. Update nav bar with youtube and patreon links, and collate the documentation somehow. While I'm at it, the main toybox page should go to about.html but with release a release note link from there... probably that goes at the top of the page?

The last time I tried to collate that I hit the fact that the summary at the top of news.html and the summary at the top of about.html are redundant to each other, but not as easily mergeable as I liked. Really, I should have a summary blog in header.html (which produces the nav bar via ssi, and yes I have a todo item to make toybox httpd support ssi...)

This is my problem doing this sort of work, it's fractal. Attempting to close tabs winds up opening more tabs. Doing the work makes the todo list longer, and there's sort of a hydra thing going on where trying to find half-finished stuff and finish it results in more half-finished stuff...

(You can tell I'm in debug mode. I'm angry at software design again.)

As always, updating documentation leads onto a development tangent. In this case, now that I've got binary toolchains uploaded I want to explain CROSS_COMPILE= as its own part of the FAQ, and the examples I have for that are both statically linked, and I want to teach mkroot to set up dynamic library directories in the target (as a new "dynamic" argument like dropbear), which means writing a new scripts/root/dynamic so you can "scripts/mkroot.sh dynamic" to add the shared libraries the way you can add dropbear. (And then move those builds before toybox, so the toybox build can test -e $ROOT/lib/libc.so and do a dynamic build if it's there.)

But despite all the talk about merging /bin and /usr/bin, what Devuan did NOT merge is /lib and /usr/lib, which is annoying for me trying to add a scripts/root/dynamic that works with the host toolchain too, because libc.so lives in /usr/lib and libc.so.6 lives in /lib. (Well, both add a gratuitous x86_64-linux-gnu-gnu-gnu-stallman-forever-ftaghn-ia-ia-all-hail which has nothing whatsoever to do with Linux and never did. The migration from the libc5 the kernel guys wrote to libc6 (Ulrich DrPepper's fork of glibc is where "glibc 2.0" came from, pretty much the same as egcs where the original project was long dead and linux people frankensteined the corpse into something useful) was because Ulrich's project was all about adding thread support (I.E. spraying everything down with gratuitous locking) and the flood of Java people in 1998 (tripling the size of the community after Netscape released its source code and credited Linux for the decision) outright DEMANDED thread support because James Gosling hadn't included bindings for unix staples like poll/select in the standard Java library so the ONLY way to do nonblocking I/O was to spawn a blocking thread for each background read, so Java devs who'd learned nothing but threading couldn't function without it. (Sun invented threading because forking processes on Solaris was so heavyweight, Linux could fork processes faster and cheaper than Solaris could spawn threads so had never bothered with threads, but if twice as many cobol programmers had shown up to use Linux than the entire previous community, they'd have gotten good cobol support in a year or two, so... Anyway, the FSF was the largely passive recipient of a lot of external development work that tried to use them as a coordination point largely for historical reasons. The gnu project was announced in 1983 and was already vaporware dinged by Linus' 0.0.1 announcement in 1991. Stallman initially responded by telling people not to use it and then pivoted to trying to take credit for it. And no moving off libc5 wasn't about ELF support, that's what the move from libc4 to libc5 was about. Libc6 was about threading.)

So anyway, I need to copy everything out of the directory that has libc.so, the one that has libc.so.[0-9], and the one... No, hang on. I don't care what the host has, I only care what the toolchain has, and rather than doing library whack-a-mole I need to use cc -print-search-dirs and parse that.

Still need to special case the dynamic linker through. (/lib64 should not be a thing at the top level. Seriously. You ALREADY ADDED multilib directories UNDER /lib, and then you have another one on top. That's very gnu, used here to mean "trying so hard to be clever it wraps back around to stupid". The kind of clever that piles higher and deeper without ever clarifying anything. Eternally accumulating, never simplifying. I'm trying to eliminate problems so you don't have to deal with them anymore, this nonsense confronts you with endless "look what I did, you'll never understand it, worship me" chest beating. When I'm successful at my goals, it doesn't look like I did anything. I'm not building monuments, I'm TIDYING. Grrr.)

I'm trying to update qemu, and once again it's "upgraded" itself into uselessness. I want to run "make install" without root access to see where it's trying to install stuff. (Is it in /usr/local/bin or /usr/bin or /opt or what?) But when I run "make install" it pops up a gui window prompting me for my password so it can sudo, and if I don't give it the password it exits with failure. There is no "continue as this user" option so I can see it try (and fail) to install and thus know WHERE it's trying to install stuff.

So I try to find and rip out the call to sudo from the build plumbing, but build/build.ninja is calling meson/meson.py which is calling some library I can't easily find, so second attempt: make a NOP sudo wrapper script that does nothing but run its arguments and stick that in the front of the $PATH... and it's not calling sudo at all, this GUI pop-up is some freedesktop.org thing (I.E. it's from the systemd people). The COMMAND LINE BUILD is going through GUI DESKTOP PLUMBING to INSTALL FILES. That's not right.

I remember now why I haven't upgraded qemu in so long. I WILL NOT run a random script as root to modify my system in ways I can't see before it does the thing. That is not happening. I tried running it under fakeroot, but that said it couldn't LD_PRELOAD fakeroot.so in one of the child processes and popped up the same GUI prompt.

Ok, what I needed to do (after reading files under qemu's "meson" subdirectory, which have spaces in them and lots of references to running on windows) was add --dry-run to the install command line in build/build.ninja, and THEN it says it's trying to install stuff to /usr/local. Ok then.

The new qemu-system-ppc64 is 102976760 bytes. That's over a hundred megabytes. It is not statically linked, ldd lists 68 shared libraries on TOP of that nonsense. Grand total 3.7 gigabytes of JUST /usr/local/bin/qemu-* files, and that's not even counting the crap it installed in /usr/local/share and who knows where else. Bloatware of the highest order. Meanwhile, Fabrice's Bellard's new tinyemu is a 250k source tarball.

(Ok, as Rich points out, stripping qemu-system-ppc64 brings it down to only 20 megs. But that's still insanely large, and qemu installed the unstripped binaries by default. The Devuan install ISO is only a 1.2 gigabyte download, QEMU's default install is THREE TIMES BIGGER THAN THE DOWNLOADING THE ENTIRE OPERATING SYSTEM. Yeah sure, compressed vs non-compressed, I don't care. That's NOT RIGHT.)

This morning I MEANT to sit down and fix the sha1sum endianness issue, but instead I got sucked into cataloging various data types the file command doesn't recognize, and now I'm trying to design file -i support, to output mime types instead of the ad-hoc text format file usually produces. There was zero provision for that in Elliott's original design, and it's not easy to retrofit.

Ok, technically I want to implement file --mime-type which is JUST the mime type without the stupid ;charset=potatosalad decoration gnu/file keeps wanting to crap over things. But I have an aversion to long options without corresponding short options (it's not unix!) and thus want to an -i that does the concise/simple output, EXCEPT that makes TEST_HOST harder to support in the test suite.

(If Elliott hadn't submitted his own file.c, I'd probably have done one that JUST produced mime output to start with, and added the long text version later when people asked, and then probably not for everything... but that's not where we are today. Hmmm... This needs a long walk and a design think.)

Ahem, but AFTER I get a release out. And a bunch of videos filmed edited and posted. (Put the shiny thing down and back away slowly. Doesn't matter that I've just half-assed it based on file extension in httpd.c, I need to CLOSE tabs...)

Grinding along doing release prep: building all the targets and testing that the network actually works in them, by running my new httpd on loopback on the host and having wget fetch the toybox README file and sha1sum it. (Remember: qemu's 10.0.2.2 address passes through to 127.0.0.1 on the host, so ./netcat -p 8080 -s 127.0.0.1 -L ./httpd matches up with wget http://10.0.2.2:8080/README -O - | sha1sum which would be nice to automate somehow. I have not reopened the can of worms that is build control images yet, but I'm working towards it.

This is a kind of test I've done piecemeal over the years, but there's been nothing systematic up until now because I didn't have the pieces, and now that I _AM_ being systematic-ish about it (or at least retesting all the targets)... my armv7l fix for the network card wasn't applied right. (It's a QEMU bug: QEMU's arm device tree is the same for 32 bit and 64 bit platforms and the address it gives for the network adapter is up above 4 gigs and thus inaccesable on 32 bit targets. You can work around this by enabling the page addressing extensions on the kernel so it can map high memory down into the 32 bit address space, but you really shouldn't HAVE to. The QEMU bug that was opened about this got marked invalid, so we're stuck with the workaround. Anyway, the PROBLEM is when I added the symbol I added CONFIG_ARM_LPAE to the csv list for the armv7l target, and the plumbing prepends the CONFIG_ itself so it was setting CONFIG_CONFIG_ARM_LPAE=y which didn't work.)

Meanwhile, on m68k the wget seems to be working fine, but sha1sum is producing the wrong hash! (Not just for this, echo hello | sha1sum produces the wrong hash too. I thought I had the endianness and alignment right for sha1sum? Hmmm, mips is getting it wrong the same way, that's looking like endianness. (Yup, mipsel is getting the right answer.)

Let me guess: In 2014 Daniel Verkamp did a commit which made md5sum and sha1sum run faster, but made the code bigger and more complicated. Shortly thereafter I did a commit which made them work on big endian. Then in 2016 Elliott did a commit to add a config option to pull the assembly optimized versions of md5sum and sha1sum (and sha256 and friends) out of openssl. Then in 2019 I did a commit which reverted Daniel Verkamp's changes putting the non-openssl versions back to the slow-but-tiny versions. So the question is: did I forget to re-apply the big endian fix?

The other thing I'm not really testing this pass is block device support. Everything's still running out of initramfs, no "create an ext2 formatted hda.img and mount it on /home". That's an important step for doing native builds.

Oh hey, powerpc64le found a new way to fail. Can't tell if this is a qemu bug or a kernel bug, so I asked both lists. Probably going to be ignored by both too, due to the list traffic volume. (The powerpc list isn't that high traffic, but it gets giant piles of inappropriate crap cc'd to it. Standard kernel-dev failure mode for the past decade, signal buried in noise.)

Huh, sh4 is doing a similar failure, except it doesn't "catch up" when you hit a key, the extra data has been discarded. (It got the sha1sum of the fetched file right when I typed it in manually though.)

I caught up on the git commit log, and have (unedited) release notes covering all the commits. Now I need to figure out what else should and should not go in the release, so I'm back to picking through the "git diff" in my toybox dirs. I have a conversion of nbd to work on nommu (it daemonizes which was using fork and should now use vfork(), but htat's not a trivial conversion because fork-and-continue is not available with vfork, you have to re-exec yourself). But it's not very tested?

And THIS reminds me that I was trying to come up with a nommu test environment for qemu, because having to pull out my turtle board and sneakernet files onto it via sd card is a slow and awkward way to test compared to mkroot and qemu. (Sure I can manually set CONFIG_MUSL_LIES_ABOUT_HAVING_FORK, but I never quite trust that isn't leaking something that won't work on a REAL nommu system, and having an actual nommu target in the mkroot list gives me testing organically.)

And specifically the nommu target I was trying to bring up a week or two back was coldfire, ala nommu m68k which qemu supported long before it supported full m68k.

... and my laptop decided to power down walking to the table, so instead of resuming from suspend I've lost all my open windows again.

A few months back Joe Biden tried to change the IRS so any account with $600 in it must report every single transaction (no matter how small) to federal authorities. (This neatly disarmed the "double the IRS's budget so they can go after Billionaires" push from the left, by offering to double the IRS's budget so they can go after the poor. This doubling is of course undoing decades of cuts by the GOP reducing the IRS to a shadow of its former self, it's shrunk 20% just since 2010 because the rich hate taxes and the GOP wants to shrink the federal government to the point they can "drown it in the bathtub". This would allow billionaires to become kings and keep slaves again, the basic goal of plutocracy.)

Now Boomer Biden's taking his next step in the War on Cash: he wants all online sellers to provide the government with Photo ID to Protect the Children... no wait (checks notes) because intellectual property law.

A recent Paul Krugman Column included the paragraph:

I still sometimes encounter people who say that we live in a digital age, so we should be using digital money. But we already do! Like many people, I pay for most things by clicking a mouse, tapping my debit card or pressing a button on my phone. I used to keep singles in my wallet to buy fruit and vegetables from New York’s ubiquitous sidewalk stands, but these days even they often accept Venmo.

And later in the same column:

And we shouldn’t discount the importance of illegal activity. There’s about $1.6 trillion worth of $100 bills in circulation — 80 percent of all U.S. currency — even though large-denomination bills are very hard for ordinary consumers to spend. What do you think people are doing with all those Benjamins?

So 69 year old Krugman notes that the ability to make anonymous financial transactions on a daily basis is going away, and tuts that using cash for any reason is inherently suspicious (only criminals use cash), and never questions whether that's a good thing.

Meanwhile, the GOP is empowering vigilantes to sue anyone who "assists" someone to have an abortion, and make traveling across state lines to have one a felony. (The GOP has always relied on vigilanties, they just used to wear white hoods and lynch people instead of suing them.) Even if the credit card company or regulatory agency collecting this data ISN'T captured directly by nazis, the next data leak will give a complete list of donors to Planned Parenthood or the DNC so they know who to target with vigilante armament bill du jour. And Biden's refusal to make the Supreme Court "go to eleven" means said laws proliferate and remain in force, while Merrick Garland lets all the statutes of limitations expire on rich white men who might otherwise be prosecuted ever for anything.

Krugman's a Boomer, which means baked into his worldview is the assumption that anything they inherited has always been like this therefore it requires no maintenance and no amount of abuse Boomers heap upon it can possibly damage it in any way. The Boomer world is eternal and unchanging and required no effort (from a Boomer) to be like this in the first place. Therefore nothing important is ever really destroyed or goes extinct, no matter how incompetent those lead-poisoned narcisists are at dunning-krugering their through life. Impostor syndrome only happens to people with the slightest trace of self-awareness.

Can prlimit query stack size (ala ulimit) for comparison in main() instead of hardwired stack limit for command recursion? Is that worth doing, and does it work on nommu where stack size is a per-process attribute in the FDPIC ELF headers? Does prlimit show the correct process-local stack size on nommu, and if not should I poke Rich to fix the j-core kernel...

Finished the board bringup contract yesterday: the board is essentially up now. All three issues were hardware, but were easily fixable (in hardware) once diagnosed. So I have once again done a software project which resulted in zero lines of code, and yet fixed the problem. (Yay?)

The most interesting was that the ATWILC3000 and ATWINC3400 wifi chips are pin-compatible, so when one was out of stock (ongoing chip shortage) they substituted the other. Except being electrically compatible does not make them REMOTELY the same chip: the 3000 is a standard-ish WIFI adapter with a linux driver in the vanilla tree, and the 3400 is a router-on-a-chip that boots its own multitasking OS from built-in SPI flash and doesn't just have its own TCP/IP stack but runs its own web server, does its own crypto, it runs SNTP to set its onboard clock so it can check https certificate expiration... The 3400 is designed to act as a router for deeply embedded arduino-style systems too small to do their own networking, and its manual does not contain the word "Linux" because it's not FOR that.

The fix was to find 3000s in stock at digikey and swap the chips back. In THEORY we might have botched up some variant of a PPP connection to let Linux use the 3400 as a gateway to talk to the net through an extra hop, but that would be a HARDER hardware change because they hadn't wired up the serial port pins (which the 3000 only uses for testing).

There was a whole lot of head scratching along the way to figure out why trying to load 3000 firmware into the 3400 providing the WEIRDEST driver failures. (Oh, the driver will panic the kernel if the firmware isn't in the format it expects, because it only checks that the SOURCE ranges it's copying data from fit, not that the TARGET ranges it's copying data to fit. Needless to say the 3400's SPI flash image and the 3000's SRAM image aren't quite compatible.) Whole lot of sticking printfs into the wifi driver to answer The Wrong Questions. Searching for keywords in the giant programming manual and not finding them (because they weren't there) was also frustrating. Reading the sales brochure is what finally clarified things: oh, wait, this is the wrong category of product. (And it's pin compatible? Really? Yes, yes it is... Why?)

When you have a really FRUSTRATING debugging session, often you've missed something more fundamental/obvious than you're examining. This was an "I am in the wrong house" level mistaken assumption. Took quite some backing up until I could see it.

"Streaming" means things vanish. You either have your own copy, or you don't. (This is a subset of "intellectual property law destroys intellectual property". Years ago Cory Doctrow said we're living in a dark age, because so much will fail to be preserved from today due to IP law that future historians will have a much clearer picture of the 1800s than today. Or was it lawrence lessig who said that? I googled, but the talk has fallen off the net...)

Visiting Fade and doing board bringup contract. Not very active on other stuff at the moment.

Fade set up a new twitter account for me so I can read threads without Twitter's stupid paywall page blanking the screen as soon as you scroll down three tweets, and while I didn't intend to post anything with it I got tempted into replying to stuff. Yesterday, I replied "We're all just waiting for the Boomers to die" and today:

We have determined that this account violated the Twitter Rules. Specifically, for:

Violating our rules against hateful conduct.

You may not promote violence against, threaten, or harass other people on the basis of race, ethnicity, national origin, sexual orientation, gender, gender identity, religious affiliation, age, disability, or serious disease.

As a result, we’ve temporarily limited some of your account features. While in this state, you can still browse Twitter, but you’re limited to only sending Direct Messages to your followers –– no Tweets, Retweets, Fleets, follows, or likes. Learn more. Your account will be restored to full functionality in: 12 hours and 0 minutes.

You can start your countdown and continue to Twitter once you:

Verify your phone number

Delete the content that violates our Rules

1 Tweet

If you think we’ve made a mistake, contact our support team.

So twitter considers "waiting" to be hate speech. (This Onion article is from twenty years ago before the Boomers got NEARLY this bad.) Meanwhile, the "your account can still browse twitter" is clearly untrue because now it's the phone number entry paywall at every page (paying with information is still paying, and my spouse already GAVE THEM her phone number to create the account so they HAVE THIS INFORMATION, it can only be some kind of "we know who you are" threat?), and they want me to performatively delete a tweet they've already blocked.

This is not a healthy website.

When busybox ash encounters an elf binary it doesn't recognize (such as an arm64 binary on an armv7l system) it tries to run it as a shell script.

root@stm32mp1:~# /boot/toybox-arm64 
/boot/toybox-arm64: line 1: ELF��@@�C
                                     @8@
                                        : not found
/boot/toybox-arm64: line 2: syntax error: unexpected "(" (expecting ")")

Meanwhile toybox notes any utf8 parsing failures on the first line (I.E. anything >127 that doesn't decode as valid utf8) and goes "this is a binary". Not the best heuristic, but it catches most things.

Here's a tangent I edited out of an email I was sending:

I'm trying to make a minimal self-hosting system for a bunch of reasons, and C remains the obvious language to do that in. But "where the source code comes from" is not strictly required to be part of that core. My Aboriginal Linux system would wget tarballs, and the automated building plumbing at http://landley.net/aboriginal/control-images/ created squashfs images of pre-extracted source code so the VM could just refer to a mount point with source code for each package in a separate directory.

The source code download mechanism doesn't affect "here is a classroom of students studying everything to understand it all". It doesn't affect the "Ken Thompson's Trusting Trust attack" analysis. It doesn't affect "here's an archival system you can rebuild known state from" (because if you can't reproduce it from scratch under laboratory conditions what you're doing isn't science). If you don't everything locally cached before starting you've thrown the "reproducibility" part out the window.

(Sigh, I need to do DESIGN videos. Lots of explanations about _why_...)

Flying to Minneapolis.

The Posix committee has approved adding strlcpy() to the next release (SUSv5, Issue 8, and POSIX-whatever year they get around to publishing it). Which raises an interesting issue of how to deal with that in portability.c: I independently invented "strlcpy()" back under Turbo C on DOS, and was happy to find it in SunOS' toolchain and BSD and so on, but it was never in glibc because Urlich DrPepper hated it for some reason. Fine, easy enough to implement my own. But if it gets added to libc and the standard #include headers, that's going to clash. (We're 100% going to disagree about "const" if nothing else.)

Do I put the definitions inside #if __STDC_VERSION__ < V4_RELEASE_DATE perhaps? Dunno. I've been using c99 and basically ignoring c11 (and bits of c99 to be honest, LP64 is better for known integer type sizes and declaring variables at the start of a block lets you FIND them), but I haven't been saying -std=c99 because I haven't needed to: both gcc and llvm have given me reasonable behavior as the default. I never had to tell /bin/sh -std=susv4 to specify which version of posix to use either, because THAT'S STUPID. I just run a sufficiently capable version of the shell that can handle my scripts, and it does the thing. If you find yourself having to say "python3" instead of python, it means the python transition faceplanted spectacularly.

Hmmm, looks like somebody on BSD complained (they have their own version in libc) and I already removed it from toybox. (I forgot.) Oh well, bullet dodged I guess?

The USA had nine mass shootings over the weekend.

Completely lost in the noise of course, but no party is "pro life" or "protecting children" when active shooter drills have replaced singing "duck and cover" and THEY ARE OK WITH THIS.

Back to work on the stm board. Serial console works now!

Something about the yocto install on this board puts it in a mode where the mouse wheel scrolls through command history instead of backscroll. The fix is "tput rmcup" which isn't installed on the board, but running that on my laptop and piping it to hd says "echo -e '\e[?1049l\e[23;0;0t'" should fix it (and did). The man page implies that the first one is involved in "mouse tracking" support but doesn't really properly explain (it says waht ?1000l does but not ?1049l), and if it mentions the "t" one at all I can't find it. Still, it worked, and that's the important part.

So the first thing I'm trying to fix is the gigabit ethernet jack. In theory it's the same chipset and driver as the raspberry pi form factor evaluation board, but something about it is unhappy. I set up a static PTP connection between it and my laptop, and the board never sees packets sent from my laptop, and my laptop increases the rx error count for each packet (although occasionally some get through!) Hmmm...

Boomer Prudishness has gone completely insane.

Trying to close tabs, in this case tabbed terminal windows within which I have a bunch of half-finished commits, and also the various "clean1, clean2, clean3..." directories in my toybox workspace where I was doing something off on its own branch. (Mostly toysh changes.)

One of the abandoned forks was a very long rathole trying to figure out toysh crash where free() was aborting on one of the targets, which turned out to be 2 things: 1) I was popping an entry off of a list but had a cached pointer to it that needed to be updated, 2) I was hitting a bug in musl-libc's realloc() that corrupted the heap. Apparently this bug only existed briefly a year ago, and of course the toolchain I built (with git commit du jour since musl hasn't had a release in over a year) was within the range that had it.

I really hope the fall of Putin takes down the GOP he funded. (And yes, Manchin is a republican.) Texas suppressed 19% of the mail-in ballots this last vote.

So, definitely not vimeo then. (They're charging patreon creators thousands of dollars per month for hundreds of views per month, as part of pivoting away from hosting video for individuals into hosting video for corporations. Patreon has not yet caught up with the change, and continues to think it has a partnership with them.)

Started on that very brief contract (just through the end of the month) to keep the lights on while the Google guys work out how to get money from point A to point B through a legal briar patch, and we just worked out that the board I'm doing bringup work on was damaged in transit, which is why I couldn't get serial output. So I packed it up and overnighted it back to them, and am taking tomorrow and friday off.

Started the board bringup contract. Oh goddess another Yocto build. But I guess that's why they need help: if they weren't using Yocto they'd be done by now.

The ELC-in-Austin CFP closes in an hour and a half, and I still can't come up with the enthusiasm to submit proposals. I mean, I could do "building the simplest possible linux system" again, presumably much less jetlagged this time and with mkroot actually functioning. I could do "the state of toybox". I could propose a talk on mkroot. I could do a talk on android convergence. Heck, that stupid colocated "community leadership conference" suggests "Open Source Governance and Models", meaning I could dig up "the prototype and the fan club" talk again and try to get a version properly recorded.

But I'm just not feeling it. I don't feel connected to this community. This is The Linux Foundation: stamping out "community" is what they do. Chasing away hobbyists so the suits can charge rent. I wouldn't be going there to see anybody...

*shrug* Looks like it's moot anyway, the website closed the CFP earlier than the posted time. Probably using client-side javascript to use my timezone instead of the stated pacific time, but I'm not resetting my laptop clock to get around that.

The dollar value of Asset Forfeiture has surpassed burglary in the USA. The police are now MEASURABLY stealing more money than the "criminals". They need to be defunded.

February 27, 2022

Darn it, decided to experiment since I was feeling "tired but not anxious" and HEB was out of the milk tea I like, so I got a bottle of the japanese milk tea (not the brand I get when I'm in japan, but they don't have that here, but this is still a fairly standard sweet black tea with milk), which I'd stopped drinking because the green tea version of it seemed to be triggering anxiety (I.E. upsetting my digestion for a day or more at a time, in a way that messes with my emotions). Two hours later, I'm full of anxiety. Funny that.

February 26, 2022

So I was sick and then Ukraine got invaded which is quite distracting. I haven't emailed Denys to see if he's ok because I don't want to bother him right now, but I did finally give the busybox list a heads up that he might be a bit distracted. He's posted a single terse message to the list this month.

February 25, 2022

I've done almost no work since I got sick, partly because I'm still recovering and partially because Ukraine got invaded, which is hugely distracting. I have no special expertise here and there's nothing I can do about it, but strangely despite draining my energy it's also reducing my anxiety, by taking things that seemed unresolvable and making them... much less abstract.

You know how in Persona 5 the Phantom Thieves have to send a calling card to manifest the treasure so it becomes real enough they can steal it? That's how I feel about Ukraine. All the nebulous Russian influence behind Trump and Brexit where the Mueller report can come out and be ignored because untouchable oligarchs endlessly throw oil money around to run troll farms and bribe politicians and talking heads. The guy who ran Trump's presidential campain got a presidential pardon and the russian oligarch who got Boris Johnson elected was elevated to the House of Lords.

Russia's invasion of Ukraine brings all that stuff to a head so it can actually be addressed. The invasion is NOT plausibly deniable, and the responses to it aren't endlessly delayable until the statute of limitations expires. Tanks on the ground means now, and as with Saddam Hussein invading Kuwait there's no middle ground to hide in.

Faux News can't be propped up by Russian money when Europe has permanently cut BOTH nordstream pipelines and blocked Russia's largest banks. (Creating "the unbanked" is fine when it's punching up, doing it to a country in response to firing missiles at apartment buildings is way different than doing it to individual civilians for being seen naked.) Putin turns 70 in October: the math on a dictator for life changes when he's 3 years from the average russian male lifespan (BEFORE covid) and the sanctions only come off the day he dies and not one second earlier...

This defines the rest of Putin's life. Russians are out in the streets protesting again despite ALL the crackdowns. The morale of Russian troops is terrible: one group of deserters said they were told they'd be welcomed as liberators, a group of POWs say they were told the invasion was just more "exercises" until they were well past the border. Putin turns 70 in October: the math on a dictator for life changes when he's 3 years from the average russian male lifespan (BEFORE covid) and the sanctions only come off the day he dies and not one second earlier...

There's a kind of "big boat stuck" aspect as well, a concrete problem where everybody who doesn't agree what the issues ARE is outing themselves as a transparent shill (and even those are backpedaling). Alas this one is killing people and turning children into refugees and has a small but nonzero chance of escalating into World War III, while also bringing a bunch of other looming nebulous sword of damocles issues down into reach of resolution.

So yeah, this is distracting me and draining all my energy. (The weather being too cold and rainy to go to the table isn't helping there.)

I've still got a backlog of other links, as usual: Guillotine the billionaires, the GOP are all hyopcrites, defund the police, the USA's capitalist medical system is useless... And here's a strong argument for school lunch being free for all students: if you're legally required to be there and can't leave, how do they get to charge for feeding you?

Apparrently Apple's "security" is entirely performative. Youtube has a new feature to make sure the next video of police misconduct posted on a brand new youtube account can't go viral on their platform. The united states remains uniquely prudish and easily scammable via sex panics.

Fascists are murdering protestors on US soil again. It's only ever the GOP doing this, with the police equally blaming the victims. (The protests were against the police, the police say the protesters deserved to be shot, pointing out the large number of witnesses contradicting their story makes the situation "very complicated". This is AFTER portland police were documented attacking protestors 6000 times and got censured by the Department of Justice.)

Another day, another leak of secret billionaire bank account info. Who owns $80 billion of swiss bank accounts and what did they spend it on.

Of course the scandal isn't that "they still bank here": adding individuals to "the unbanked" is bad whoever they are. The scandal is the tax evasion and that billionaires anywhere remain unguillotined. "Punching up" is good, "punching down" is bad, even though both actions are punching. Basic income (plus a National Health Service and rent control) fixes most of this: let people afford to do what they want to do. And stop criminalizing sex work: we don't criminalize chocolate harvesting despite modern corporations enslaving children to do that. The triangle shirtwaist fire was about textile workers locked in and unable to leave when a fire broke out. Sex is NOT THE PROBLEM here.

The above article about the most recent billionaire banking leak also details the path by which Visa/Mastercard/Chexsystems achieved a proprietary stranglehold on the US dollar so it can unbank "immoral" customers like marijuana dispensaries and camgirls and escalate into the "war on cash" to eliminate all untracked transactions. I'm hoping that The Great Retirement (followed by The Great Die-off) will take the wind out of their sails, but really the problem of cornering the market and creating oligarchs is inherent in capitalism, going back to the Guilded Age with railroad robber barons in the 1800s, who were only displaced by the spanish flu pandemic, World War I, the 1929 stock market crash leading to a decade long Great Depression, World War II, and finally Sputnik and the "atomic bomb delivered by ICBM that no amount of radar and fighter jets can defend against so we will REORDER SOCIETY because of it". The period from New Deal through Cold War beat back the oligarchs for most of a century, but that century is up. AOC's Green New Deal was not NEARLY aggressive enough, capitalism needs to END, along with corporate personhood and civil forfeiture and the existence of billionaires. (Both as a category and as individuals. Guillotines are designed to divid. Proper tool for the proper job. Until then, Billionaires continue to literally kill people.)

Among the many MANY failures of IPV6 is the way it defeats half the purpose of ssh: every time I connect to a server via an ipv6 address it's a DIFFERENT address, and it goes "Warning: permanently added host key for address BLAH to known hosts" which provides ZERO protection from man in the middle attacks. (I know the "Earn It" act is poised to outlaw encryption in the USA entirely in the name of locking up all the Boomers who took naked baby pictures of their kids and are thus guilty of child porn by insane modern standards, while also denying that one House episode was actually describing a real thing because THESE days nobody does that under the age of 30. But it's still annoying for the infrastructure to get this stupidly wrong in the meantime.)

Grinding away merging lspci and lsusb, so they can share the .ids file loading/parsing/searching infrastructure, and while we're at it the parser for the uevent keyword=value stuff is pretty much the same too. (The lspci DISPLAY logic is nuts, but that's because it's got a half-dozen different output formats. -m, -n, -nm, -nn, -nnm, plus -e and -k modifiers for all that. And -nn puts the data in a DIFFERENT ORDER than all the others, the vendor:device data gets collated. Everything else is string [numeric] with fields optionally dropping out, but that one emits string string [numeric:numeric] unlike anything else. Hysterical raisins.)

This tweet reminds me I miss the non-STEM parts of college. I got an english minor, and had a bunch of courses in sociology and psychology and cultural anthropology and comparative religion and so on, and those were the actually useful ones. (Open source is an example of a potlatch culture!) The computer science I taught myself starting in 5th grade, and only wound up taking classes in it to pull my GPA up after the disastrous foreign language requirements. (The only F I ever got in high school was the second year of German: I remembered nothing from the first year. The only D I got in college was French: my french teacher taught french entirely in french. I tried to explain that if I knew french I wouldn't need to be there, but it didn't work because he wanted me to explain it to him in french. I WORKED for that D. Don't remember a darn thing from it, I eventually traversed the foreign language requirement with two back-to-back summer courses in spanish where the compressed schedule plus grading on a curve meant 50% was a C+.)

Oligarchs try to divert all funding to STEM and away from humanities because STEM isn't a political threat to oligarchy, and humanities are. Silicon valley techbros are politically fungible.

We still need to defund the police. Each misdemeanor arrest costs NYC $1750, so buying and giving away diapers and food would be signifcantly cheaper than policing petty thefts of essentials. And here's police teaming up with fossil fuel interests to stage an attack on themselves as an excuse to evade an eviction order by the native american tribes who legally own the land. They don't have to fool anyone, they just need a paper-thin excuse to "yeah but" silence protests at a casual glance: "Isn't that horrible?" answered with "Oh no, she was asking for it..."

Law enforcement is always protecting the powerful against the powerless. and the silver-spoon crowd are unused to being on the wrong side of that, so every time they meet a bigger fish or swim out of their little pond they cry like babies over the injustice of this thing that has never happened to anyone else before.

The war on cash continues. All purchases must be trackable back to an individual in realtime, and the recipient of the money potentially culled from the financial system. Switching everything digital is a bad idea. Once upon a time you could do the NYT crossword without having your data harvested, but of course Wordle is full of trackers under NYT ownership.

If the Earn-It Act punches a big hole in Section 230, Google's ad business could wind up with an awful lot of legal liability, and they'll still somehow manage to avoid going after the real problems (which existing laws apply to just fine). Meanwhile creeping prudishness continues to have knock-on effects, and the GOP is openly attacking contraception already. They consider taking away abortion a given at this point because of the supreme court packing, they're on to the next misogny. And as always, Boomer Biden fixes nothing.

Scammy right wing loons never stop fighting to chip off the most vulnerable 10% of everyone else.

Hmmm, got a bug report that sha1sum and friends treat directories as empty files, when they should treat them as -EISDIR and skip them with error_msg().

This is actually a library code problem: an awful lot of stuff should refuse to accept a directory in place of a file. There doesn't seem to be an faccess() variant that works on a file, just fstat(), so the check is doable but a bit more heavyweight than I like. (Eh, the overhead of copying a bunch of dentry data into a struct is probably swamped by the system call entry/exit code.) The two obvious places to put this are in loopfiles_rw() or in notstdio(), and although the second is the more thorough fix, adding a stat() to basicaly every open() call makes me wince, and auditing all the calls for correctness (69 xopen and 23 xcreate under toys/ according to a quick grep, plus more in lib/ and possibly main.c) seems uncomfortable. And those are just the DIRECT calls, not through something else.

Speaking of which, there are 48 loopfiles calls under toys/*/*.c (each of which goes through xopen), and just auditing those if that's where I put this fix is... annoying. But that's probably the first place to fix it.

I checked in bash and "echo hello < dirname" is not an error. (No, I'm not poking the maintainer. "The implementation is the specification" is hard enough to cope with, and then he keeps FIXING things I point out to him when I just want to understand why it's doing that.)

Boomers solve nothing. Boomers can never resist letting Russia spend money at them, which is why they're so easily hired to push Russian agitprop. Sedition is just another commercial endorsement deal to them, like hawking shoes or burgers. They keep letting the same clowns cause trouble over and over, without learning anything ever. (Speaking of which, Canadian truck loons are still in the news.) Everything the Boomers did was ultimately self-defeating, and the lead poisoning combined with senility has made them really stupid.

Oligarchy uses racism as a shield and a distraction. (Keri Leigh Merritt wrote a whole book about this in 2017.) Meanwhile actual racists are stupid and spend their time manufactuing victimhood and being profoundly hypocritical.

Quartz' translation of the spotify apologia is weapons grade snark. And here's a a long 4 part thread of actual analysis of the podcasts triggering the backlash. But the rot at spotify goes far beyond Joe Rogan.

Amazon needs to be broken up: turns out Audible's shafting creators too. Abolish ICE. Remove cars. Slumlords are a problem. An excellent analysis of why all blockchains suck. Uber is a scam. Facebook is also a scam. Youtube continues to suck. The great resignation is happening for many reasons. Religious parables are often kind of horrifying if you understand what they made a story about. Defund the police, then defund the police some more. Funds meant for other things keep getting diverted to police.

Don't forget how bad the guy behind the Comstock Act was. historical censorship has always been insanely stupid, but the right wing loves endless creeping censorship, intimidation, voter suppression, and conspiracy.

The earn it act continues to be horrific. First they came for trans people, then they came for sex workers, then they came for gay teens (and teachers), then they came for abortion rights... How any of the "freeze peach" advocates support the end of anonymity and banning encryption in the name of being shriveled up old prudes is beyond me. (Sex workers look out for each other, the reason there's so much trafficing is they're prevented fromm doing so, and the ones who are preventing them from helping each other are the ones victimizing them: the same rich white men who are openly persecuting them are paying for the trafficing. It's the old "homophobe turns out to be gay" issue. No catholic priest gets to weigh in on this issue.)

No it's NOT the stimulus payments: inflation is happening because the cost of shipping things from china went from $3k/container to $30k/container with a month long wait to unload in the port of los angeles, and it goes back down when we finish onshoring/reshoring the manufacturing within NAFTA. Playing with the demand side (via money) only has an impact when supply can't keep up, it's SUPPLY constraints that cause inflation, not demand. The classic example is rare comic books where there's only a few dozen bidders for that issue in the WORLD and if you found a box of 1000 of Batman's first appearance in an attic somewhere the price of all of them combined would plummet to less than what ONE costs now. "Act now supplies running out" is a classic sales line trying to induce demand via the illusion of scarcity. Cornering the market raises the price, it's how Late Stage Capitalism works: when everyone is adequately supplied, the thing being produced is cheap. Only when there isn't enough to go around do prices get bid up. It doesn't matter what the DEMAND is, air is still free and clean water coming out of the tap is cheap enough to shower in the stuff without a timer. Inflation is happening because of disrupted SUPPLY chains, not a change in demand.

The reason I keep coming back to twitter isn't what twitter thinks it is. Unfortunately last year twitter was purchased by a right wing loon, just like the Wall Street Journal and Newsweek were. "Conservatives" are all about censorship. They want everything they say to be mandatory viewing (clockwork orange style when at all possible), but to be the only ones allowed to speak and immune from criticism.

Elon Musk, "incompetent narcissist". He should have it on his business cards. Guillotine the billionaires.

The reason for fiddling with struct string_list yesterday is I wanted to add struct dev_list next to it, because lsusb and lspci are reading basically the same file format for their device ID databases. There's a third layer to pci.ids, but toybox's lspci isn't using it (because it doesn't implement -v, which turns out to be kinda fraught).

But I've sort of changed my mind and decided that merging lspci and lsusb into the same file probably makes more sense? Because I don't see a third caller showing up any time soon, and neither of them is particularly big (each a little over 100 lines).

Meanwhile, I REALLY need to break up ps.c and move the shared code to lib/ps.c, and that's... a can of worms. But triaging commands to do a new round of documention on them (videos) is resulting in various cleanups. As documentation always does. It's easier to fix it than explain it. (Which can be read as BOTH avoidance productivity and filing off corner cases that are too much backstory to digress into during the writeup.)

Reading the C99 specification is no fun. I forget what I was trying to look up but along the way I convinced myself it did NOT require unmentioned structure members to be zero initialized when you did &(struct boing){.potato=37;} (I mean, it WORKS, and you can see the explicit initialization via objdump -d, but I wasn't sure it was RELIABLE because {1,2,3} and {.a=1,.b=2,.c=3} seemed to be treated different by the c99-draft? The confusing part is that section 6.7.8.9 says "Unnamed members of structure objects have indeterminate value even after initialization", but apparently section 6.7.8.21 is the part that says to zero initialize them, and "elements or members" clarifies it applies to both styles of initialization. (According to grep '[)]{ *[.]' main.c lib/*.c toys/*/*.c the only command using member initialization in compound literals is host.c, which I rewrote NOT to do that and then didn't check in because... no, it's fine as is. I think.)

I then tried to look up what "unnamed members" _ARE_ but it seems to be 1) a bitfield thing (might have seen that but I can never trust bitfield endianness and the compilers traditionally generate HORRIBLE code for bitfield access, so I always shift and mask myself) and 2) a way to add padding to a struct? (Never used it, never encountered it. Lots of stuff adds padding to structs but they always call it stuff like char __unused1[2];)

Ah right, I was trying to look up whether the variable length array element at the end of a struct should have [0] or just [] and it's section 6.7.2.1.15 that says it's char blah[]; with nothing in it. (It of course adds some nonsense about access being "undefined" and then immediately subverts it via the examples in 16-18, and I refuse to be baited twice by the same standard in the same session. This one I KNOW, I'm just making sure that removing the 0 from struct string_list in lib/lib.h is the correct thing to do. They're equivalent with or without the 0, but static checkers throw a wobbly and people keep trying to use them...)

Heh, I apparently need a current writeup of my whole "Linux uncontaminated by gnu" plan to make a system you CANNOT call "gnu-slash-linux" because it isn't even built with gcc. I've mentioned this multiple times over the years, I think it was in my 2013 "rise and fall of copyleft" talk at ohio linuxfest for example, and probably on the old aboriginal linux mailing list long before that. Heck, it was probably in my old livejournal back in the busybox days, and a quick google finds people referencing me talking about it back in 2007. And, of course, my original sin that interested me in undoing the damage from the toxic waste spill I'd initially triggered in the first place...

Yay, I figured out how to switch the keyboard backlight in this laptop on and off. (Despite poking the geology building's management at UT a couple months back, they have yet to fix the timer that turns the lights on at midnight, so if I get there at 10pm I'm in the dark for a couple hours, and the keyboard backlight is very convenient.) It's function right-arrow. While fiddling with that, I toggled something with a moon symbol, and something that's three squares fit together. (I was thinking it MIGHT be a keyboard but it's probably supposed to look like the mousepad and the left and right click buttons under it?) I hope selecting whatever it is an even number of times turns it back off so I don't notice some weird new symptom I forget how I triggered days from now. Ah, it turns out there is a webpage explaining that stuff, and most of it doesn't work on Linux anyway. Oh good.

I'm going through lots of little toybox commands that might make good short intro videos I can both get out easily and use to learn video editing. (This is resulting in lots of little cleanups to commands I haven't looked at in a while.) There's a lot of "I need to explain everything before I explain everything else" circular dependency stuff in this, but there's pretty clearly gonna have to be playlists: how to use commands, how commands are implemented, common infrastructure, building bootable Linux systems...

I want to do videos with GROUPS of small commands, because true, false, clear, reset, sync... they're all basically one function call. Another genre is "open a file and call an ioctl on it", maybe with or without a loopfiles(), which covers freeramdisk, fsfreeze, partprobe... (Huh, I have a "detach.c" in my tree that isn't checked in, I should look into that...) It seems a bit silly to have a "how to use unlink" video, and then an "unlink.c implementation walkthrough" when each is under a minute: the implementation is "if (unlink(*toys.optargs)) error_exit("message");" plus the usual command boilerplate and help text.

But at the same time if each command has its usage video and it's implementation video, it's easier to do a table of them. A few commands (ps, sed, sh...) will probably need multiple implementation videos though. I REALLY need to do the breakup work on ps.c to make the 5 commands in it have their own files and move the rest to lib/ps.c before doing an explainer video on that. (The shell and sed are just big, I expect awk and vi to be too. I see tar.c has crept over 1000 lines too, that one's not CONCEPTUALLY hard to explain but somewhat voluminous.)

Everything else (that's completed and not in pending) probably fits reasonably well into a single implementation walkthrough video that isn't exhausting to watch. But even the "how to use" video for sed or sh is gonna be a bit of a marathon viewing if it's all one go...

To avoid burying the lead: people are organizing against the latest monstrosity, and here's Stephen Fry on Graham Norton decades ago explaining the fundamental problem.

So the anti-sex evangelical loons have a new bill to make showing ankle on the internet illegal: the Earn It Act. (Of course the main people stopping sex trafficing are sex workers, but victim-blaming is fundamental to christianity. Eve listened to the snake that God made and let into the garden, so God held a grudge for thousands of years and had to arrange a human sacrifice to give himself permission to (conditionally) forgive their descendants. That is literally the core story of christanity, so if the church admits Eden is a "metaphor" then original sin needing to be "forgiven" by this benevolent god is...?)

Wikipedia blacked out to stop SOPA and PIPA, and white guys spoke out against it, but not FOSTA and SESTA because that was "icky speech". Germany's nazi party started out by treating trans people as non-human, and expanded it to concentration camps for millions of people. Of COURSE the misogynist evangelicals going after camgirls won't stop there, they want to rigidly censor the internet and bring back the Comstock Act.

But the real driver is to force everyone participating in the great resignation back into the crappy jobs they left making plutocrats money. And yes, they're still using other tools like civil forfeiture (except that example was Amazon convincing a friendly judge to steal the money straight from bank accounts), and of course both of the news stories linked from that gofundme are behind paywalls.

The evangelical clowns burning a zillion books are never addressing actual problems. The actual fixes are all undoing damage the GOP intentionally caused in the first place (or at least pushing back against it). There are zero bright lines with prudery, just one big long gradient smoothly shading through every possible grey. The censors will never have enough, and will keep grasping until the Boomers die and we end capitalism.

Speaking of capitalism, Faceboot's cover-up of its role in Cambridge Analytica just blew up in its face. The billionaire whose seed money was inheriting blood emeralds from apartheid south africa might be a giant racist (who could have seen that coming) and he continues to be arrogantly incompetent.

It warmed up a bit, and I went to the table and actually stayed for a while and got work done. (Also, HEB restocked on my lovely checkerboard tea.) Enjoyable evening, on the whole.

It's fascinating, I can never remember the name of the hackermonthly PDF (let alone the URL for the original slightly wrong explanation I posted to the busybox mailing list off the top of my head a dozen years ago) so I googled for "landley usr bin" to find it and...

No, Ken and Dennis did not have floppies, the PDP-7 used a ramdisk loaded from paper tape. (And /usr wasn't "renamed", it was the user directories with a three letter name like bin, etc, lib, and dev because the pdp-7 had 1024 18-bit words with half running the OS and half being used as a ramdisk so yes the extra letter counted and they kept the name when they moved to the pdp-11.)

No, dynamic linking did not solve any problems with static linking, it made things WORSE so that the proposed "mount this, then that" approach wouldn't work reliably. (And yes, the a.out executable format had shared libraries back before ELF.)

Obviously I can't really throw stones when my own first stab at the explanation got the disk sizes wrong. (I remembered the TOTAL size but not how it was partitioned, nor how MUCH faster the tiny system disk was than the external disk packs. In modern terms they filled up their SSD so leaked the system onto the external USB disk holding user files, and then got a second USB disk later to move the user files to so the system could have all the first one.) That's why I went back and corrected it when a magazine asked to republish it, and gave links to primary sources at the end.

I have filled out the "Embark" online paperwork. I already know one thing I got wrong, but that's normal for me filling out paperwork: back in middle school I expected about an 85% on tests where I knew the material cold, just because it was paperwork so I would of course fill some of it out wrong, miss the back of one of the sheets (of the instructions if not the test), and then hand it in to the wrong place. (Programming is all about iteratively fixing the thinkos when the machine points out where you got it wrong, sometimes backing up and frogging entire sections when something's misaligned.)

To relax, I'm trying to get toysh to match bash's behavior for moved and deleted current working directories. I already added a test:

mkdir -p one/two/three
testing 'cd in renamed dir' \
  'cd one/two/three && mv ../../../{one,four} && cd .. && echo ${PWD: -9:9}' \
  '/four/two\n' '' ''
rm -rf one

Which the host passes but toysh doesn't, and the logic is sadly a bit tangled. But then bash is a bit confused too. When launched in a deleted current working directory, bash goes:

$ env -i bash --norc --noprofile --noediting

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

bash-5.0$

Except I ran it under strace and piped the output to less so I could search it (even with --noediting bash and less are trying to read from the same tty and getting alternating letters until you hit control-C which kills the bash instance but not the less instance for some reason; probably gotta make my less implementation do that too eventually) there was only ONE call to getcwd() (with a fixed size 4k buffer instead of an automatic allocation), so I have no idea what it's talking about "parent directories" plural?

I added support for "cd -" while I was there (and already emailed Chet back on January 20 to add it to "help cd" in bash because seriously, it's in posix: 'cd -' is 'cd "$OLDPWD"' the same way 'cd ~' is 'cd "$HOME"').

It was already exporting $PWD and $OLDPWD only on the first cd, because that's what bash does (and yes I should add a test):

~$ cd sub
~/sub$ cd ..
~$ unset OLDPWD
~$ cd sub
$ declare -p OLDPWD
declare -- OLDPWD="/home/landley"
~/sub$ declare -p PWD
declare -x PWD="/home/landley/sub"
~/sub$ unset PWD
~/sub$ cd ..
~$ declare -p PWD
declare -- PWD="/home/landley"
~$ env -i bash --norc --noprofile -c -- 'cd sub && env'
PWD=/home/landley/sub
SHLVL=0
OLDPWD=/home/landley
_=/usr/bin/env

It SETS them both each time you cd, but only _exports_ them the first time, and just does a local assignment the rest of the time. And I do mean "local":

$ x() { local PWD; cd .; }
$ unset PWD
$ x
$ echo $PWD

So I think my code was already getting that right (I need to add more tests) but before it was wasting an entire int for this in the TT globals block and I switched it to using the high bit of TT.options, and collated the export statements under the same test.

When I say I'm not just writing a shell but specifically a BASH replacement, I'm SERIOUS about that. My spec is not posix, or even the bash man page. It's WHAT BASH DOES. Tied to the comfy chair, poked with the soft cushions, and added to the regression test suite. I'm reluctant to communicate with Chet about stuff because he keeps FIXING THINGS I POINT OUT and thus changing the behavior. Which probably breaks somebody's script somewhere...

Still not sleeping well, because anxiety. I did not used to have anxiety. It comes and goes, but having it lurking is INCREDIBLY ANNOYING.

Corporations are not your friend, and regulatory capture is a thing.

Intellectual property law destroys intellectual property. Ten years ago the netflix DVD library had 100,000 titles in it, but the online streaming version they transitioned to keeps losing content and these days has 15,000 and shrinking. All the services are like that: Youtube just showed me a clip from something called "Sakura Trick" which Google says came out in 2014 and was on Crunchyroll, but of course it's not on Crunchyroll anymore. It's about two high schoolers dating; if they were heterosexual it would be "cute" but if they're gay cue the pearl clutching about corrupting the youth. Me, I'm thinking it might be better "I want to learn japanese" background noise to put on without subtitles than Hulu's run of the original Sailor Moon, because watching Usagi murder people and be lauded for it "because protagonist" gets old. Each week she kills somebody who was passing as human just fine at the start of the episode, and I have yet to see any of her opponents actually permanently harm anyone? As I said, watching without subtitles so I couldn't tell you much about the plot. It's hard to find things that aren't so bad they're unwatchable but aren't good enough I turn on the subtitles to follow them after a few minutes.

Of course the Boomer extinction burst of evangelical prudishness is cover for late stage capitalism misoginistically shoving self-employed young women back into retail jobs, and the war on cash (no untracked purchases for-profit corporations can't individually veto and PUNISH you for trying!) has literally escalated to police seizing armored cars. (And no, crypto solves nothing.)

I'm confused by Mark Cuban. He's a billionaire, and he's maybe possibly doing a good thing? Most likely because he wants to run for president (which should DEFINITELY NOT HAPPEN), so it's really for his own personal gain, but... I wouldn't want to STOP him from subsidizing affordable medications, if it is true? It's not Mackenzie Scott's get-out-of-guillotine free pass (checkboxes: did not voluntarily become a billionaire, is convincingly and consistently attempting to STOP being a billionaire, has embraced Uncle Ben's Spider-Mantra in doing so). Mark Cuban isn't absolving himself of the slumlord guilt of having taken a billion dollars from poor people and hoarding it (starting by giving it BACK), he's doing the "little people" a favor by pushing back against a specific societal problem that plutocrats have been consistently exacerbating since 1970 (here's a lovely writeup on that). But that IS one heck of a favor... Maybe it means he goes to the back of the line leading up to the guillotine? Hmmm... (Obviously it's not my call, but I'm trying to figure out what to lobby for. In a just society, hoarding a billion dollars while anybody is homeless would be a capital offense.)

This is an unusual exercise because most billionaires constantly broadcast how bad they are, and even PR-minded twerps like Elon Musk are much clearer: sure Muskrat's cheerleading the advance of cool tech, but none of it's happening because of him. He bought Tesla, he bought Solar City, he bought Maxwell. All the advances in electric cars and solar panels and batteries were already happening and he literally purchased credit for other people's ongoing work. ("Oh but they might not have been sufficiently funded otherwise" that's only because for a hundred years the USA massively funded basic research but we STOPPED when Reagan lowered the anti-billionaire tax shield until plutocrats could hijack the government and iteratively "drown it in a bathtub". The Boomers have been eating the seed corn since they took over.) Musk's assets have been mortgaged to the hilt for decades, all of Tesla's money (like Uber's) comes from friendly financiers willing to hand him tiny slices of the $21 trillion the USA printed after the 2008 mortgage crisis and gave to the BANKS rather than using it to bail out any homeowners or students with loans or the piles of predatory credit card debt... That man is a scam artist. Adam Something and Common Sense Skeptic have multiple video take downs of his endless dumb ideas. Ford's F150 Lightning is still restricted by the dealership model, even though they're inching away from that model they're hamstrung by stupid protectionist laws from years ago, and nothing Musk did to punch holes in that for Tesla fixed the dealership problem for anyone else, because he's not ABOUT fixing anything for anyone else. He's about getting a spotlight shone on him as a savior because he inherited african blood emeralds during Apartheid which gave him seed money to hang out with finance bros during the 1990s dot-com boom.

But even without the threshold being "net positive", it's rare to see a billionaire where guillotining them immediately would have ANY discernable downside. Encountering a second brings us from "Spiders Georg" territory into... black swan event maybe?

Walked out to the table for the first time all week. It's been below freezing each night, and the UT Carrilon plays "I've been working on the railroad" (no really!) twice through very slowly starting at 9pm, but in between there's a few hours were it's quiet but not TOO cold yet.

Didn't really get anything done, but oh wow I needed the exercise. Couple hours of walking. Listened to podcasts in the wifi headphones. Good times.

Various links from my talk went up on the golug website. Basically stuff I'd pasted into the chat channel. There's no archive of the video that I can find, and the page itself doesn't seem to be set up to have a persistent link to any of the entries. Presumably archive.org will snapshot it at some point? Still, I'm glad there was interest, and I need to make my own videos on this.

Some good political news: multiple federal courts have struck down republican gerrymanders in upwards of 5 states. (Doesn't stop the voter suppression, propaganda networks, replacing the people doing the counting with partisan hacks... But still, step in the right direction.)

So it looks like logging into Google Embark (their contractor payment thingy) with my main email address has a hiccup, namely that I break everything.

I've had the google account my email is spam filtered through for many years, through multiple upgrades of Google's systems, and it's... kinda borked. For example, to this day I have to log OUT of google to use maps.google.com because otherwise it's not an "authorized service".

Elliott recently pointed me at a bug in issuetracker.google.com (that's what those /b/ links translate to for the rest of us) and I should probably log in to comment on it, but I used my "work account" (at coresemi.io) to view it. The gmail account my personal email goes through is some sort of "google domain" organizational thingy set up in 2008-ish by a friend who has since undergone a religious conversion and moved to Europe. Once upon a time gmail wouldn't accept mail for other domains unless without special setup, and if I ever had the domain-side login to that it would be a half-dozen laptops ago.

Both the organization account and my previous individual Google account had the login "landley" (yes, at the same time: initially different namespaces), and when Google collapsed them together it had a name collision and got itself VERY confused (meaning I couldn't use Google Plus for the first two years of its existence), and Google periodically sends me emails like this one from Thursday:

We are writing to let you know that your G Suite legacy free edition will no longer be available starting July 1, 2022. To maintain your services and accounts, you need to upgrade to Google Workspace.

Which I have historically ignored and they auto-migrated whatever they had to and email still worked so I left it alone, but I do not consider an account with a pending Windows Scheduled Shutdown timeout to be load bearing, and if it STOPS passing email I move the DNS record to dreamhost's mail servers and update the pop3 and smtp server addresses in my client.

I've been thinking of preemptively pointing my domain's MX record at dreamhost's servers anyway to cut the gordian knot, but to be honest have been reluctant to touch it. And now I need a Google account to use Google vendor stuff and should probably interact with android's bug system directly and...

The thing is, Google's "Embark" draws a bright line between "gmail" and "non-gmail" addresses for some reason, and moving the MX record in dreamhost's nameservers (where the domain is registered) doesn't clean out the account association on Google's side: it still thinks "landley.net" goes with the 15 year old fused pair of gmail accounts that have undergone multiple system migrations. The tutorial video warns that if I try to tell Embark my email is an external address when it smells gmail, it will throw an "email mismatch error".

I could use my phone's gmail account, but my phone has NEVER had authorization to spend money, except for one fixed-size gift card I fed it back in Milwaukee. (The problem with programmer paranoia is it tends to be justified, not that intel is any better. It's not that the phone is inherently less secure, it's that the always-on orwellian tracking device with the whole-room microphone, camera pointed at the user's face, and second-by-second GPS tracking is a much higher profile target, and reusing google accounts between phone and payroll smells WAY too much like reusing passwords.)

Besides, until recently I just created a new gmail account for each phone. I kept it for the most recent phone because I remembered the login/password and because I wanted my youtube music playlist, but youtube has lost that capability and no longer lets you listen to two consecutive songs without 60 seconds of advertising in between. Youtube has taken away the POINT of having a persistent account for phones, and I'll probably just create another one once the Pixel 3a stops being supported (or allowing battery replacements).

I asked Elliott if there was somebody at Google I could talk to who could maybe apply cleansing flame to the situation, and he's never even MET anybody who works with gmail, and doesn't believe there are any at the Googleplex. (My own theory is it's entirely automated.)

Anyway, if you're wondering why I compulsively document and make everything be a self-contained reproduction sequence script with minimal external dependencies and regression tests... (And also why I use a vanilla unmodified distro on my laptop with a fresh reinstall each version upgrade and a written checklist of setup changes, and why I leave my phone running the stock Android version with nothing sideloaded.) Bit-rot is sadly real: the context shifts out from under you, the space something lived in stops being maintained, migration copies symlinks but not what they point to (at a dozen different levels of abstraction), they back up the data but not the reader/parser...

I got innoculated aginast this early, by the time I'd moved from a C64 to Amiga to DOS to OS/2 to Linux I was ALREADY expecting "this too shall pass" and becoming a digital pack-rat. But it's hard to be paranoid ENOUGH...

February 3, 2022

Austin is paralyzed by an ice storm. HEB is open but understaffed, which is ok because there's almost nobody here. (And then they closed at 5pm.) This is nothing like last year's blizzard, though, this is just our version of a snow day, we had a worse ice storm than this the first year I moved here. (That was over an inch of ice on everything, this is like, a third of an inch tops. Makes the trees pretty, but they'll live.)

February 2, 2022

HEB was positively SLAMMED today because there's an ice storm coming, so the line to use the self-service checkout machines ran all the way around the deli and into produce aisles. They apologized for being understaffed (despite offering $17.50/hr to start according to the signs), but it's panic buying. Last year we went down under 10 degrees celsius and STAYED there, all the pipes froze including the gas pipes leading to the turbines generating electricity: it comes straight up out of holes they drilled into the pipes with no processing, which means they didn't bother to remove the steam and such mixed into it, which means if it's cold enough ice plates inside the pipes until they close up.

Meanwhile solar panels are self-de-icing if you install them at a slight angle, because the parts that aren't lit up act as resistors and heat up when current from the lit bits tries to flow through them, so the snow on them melts at the bottom until it slides off, and the newly revealed bits collect more electricity to heat up the remaining parts faster until it's all clear and dry. (This is why casting a shadow over part of a solar panel disproportionately reduces the amount generated, and they use microinverters to wire them up individually instead of in series when they can, so the shaded ones merely don't generate instead of consuming the power produced by the other panels).

But that was a hard freeze lasting days. This is just getting down into the 20s overnight (below freezing but not by that much) with rain putting a thin layer of ice over everything, and then going back up above freezing during the day and then down into the 20s again the following night (repeat for most of a week). That's not even climate change, Austin was doing that when I moved here in 1996. (We have six weeks of winter every year, they're just not consecutive. Last year we got in trouble because two of them WERE consecutive: the destruction of the Polar Vortex means nothing's keeping the the arctic winds up where they belong anymore. So the ice up there doesn't get built up and maintained, and all the cold leaks down here where it's just gonna melt again immediately. Leaving the freezer door open does not mean your frozen peas are going to last longer just because it feels so cold in the kitchen. It's a BAD sign, not a good one. The cold should not be down here, it should be up there: it went walkabout because the oil companies screwed up the weather.

February 1, 2022

Turned 50 today. Forgot until well after I woke up, and then wound up taking the day off and (let's be honest) moping.

Fade ordered me a fresh set of headphones. Yay. (I do break them every 6 months or so; they have planned obsolescence plastic hinges that are always the first thing to go. If those were metal I'd probably still be on my original pair. Capitalism again.)

January 31, 2022

The plan for today was to file new LLC paperwork at the Secretary of State's office when they opened at 8:30 am, but it was pouring out when I woke up. I tried to wait it out (the #10 bus picks up across the street but I still have to walk several blocks downtown and then I'd planned to walk to UT from there to work with my laptop), but by around 2pm it was clear there wasn't going to be a good gap so I went anyway.

Talked to the LLC specialist in the little booth for half an hour, and came away with a stack of paperwork to fill out and return. They say that with covid it could take up to 40 days to process, although I can bribe them an extra $25 to cut that in half, so that's nice.

I have to do name clearance searches for a new name because my old LLC from 2008 (Impact Linux) was apparently never properly shut down and still sort of zombie-exists but not in a useful way. (It went "inactive" in 2011 due to unpaid annual continuing-to-exist fees. The person who handled that at the time was born again into one of those religions where you get a new name, dress differently, and move away from everyone who knew the old you.) At this point properly killing the old LLC off means fully resurrecting it first, which is MORE paperwork and fees than just starting over with name clearance searches for a new name, which is like finding an unused domain or twitter login, only worse and probably requiring trademark searches too (which the initial state search doesn't do). Decisions, decisions...

Got caught in the rain on the way back, dried off at the Wendy's in Jester Center. Which is open 10am to 5pm, and for once I was there during daylight hours. Bought a new frosty keychain and everything, although sadly I didn't have those coupons with me. And then trying to walk home from UT during what SEEMED to be a break in the rain, it started pouring enough I gave up and asked Fade to order me a Lyft while huddling in a parking garage. (I haven't got the app on my phone; nothing on there is authorized to spend money. Yeah, I should have just taken the bus home, but it LOOKED like a gap in the rain. Tricksy weather.)

At the table, but... really tired. I am not load bearing when it comes to ANY additional stress right now, not even GOOD stress. I am highly pleased to be starting work on Google's toybox hermetic build todo list, but tomorrow I have to fill out paperwork with the state of texas (can I reuse the name Impact Linux from 2008, or do I need to find another one that passes a conflict search), and then do I need a federal EIN or not? I finally got the tax envelope from Triple Crown (now that the box of stuff has returned to Stenograph; packed it as well as I could with those air baggies from amazon and wads of paper, but that thinkpad never had a case, I hope it made it there ok) and need to file taxes for 2021 soon, I hope Juanita can tell me how to do the "passthrough ignore" thing for a single person corporation.

I want to get work done. I need to get a lot of todo items cleared. I wrote a bunch of things I need to remember to do on the back of an envelope this morning, and of course left it at home.

I did a little programming doodle yesterday, which was great stress relief. That's the kind of programming I do for fun. I was going to do a quick ldd and posted about it on the list in hopes somebody had a trick to make dlopen() work from a statically linked context (ideally to portably poke a shared library loader to tell me about foreign libraries in a chroot context; I want the load address and maybe whether all the symbols resolve, but I don't need to USE the result), and Elliott reacted like I'd tried to poison his cat. (We've been in a long thread about it all month.)

My working style has always spun off todo items faster than I can close them, and the really satisfying work sessions are the ones where I can close tabs, finish stuff, document it, and ship it. I'm trying to clear my plate before diving into a new chunk of todo, and... I'm having the programmer vesion of weak noodle-arm can't lift anything. I'm TIRED.

I'm about to sign up to serve two masters, because Jeff's doing the J-Core ASIC that I REALLY WANT TO SEE SHIP, and Google's finally ready to get serious about toybox in a way I REALLY WANT TO SEE FINISH. And first I have to work out the division of my time between them (both want me full time; I would LOVE to do both full time) and do a bunch of paperwork.

I need to do more toybox videos, those should probably be weekly. And I signed up for a lightning talk at golug on wednesday (the topic is an intro to mkroot), and Jeff and I are speaking about making an open ASIC at Japan Technical Jamboree #78. Both online via different streaming clients, of course.

Still haven't decided if I should submit an ELC talk. My "not sure I really want to go" talk strategy has been to submit ONE proposal instead of the usual four or five for the committee to chose from, but then I feel rejected when they don't select it. I'm aware that makes no sense, and yet I know my failure modes. And coping strategies. I have decades of experience representing a project rather than myself, and met my wife when she was working the SJGames table at various conventions I was pitching Linucon at. I could be poised and confident pitching the convention to potential vendors because it was a great convention and other people not seeing that was their loss. I was just coincidentally creating it, the work stands on its own. But "would you like me to talk" can't avoid being at least a little about me, it's not a context in which I can easily intellectualize emperical metrics to moot a subjective judgement, and I can't dismiss it with "this person's opinion doesn't matter" when I'm the one who asked to be judged.

(Oddly, if I submit 4 topics and they choose none of them, it's easy to dismiss the conference because clearly our interests do not overlap. It's less of a personal rejection. I didn't ask them to trust my judgement about what would make a good talk but offered them a menu, and they didn't want anything on that menu because they don't get why any of that is important. Ok then.)

Sigh, I need to vent but I'm too tired. Fuzzy and I tried to walk through the Wendy's drive-through today (corporate is still mailing out coupons) but they were too understaffed to serve food. (When we knocked on the little window, the guy behind the register ran away, and the woman who (eventually) answered said come back in half an hour.)

Here's a good story about how the last wave of american prudish censorship got beaten back in the 1980s (between panics about dungeons and dragons being witchcraft and AIDS being a judgement from some god or other). When television ruled the culture they got a strangehold on broadcast television and vetoed every image and word spoken. Now that late stage capitalism has its tentacles everywhere they're making moral judgements about every dollar spent and rendering those they deem insufficiently pure "unbanked", and the gerontocracy's promises about properly funding the IRS to get billionaires and corporations to actually pay taxes turn into monitoring the poor to make sure the have-nots pay every penny of tax on every tip and side hustle. (Of course the increase in the IRS's budget is all about punching down, not up.) Here's a good link about how plutocracy is a problem from my wife's tumblr, and Dr. Sarah Taber pointing out that the war on camgirls is part of plutocracy pushing back against the great resignation: how dare anyone pay their bills WITHOUT flipping burgers for a large corporation.

A clear explanation of how modern christianity turned into a toxic death cult of doomsday preppers, and thus why church attendance is plummeting.

David Graeber talked about "direct action", which takes lot of forms.

YES! Google has approved funding to sponsor toybox work! (Specifically to advance the "hermetic build" work I've been poking at for many years.)

At the start of this (a full 6 months ago) Elliott didn't think I'd need to resurrect my old LLC, but now that we're actually doing it... yeah, down to the Secretary of State's office monday morning to fill out paperwork.

And I'm still hip deep in the J-core ASIC work and don't want to leave Jeff in the lurch, so I'm presumably doing each half-time for a bit? No idea how long that's going to take, but I want to see both these things FINISH.

Speaking of which, back to explaining how hardware toolchains work:

The toolchain we're currently trying to make our ASIC mask with is qflow, which is a set of build wrappers around four other tools (greywolf, netgen, qrouter, magic), plus then you need to feed it the fab's #include file with the standard cells for the process you're actually trying to build a mask for.

I mentioned last time how insanely proprietary most fabs are: their SDK (except it's a PDK here, Physical Development Kit) is TOP SECRET. Various NDA material has leaked over the years (especially for the older fabs that have been passing this info around for decades), but if you present a self-made mask to such a fab without a good explanation of how you got it, they will be EXTREMELY unhappy with you. The fab WANTS to get your netlist and make their own mask, because that's a service they can charge you for, and sprinkle in lots of per-chip royalties for their libraries, and it means your design is completely unportable so if you ever want to fab it through a competitor you basically start over from the netlist and pay all over again.

BUT: back at the start of the pandemic Google paid one fab (Skylake) to publicly release their PDK data without an NDA, for a complicated but powerful 130 nanometer fab process with its physical facility up in minnesota somewhere. This includes (quite a large) standard cell library, and corresponding process data such as how thick the P in a P/N junction has to be and how close two wires can be together before they interfere. (There's hundreds of weird little corner cases worked out by scientists in lab coats when they were designing the fab process, which are usually part of the NDA data without which your placer can't place and your router can't route. Sky130 is extra weird because the layers are different sizes: five volts up top, 3 volts in the middle one volt down at the bottom, so it can do a lot of flexible signal processing stuff and is frankly a bit odd to make a CPU with.)

Before this, the other main option was the OSU standard cells, which a university made years ago using extremely conservative estimates for a very old process, and would PROBABLY work... if spaced widely and clocked very slowly. And original targeting 350 nanometers, I think? It's a terrible option, but at least does not involve an NDA, and was used to develop a lot of these open tools back in the day. And I've talked about Graham Petley's work before, but it's more a generator than a ready to use PDK package, and the guy who did it apparently retired to paint watercolors. Jeff was involved in that project back in the day and could pick it up again, but there's a lot of heavy lifting to actualy use it in a real modern process and we'd like to establish a working baseline first. But the advantage there is you could (in theory) assemble an PDK for any process from maybe 25 nanometers on up. Below that it gets weird and quantum, and they cheat by turning the cells on their side to pack them more tightly together...

The standard modern file format for this stuff is LEF (Library Exchange Format) and DEF (Design Exchange Format). The PDK is generally LEF and DEF files, and the final mask file is a big DEF file. The netlist could be supplied as a DEF file instead of as verilog but usually isn't for historical reasons, and qflow isn't JUST a pile of glue scripts but also has a half-dozen little C programs to do file format conversions.

So, back to qflow: Greywolf is the placer, qrouter is the router, netgen does multiple things (most notably collating the list of labeled pins into a list of networks), and magic is the design rule checker (and it's a GUI visualizer, and format converter, and it actually calls back to netgen to perform some of those design rule checks).

The input to qflow is a netlist describing the circuit (which can be a bunch of formats but verilog is popular). This is still literally just a list of components with pins, and a corresponding list of "nets" which are just groups of pins to wire together. (I was calling them "pin pairs" until Jeff corrected me that there can be 3 or more pins in a net, all of which get wired together so current flows between them all. So networks, not just wires.)

The output is a "routed netlist", a bit like a PDF or photoshop file (except in DEF format) with a bunch of layers each containing a bunch of rectangles. Each component from the input netlist has been assigned to a physical location (layer, x, y, and orientation: greywolf places cells in a grid but can rotate or flip each one to make the routing work better) and broken into the series of rectangles of each type of material that implement that component. Each net from the input netlist has also been filled out with the set of rectangles (on metal layers, plus the vertical bridges connecting layers) needed to route the connections (I.E. the output of place and route).

The rectangles on each layer are all filled with the same thing, like "metal" or "insulator" or "dopant" (the chemicals that turn silicon positive or negative) or vertical connectors between layers punching little squares in the insulator layers. These layers are defined in the fab's PDK, and the PDK also has rules for each layer describing how thin the rectangles can be and the minimum gap needed between them, and the design rule checker's job is to verify that none of the PDK's rules are violated by a given DEF file generated by the toolchain: it checks the output for errors after you've generated a mask candidate.

The components don't just connect to each other, they connect to power and ground and clock. On the outside of your chip is a "pad ring" with special cells that talk to the outside world; there's generally a separate pad ring generator that produces all that, and then your place and route fills in the space inside the ring and links up the relevant nets to the ring's connections to satisfy all the references to the outside world.

The hardware version of QEMU is called spice. It's a physics simulator package that can actually "run" a circuit in a mask definition file. (Very slowly and unreliably, and it hasn't benefitted from 30 years of Moore's Law the way the other tools have, although there are ongoing attempts to rewrite it. But if the stars align and you chop your chip into tiny enough pieces, it can show you little pieces of your circuit working before you spend $$$ to fab it.)

So that's what we're TRYING to get the ASIC toolchain to do: combine a verilog netlist (produced by GHDL from the ICE40 J1 repo) with the sky130 PDK to run it through the qflow tools to place and route a mask file. We also need some libraries, most notably an SRAM generator (what's wrong with openram is its own whole writeup, but the tl;dr is it SHOULD have a few bits of SRAM available as a macrocell we can stick in the PDK and place and route instances of like any other component, and instead it tries to takes over a chunk of silicon wafer space and produce raw rectangles, routing its own power lines and everything (badly). In software terms, we want some C functions and it gives us a .o file with a giant binary blob in it that isn't even -fpic).

One persistent bug in qflow is that verilog uses an insane escape sequence for names with punctuation in them: it sticks a backslash on the front and a space at the end (which either gets stripped so the symbol no longer matches, or causes a syntax error, in any other format the symbol name is copied into). The proper thing to do when converting from verilog to any other format is to strip this escaping, and re-escape any characters that need it in the new file's format. (So far, we haven't hit a case where anything WOULD need to be escaped anywhere else, it's mostly that verilog hasn't got a lot of concepts VHDL does so things need to be made very explicit for it, which involves creating compound symbol names to represent components of VHDL objects, and GHDL uses an "illegal" character (period) to glue those together to avoid potential in-band signaling conflicts. Period is a valid name component requiring no escaping in all the other file formats EXCEPT verilog...)

What qflow decided to do instead (which is nuts) was keep the leading backslash and turn the trailing space into another backslash. And it didn't do it CONSISTENTLY, lots of names leak through unconverted. Sometimes this broke the reader of the new format, and other times it meant bits of the design didn't match up (mostly broken nets because it couldn't match up pins when the converted and unconverted names didn't match at each end). The other problem is, while LEF and DEF don't seem to escape anything and just treat backslash as a normal character... json doesn't. And yes, one of the conversions is into and out of json, because that's what some tool wanted. So the json parser threw a syntax error trying to read the file, which we hadn't noticed before because it didn't get far enough to HIT that failure mode before we fixed several of the other conversions. (The name mismatches were showing up as design rule check failures, now they're showing up as a build break, and this is PROGRESS. Sigh.)

So the PROPER fix is to go back and strip out ALL the escaping: when the vhdl reader library encounteres an escaped name, it should use the escaping to parse the name, then strip it. Unfortunately, this both means digging through qflow's vhdl code to figure out WHERE to strip the escaping (it needs it to figure out where symbols end during some of the parsing), AND it means ripping out all the wrong conversions later turning the "\name [123]" format into "\name\[123]". (Whether or not the [123] should be part of the resulting symbol is one of those things I need to dig further into, but... probably? Work out "what would this do if the name didn't need to be escaped" and make it do that.)

It would have been easier if the earlier code had been at all consistent or systematic, but there's a lot of whack-a-mole fixes that happened here over the years, which need to be ripped out again.

Qflow was apparently trying to preserve the escapes because one of the big Design Rule Checks at the end makes sure the input and output files contain all the same components (nothing got dropped), so it wanted a conversion that was trivially reversible. But the question "how do you know when to put the escaping BACK so it looks like the original" was THE WRONG QUESTION to ask. You don't, you consistently strip escapes EVERY time you load data from verilog: you now know the length of the symbol because it's in a C string, and you don't CARE what punctuation is in a null terminated C string. Teach the design rule checker to strip the escapes when it loads verilog, so it doesn't HAVE to escape the data it loads from the DEF file.

And if you DO need to write the data out into a new verilog file, you know to escape that symbol name BECAUSE IT HAS PUNCTUATION IN IT. Same way the escape wound up there in the first place when the original tool wrote out the first verilog file. You don't have to preserve a decision an earlier tool automatically made, you just have to competently make the same decision again as needed.

So yeah: qflow is a set of build wrappers around four other tools (greywolf, netgen, qrouter, magic), and then spice (or Xyce) is a physics simulator package that can actually "run" a mask (badly), and it doesn't quite work right with verliog that was generated from VHDL, and it doesn't quite work right with the sky130 PDK. We're working on it.

Poking at toysh. Combining command history with continuations is horrible. Both of these multiline input patterns:

$ for i in a b c
> do echo $i
> done
a
b
c
$ for i
> in a b c
> do echo $i
> done
a
b
c

Produce the same for i in a b c; do echo $i; done history entry when you cursor up. The newline between "for i" and "in" does NOT take a semicolon, and the code stitching the multiple lines together into a single long line knows that. This means we have TWO line continuation types: ends current command and does not end current command. No wait:

$ echo $(
> echo potato)
potato
$ echo "
abc"
abc

When you cursor up, the history entry PRESERVES the newline in both those cases. So three states so far. And you can cursor left and right past the line break just fine, which is why I wanted the code I'm writing to be the basis of my text editor implementation.

Sigh, what does busybox ash do... keeps everything entered as separate lines in the history and does no stitching together. Makes sense, but a lot less useful for rerunning things.

Alas, that's all the time I have for this today. Jeff needs me to fix vlog2Spice and vlog2Def the way I fixed vlog2Cel yesterday, and then look at the other project he found that's also trying to use VHDL with qflow to build a chip, and see if we can collaborate at least on toolchain fixes...

Working on the qflow stuff (toolchain for making our own ASIC). Trying to debug various design rule verification failures, and work out what magic's display output is trying to say when there are a bunch of layers... Ok, back up: I should do videos about this stuff too.

We've had a fully open source "VHDL to bitstream" toolchain for a while, which compiles human readable VHDL source code into a "netlist" file describing the circuit, and then the toolchain links that netlist into a bitstream which you can load into an FPGA to run your circuit.

The fully open source toolchain uses GHDL to parse the VHDL source code, and yosys to produce the netlist, then feeds the netlist to icestorm to turn it into a bitstream for Lattice's ice40 or ecp5 FPGAs.

We've recently updated this bitstream toolchain: the previous version was using a package called ghdlsynth-beta that's since been merged into one of the other packages. Unfortunately the new one reads a vital component via dlopen() and thus can't be distributed as a statically linked binary, which isn't exactly an improvement. But anyway, open source FPGA bitstream generation toolchain (for lattice): solved-ish problem.

An FPGA is primarily made up of LUTs and switching fabric. Switching fabric is a bunch of wires connecting all the inputs and outputs of every other circuit in the cell, which run together into junction boxes full of programmable switches that control which lines gets plugged together or kept apart.

LUT is short for Lookup Table, which is a tiny amount of SRAM with a set of input pins and one output pin, so each possible set of inputs can produce a known output (and thus act as any combination of and/or/nand/nor/not gates turning a set of binary inputs into binary output) just by using the input pins as address lines to look up that bit in the SRAM.

Historically there were 3 big FPGA vendors: Xilinx, Lattice, and Altera, but Altera was bought by Intel in 2015 and isn't available to individuals or small business. The J-core stuff builds for Xilinx and Lattice FPGAs.

Lattice uses 4-input LUTs (each indexing 16 bits of SRAM), and Xilinx uses 6-input LUTs (64 bits of SRAM). Neither's really better, it's just the granularity with which you can program the circuits, but it does mean that 1000 Lattice LUTs is potentially less circuitry than 1000 Xilinx LUTs so you can't compare them directly. (Although it also depends on how efficiently the toolchain's optimizer uses those extra inputs and how many are left unconnected, so it's not a strict ratio either. It's ballpark estimates then "try it and see".)

Xilinx was a higher-end FPGA vendor than Lattice, but Lattice has been creeping upmarket. The ice40 line maxes out at 7680 (4-bit) LUTs, while Xilinx's "LX9, LX25, LX45" were how many thousands of (6-bit) LUTs each contained. But Lattice's ecp5 line scales up into tens of thousands of LUTs, albeit at a slower clock speed than the high-end (much more expensive) Xilinx chips. J-core runs around 66mhz on cheap (Spartan 6) Xilinx and 100 mhz on more expensive (Kintex) Xilinx chips. It runs between 12mhz and 20mhz on various ICE40 chips, and isn't expected to be much faster on ecp5 (although we can do the full J2 SMP on there). This also makes things like 100baseT ethernet, GPS correlators, and USB2 noticeably harder to implement on Lattice.

One reason Lattice is hobbyist-friendly is price: an ICE40 is $3.5-$5 even with the chip shortage. An LX45 Spartan 6 is $35-$50 depending on speed, and a similarly sized Kintex is easily twice that. (And that's just the bare chip, bought in volume. Then you need boards to put them in, and Xilinx needs more expensive boards.)

But the real advantage of Lattice is the community has reverse engineered how to program it. Xilinx spent a LOT more on lawyers to sue anybody trying to understand its IP, so you can program Lattice with open source tools quite well (we've completely discarded Lattice's proprietary FPGA toolchain), but the Xilinx equivalents are buggy and incomplete and only furtively developed when Xilinx isn't looking.

So you compile VHDL to a netlist and link it into a bitstream that loads into the FPGA to program the LUTs and switching fabric (and initialize any other SRAM blocks). Great. What's a netlist?

A netlist is actually two lists: a list of hardware components (such as "and gate"), and a list of the named I/O pins sticking out of each component. The linking-ish phase of the hardware build is called "Place and route". The "placer" assigns each component of the circuit into one of the FPGA's programmable cells (including all those LUTs). The "router" programs the FPGA's switching fabric to connect up the various inputs and outputs of each LUT (by matching up the pin labels in the netlist and wiring together everything with the same name).

Conceptually, a hardware toolchain is a bit like a software toolchain. A C compiler produces a bunch of instructions bundled into functions and variables bundled into structures, and then the linker glues them together using a bunch of annotations where thing X needs to connect to thing Y. (In static linking this is all done at compile time, in dynamic linking another program does some of it at load/runtime.)

In a hardware toolchain, the compiler produces components (called "cells") instead of ALU instructions, and instead of jump/branch instructions it has I/O pins to wire together (called "nets", short for wiring networks, because when you connect together 3 or more pins that need to share a signal, or power, or ground, or clock, those wires have to be able to branch.) The toolchain backend reads the netlist so it can place the cells and route the nets.

The main limiting factors on how fast you can clock a circuit are 1) how long are the wires the signal has to travel down, 2) how much electricity does the component at the end need to fill it up? Physically smaller circuitry has shorter wires and less capacitance, so clocks faster. But layout also matters: if a wire "trace" is too long, the signal trying to go down it won't have done its job by the time the clock advances, so your circuit doesn't "meet timing". You can annotate signals with minimum required timing (if this 100baseT transciever is passing 4 bits at 25mhz, the circuitry at the far end has 40 microseconds handle it), and place and route iteratively try to get the best timing and tell you if anything failed to meet its minimums. (You can "floorplan" by manually grouping chunks of circuitry together with more annotation to basically give the placer hints, but that's a more advanced optimization technique usually only done when automatic place and route just can't figure it out.)

The big differences between different targets is what components are available: the VHDL or Verilog parser produces an Abstract Syntax Tree and runs the usual optimizations on it, then generates a component list and net list from that. Each Lattice FPGA family has one set of components, each Xilinx family has a very different list, and each hardware fab has its own list of available components for making photolithography masks from. A given process's list of available components is called its "standard cells", and the last "generic" step in a hardware build is the AST, because you need to #include the target process's standard cell file to generate a netlist.

The reason we still have to use the Xilinx Web ISE package to produce bitstreams for Spartan 6 (the bigger and faster FPGA in the turtle board) is icestorm doesn't know how to place and route for xilinx, and packages like Symbiflow that try to target xilinx aren't as mature because Xilinx is proprietary and litigous and has ways of encrypting its bitstreams. We have an ok list of standard cells for xilinx, so we can use GHDL to create a netlist (and thus use a more modern version of VHDL than Web ISE supports), but then we have to place and route that netlist using Web ISE to make a bitstream. Unfortunately, Web ISE is a binary only package last updated halfway through the Obama administration, which won't run without a registration key: it's basically the Flash plugin of bitstream toolchains, if Flash had required individual registration keys for each install with a working email address and needing a physical street address (ostensibly for export control reasons). They can probably remotely disable it too, I haven't checked. (Xilinx is REALLY trying to push everyone to its new Vivado toolchain, which only supports the newer more expensive FPGA families. Think Microsoft trying desperately to pull a reluctant userbase off of Windows XP without the new releases being interesting enough to move them on their own. But Vivado is no less proprietary or tightly controlled, and can't make a Turtle bitstream anyway because they dropped support for the older cheaper FPGAs.)

The standard cells of an FPGA aren't just LUTs, there are also macrocell "libraries" including clock inputs, phase locked loops, SRAM blocks, integer multipliers, and in extreme cases entire DRAM controllers. It's a bit like pulling functions out of libc that would either be a lot of work to reimplment or which you can't really implement yourself in a portable manner (because they need OS-specific system calls). Clocks need crystals, phase locked loops need capacitors, etc. The gotcha here is that each library component you pull in has a finite number of instances available on that FPGA, which live in a specific place on the chip you have to wire the rest of your circuitry up to, a constraint which makes meeting timing harder.

Compiling for ASIC (I.E. making a hardware mask) is slightly different: we take the AST produced by ghdl+yosys and run it through a different backend to produce a netlist using that fab's standard cells. In ASIC there's no limit to how many of each component it can create, the limit is how much space it has to draw cells in. The ASIC placer puts your standard cells in a big grid, initially randomly distributed (modulo any floorplanning) and then performing a kind of sort, swapping them with each other to collectively minimize the distance between pins needing to be connected to each other. The ASIC placer also tries to leave enough space for the ASIC router to add the necessary wires in the next pass, using various heuristics (fancy word for an automated way to guess). The result is more or less a giant photoshop file full of many layers of colored rectangles, with metadata so you can trace back to where those rectangles came from if you need to debug stuff.

A hardware toolchain still produces components ("cells") instead of processor instructions, but instead of libraries it has the option to bundle them together into "macrocells" (which is just cells containing other cells recursively, but with the routing locally within them already worked out). Each macrocell can be treated like a tiny little chip that has I/O pins, each with a long complicated name saying which OTHER pin on some OTHER cell it needs to connect to.

The whole ASIC has a "pad ring" around it (thing crt0.o and friends), and sucks in "standard cells" (think libc, except this defines things like "how to make a transistor in this fab process".). The pad ring is what connects to the externally facing pins the chip will have connected up to it when it gets cut out of the wafer and packaged into a ceramic case. This has special I/O pad cells (GIANT capacitor-ish thingies to try to protect the chip from static damage, or generate an enormous output current visible from space from outside the chip), and GPIO blocks that let the chip remap which pins do what (so the same ASIC can be packaged into different form factors without making different masks; sometimes there's a runtime-programmable register that lets you swap stuff in and out, and sometimes it's fuses blown during manufacturing).

Fabs treat their standard cell libraries as crown jewels: you sometimes have to sign a contract with them just to get enough to make a netlist with, and then the fab wants your netlist and will do everything else itself. If you want just OUTLINE versions of the standard cells (reserve this much opaque space you can't route through on this many adjacent layers, with I/O pins connecting at these spots) to do your own place and route, that's generally serious NDA territory. And you will NEVER get the full version that allows you to make a complete mask, in software terms you can only make a .o file that THEY link into an executable.

Standard cell libraries aren't just transistors and nor gates, they also include big macrocells like the FPGA libraries do, but fabs charge per-chip royalties for them. Using a fab's SRAM or their DDR3 controller adds extra costs, which you pay again every time you run another wafer from that mask.

Back in 2006 I was invited (by David Mandala and Tim Bird) to speak at the Consumer Electronic Linux Forum, which I've attended semi-regularly since. This year it's being held in Austin, and I'm tempted to submit a talk proposal... except I should really be doing Youtube videos directly instead of having someone else record and post them. Maybe it makes sense to do a talk there to advertise my youtube channel? Which would involve me actually starting a youtube channel; so far I have one video and the scripts for several more I haven't recorded yet because I need to learn video editing.

Ok, technically CELF isn't in Austin this year, ELC is. (That's the new name the Linux Foundation peed over it to make it smell like them.) Except ever since the Linux Foundation grabbed it they've been trying to glue it to other random events in hopes its success will rub off on whatever random nonsense emerged from their bureaucratic focus group du jour. This year the Linux Foundation is calling the collected mess the "Open Source Summit" (since O'Reilly is no longer using the name: the Linux Foundation is nothing if not opportunistic, derivative. ostentatious... Is immitation still flattery if they're not sincere?)

The latest katamari they've assembled inclues all SORTS of random padding to look bigger. Linuxcon has always made me sigh a bit at the name, but that one's at least happened before, and I think CloudOpen also previously existed, but ContainerCon isn't realy a name, OSPOCon (Open Source Projects Office) is meta-bureaucracy, SupplyChainSecurityCon was CLEARLY just made up to cash in on current headlines, and then you've got Critical Software Summit, Community Leadership Conference, Emerging OS Forum, Embedded IoT (as opposed to all the non-embedded IoT going on), Diversity Empowerment Summit (really!), Open AI & Data Forum (not even trying anymore), and Open Source On-Ramp. (Because none of the rest has an educational component?)

So yeah, they're holding CELF in Austin this year and trying to pretend it's got 14 other events bolted onto it (not tracks, EVENTS), at least half of which they just made up. Ten years ago the event's T-shirt already had more sponsor logos than your average Nascar jacket and I'm trying to figure out if I should bother with it.

Oil producers are treating their entire infrastructure as stranded assets. No matter how high oil prices go, they're not investing in more drilling because the transition to renewables could shrink demand and make the price go negative again at any time. While the USA can scale up production, it's mostly chosing not to (except for finishing about half of the backlog of drilled-but-uncompleted wells it already found oil under but never installed pumps and hooked up to the output pipes, which got shelved and auctioned off when the price went negative and the drillers went through bankruptcy.) Other countries like Russia that have nothing else going for them are already pumping at capacity and can't increase further.

For a clue why the finance guys refuse to invest in drilling, look at the oil tanker market. The "contango" issues from last year where oil tankers were used as extra storage have all worked through, and there's now an increasing surplus, and I quote: "Crude tanker owners have been bleeding cash for over a year, and the market will have to absorb more new tonnage in 2022 than in 2021." Container shipping rates are at historic highs, but coal, oil, and LNG tanker rates are down 70-90% from last year, with one tanker that costs $26,000/day to run is able to charge $800/day now.

Shipping oil from the middle east to china has even gone negative again. When prices went up they invested and increased capacity in a shrinking market, but there was no depth to the demand, less need each year than the previous year. The money guys don't want to play musical chairs and get left holding the bag. (Rare comic books sell for huge prices BECAUSE they're rare: if you discovered a hundred issues of Superman #1 in an attic and sold them all at once, the price would plummet. It's even possible that ALL of them would collectively fetch less than ONE does today.)

That said, LNG tankers are doing way better than oil, because there's demand to take extra gas from the USA to Europe, to reduce the amount they buy from Russia, and it's in the USA's strategic interest to scale that up as much as possible (replacing Russia entirely if they invade Ukraine and trigger sanctions strong enough to personally starve Vladimir Putin to death).

As for oil, the USA became a net oil exporter in November 2019, and our oil exports are booming as the rest of the world pays through the nose for the stuff. These days the USA only imports oil to refine it, but we're converting our refineries to consume fracking output which is chemically much less complex than conventional oil because it just came out of the rocks that produced it and hasn't had time to soak through other stuff and dissolve a lot of contaminants, and converting the refineries means they can't consume imported "heavy" crude anymore. The demand slump during the covid pandemic has been the perfect time to convert idled refineries.

US refinery capacity also fell almost 5% last year, as gasoline demand fell 13% (not just lockdowns but electric cars, telecommuting, the Great Resignation...) Some surplus refineries are being turned into storage, others are becoming biofuel plants (which is stupid, but happening), and more are just plain closing.

As for the increasingly lucrative container shipping side of the Force, The profitability isn't because the industry is HEALTHY, but seems to be because it's been sabotaged by the Biden administration to drive reshoring and weaning supply chains off china. As one of the above articles mentions:

Deutsche Bank analyst Amit Mehrotra... noted that the retail sales-to-inventory ratio is now three standard deviations below its 10-year historical average...

If we assume a reversion to pre-pandemic sales-to-inventory and use current demand, retailers would need $821 billion in inventory. Taking the fastest inventory has grown — which was November 2021, up $12 billion — it would take retailers 17 months to restock.

This drives up the price of goods from china (those record shipping profits are subtracted from manufacturer's and retailer's profits), and adds delay and uncertainty to supply. And it's clearly an intentional strategy because the container ship backlog at US ports isn't decreasing, it's increasing.

This seems to be an intentional (but pluasibly deniable) strategy on the Biden administration's part to respond to China's "Wolf Warrior" diplomacy: US/Mexican manufacturing is faster and cheaper once we stop subsidizing international shipping. (Which is half of what the US defense budget's been doing since the Berlin Wall fell over 30 years ago.)

I continue to object to the privatization of government functions. Mastercard's war on porn (with the enthusiastic collusuion of visa) is bad and they need to be driven out of business (via postal banking or something; the phrase "being debanked" should not ever apply to US citizens). It's providing cover for the IRS requiring third party facial recognition to access your own tax data. Late stage capitalism is contracting into a black hole.

This is the same kind of BS that youtube vs verizon was trying to pull back in 2007: everybody uploading anything must preemptively dox themselves and provide legally binding signatures stating that they own all rights to sounds playing on the radio in the background so the corporations can easily financially harvest the little guys at will. Verizon didn't get it then, but they're using pearl-clutching about sex and children to get it now. (Nevermind the self-employed models and camgirls and cartoonists they're putting out of work, or that the real problems have nothing to do with sex: M&M Mars and Nestle are facing a literal child slavery lawsuit right now, and of course their response is to try to distract from it by desexualizing their advertising. Because the antidote to accusations of slavery is performative prudery, just ask the Catholic Church. Even when neither the slavery nor child abuse were sexual in nature, being loudly anti-sex makes you immune to any other attack on moral grounds.)

Right wing loons have been using "family values" to advance fascism for generations: Reagan did it, Nixon did it, and "The Fatherland" of nazi germany was literally named after their family values campaign. The first big nazi book burning was the library of the institute for sexology. Fascists LOVE chastity belts and the Prince Albert for the same reason they love fasting, hairshirts, and mortifiation of the flesh. The less healthy your emotional life is, the more you need the cult leader to channel all your frustrations through.

But regulatory capture so you can't spend american dollars without passing "family values" judgements imposed by private for-profit corporations? That's DISTURBING. It's a pity the Boomercrats aren't even aware what the actualy issues _are_. Upstanding german citizens wouldn't stand up to stop the 1933 nazi book burning because some of those books were about gay people and trans people and the founder of the institute was jewish. First they came for...

I get emails:

On 1/22/22 9:51 AM, XXXX wrote:
> Hi sorry to bother you,
>
> I'm a it enthusiast and recently I came across your name associated with some
> mirai sample.
>
> Can we talk about it?

My old https://landley.net/aboriginal prebuilt binary toolchains got used to build rather a lot of weird code over the years, including some old rootkit stuff, and those old versions of gcc leak the build path of the toolchain/libraries into the output. (The were uClibc based and glibc is incompetent at static linking, so the cross compilers you could fish out of debian with the qemu-debootstrap stuff were way less useful for making portable binaries.)

My toolchain got stale because I stayed on the last GPLv2 release of each package that went GPLv3, and I wound up fighting the linux kernel devs to revert stupid stuff (see the sources/patches directory on https://github.com/landley/aboriginal). I switched to musl-cross-make years ago but never posted binaries of those because I don't trust GPLv3 not to turn into a lawsuit without a large corporate legal department standing between me and the loonier parts of the FSF ecosystem.

(I keep meaning to try to convince github to build the toolchains and host the binaries for me as part of the build commit hook stuff, but I'm told NFT clowns abused that infrastructure for coin mining and it caps the CPU usage or something now? It's on the todo list to poke at it...)

Rob

Of course now that I've actually looked at microsoft github's workflow docs, it looks like they delete the retained output files after 90 days. So not a place to put stable links to release binaries. Oh well.

January 21, 2022

January 20, 2022

January 19, 2022

January 18, 2022

On Fade's advice (she's been dealing with anxiety for decades and I'm new to it; I had about 15 years of depression but usually not anxiety) I took an ibuprofen and melatonin last night, and did manage to get some sleep. Still under the weather. The second time Jeff got Covid in japan (South Africa variant at the time, now renamed "megatron" because mentioning geography is racist) he had several days where he was anxious and couldn't sleep, so I'm wondering if this is Omicron manifesting through two rounds of J&J and two previous infections? (Annual flu vaccine doesn't stop you from getting the flu, it just makes it suck less when you do. Fade pointed out you can order 4 of Biden's covid tests now, for delivery "eventually". If I had one I'd probably take it.)

January 17, 2022

Did not sleep at all last night. Lay awake in bed stressed. Took advantage of the holiday to stay in bed until almost 1, but it didn't result in any more sleep.

Not a hugely productive work day with no sleep and "don't think about a purple elephant" anxiety avoidance strategies. (Plutocracy has cornered the market on spending money, not just collecting a fee every time but passing value judgements about what you are and aren't allowed to spend money on, starting with the nominally indefensible and spreading out from there the way the first nazi bookburning in germany was the library of a gender studies institute supporting trans people 100 years ago, hindsight about which is what inspired the "first they came for" poem. Since the dollar is the global reserve currency, we're exporting this "table legs need stockings covering them because otherwise it's indecent" lunacy to other countries. And we can't even focus on this with late stage capitalism pricing everyone out of their homes, a school-to-for-profit-prison pipeline bypassing the 14th amendment, student loans immune to bankruptcy with loan forgiveness being the new pie-in-the-sky dangled in front of people but never delivered by the octagenarian president, the retirement age endlessly increasing as lifespans decrease... Right, don't think about any of that until I can be ANGRY again rather than scared about living in the christian version of saudi arabia.)

January 16, 2022

Stomach bug today, which makes me wonder how much of yesterday's stress was "my intestines are clenched up" reading as stress. (I'm still pissed about all those issues, but I usually get on with things better.)

Looking at a todo list dated December 2020 and going "eh, only a month old" is a "still writing wrong year on my checks" variant. It's "twenty twenty also" already, the whole year "twenty twenty defeated us" was in between...

January 15, 2022

January 14, 2022

January 13, 2022

January 12, 2022

Ok, I need to get off of gmail NOW. The net-nanny software has made it unusable.

I was trying to send co-workers a tar.gz of an old right click -> save as snapshot of an internal wiki page about GPS timings over a homemade hardware bus, now that database has gone the way of all databases. (Seriously, we knew about the archival format skew problem thirty years ago; even when you THINK you have backups, new programs can't read old nontrivial formats. Nothing stored in a database is going to be usably retreivable ten years later unless somebody's dumped the output to a real static self-contained storage format, and even then half the time your PDF is a cover page with no contents or an index into a backup system that no longer exists.)

But gmail is doing a net-nanny thing: "oh no, I don't like the contents of that tarball, it's got css files and javascript in it, you're not ALLOWED to send that tarball". I didn't ask it to, and can't STOP it.

This is not a receive spam filter, this is a SEND spam filter. I am not allowed to SEND that email because gmail won't let me. It doesn't properly understand the contents but THINKS it MIGHT be dangerous, and insists it knows better than I do what's in my interest to be allowed to send. It had to decompress the file and untar the file (not a format widely used on windows) in order to scoff at the contents of the file. And note how there's no "here's how to override it", the linked page is "Here is an explanation of why you shouldn't have tried to do that in the first place you naughty child you.".

This is not a service I consider usable anymore. This is not a relationship I want to have with a service provider. Dreamhost provides mail servers as part of its service package, I just hate moving my email because my preferred recovery mechanism for stuff IS email. Grrr... Dowanna, but... really need to.

January 11, 2022

January 10, 2022