I've been learning way more about the FAT filesystem format than I necessarily wanted to know. I've dug into both wikipedia articles on it, Linux's fs/fat code, Microsoft's 18 year old Word document on it (on the theory any patents in that should have expired by now), and of course the newfs_msdos source that Android's currently using for this. (I also looked at the digital camera spec and EFI bios spec that _claim_ to have FAT as part of the standard, but neither actually talks about binary on-disk layout. It's just hand-waving "everybody knows" nonsense.)

What I want to do is write a mkfs.vfat and fsck.vfat, a genvfatfs archiver, and maybe some mtools-style manipulation utilities. (Not sure which: mtype, mcopy, mdel, mdir, mmkdir, mrmdir, and mmove I guess? I don't _really_ want to do mdeltree and mxcopy, is adding a -r flag to mcopy and mdel cheating? Or should I do mcp/mrm/mcat/mmv with Linux syntax? Hmmm...)

December 28, 2018

December 23, 2018

Listening to NPR's hidden brain podcast interview David Graeber about his new book Bullshit Jobs, and the word they're looking for when they describe an unnecessary exective "staff" is an "entourage". A group of people there just to make some important person look good. It's common enough there's been a word for it since feudal times.

Billionaires justifying themselves as "job creators" is a second order entourage. They're trying to make themselves look important. But they're not creators, they're gatekeepers. The need exists, the work exists, they're just standing between people willing to do it and those who need it done, collecting a toll.

Graeber mentions how BS jobs are often parasitic: they slow other people down by demanding communication overhead, which is Brooks' Law straight out of the Mythical Man-Month. He cites studies indicating 37% of workers self-report that their jobs are BS (although how much of that is "your current job is" and "you had one at one point in your career" is kinda glossed over and could use some follow-up: I've had at least 3 BS jobs over the past 20 years, I let those contracts expire and went on to do something else).

But the more interesting question is how many jobs are in service to BS? If you've got a couple floors of marketing consultans, how many floors under them are their human resources departments, billing and payroll... So now it's six or seven floors of people. Janitors, eletricians, cafeteria staff, bus drivers, none of whom would need to be there if the marketing consultants didn't show up to that building every day. Second order BS jobs are at _least_ as many people, and _that_ says if the 37% is right, then the vast majority of ALL jobs in this country are probably BS.

Which makes sense if Keynes was accurately predicting that by the year 2000 we'd have a 15 hour work week: 15 hours in a 40 hour week is 3/8 useful and 5/8 BS. And Keynes prediction was published in 1930 about the year 2000, which was almost 19 years ago. If we went from 40 to 15 in 70 years, another 20 would put us closer to 6/8 by now.

Think about how heavily subsidized and artificial the food industry is in this country, for example. The parts of it that _do_ work are basically government controlled now anyway: farm subsidies, FDA quality control, USDA meat inspection, food stamps (used to be government cheese but republicans financialized it so they do money and not stuff; it's easier to steal money, much harder to launder the contents of a grain silo into payments to a mistress). We've been paying people _not_ to farm for decades. (And then the actual harvest is done by illegal immigrants because the "farmers" are incompetent. They don't _do_ farming, they _own_ farms. They're slum lords plants, the actual work is all outsourced.

In prison, sitting in your cell with nothing to do is explicitly a punishment. Having to sit in your cubicle and do nothing is also punishment. (Imagine you have to take an international flight. You sit in the same chair for 8 hours. The in-flight entertainment is broken. Due to TSA crackdowns you get no electronic devices or books. You have to sit there, for 8 hours. Now imagine this is your job, every day.)

Richard Nixon introduced a basic income bill called the Family Assistance Program in 1969, and it passed the House. (Democrats killed it in the senate for not being big enough.) Why does overton window no longer cover it? Because of lobbying by rich people who want to eliminate the POSSIBILITY of you not working for them. If you don't work for their shell corporations, at whatever pittance and in whatever conditions they choose, you will literally die. You will starve, freeze, can't get medical or dental care. None of these things are particularly expensive in other countries, but here housing and medical care are bid up as high as they go, and only available through gatekeepers (your credit score, group health insurance).

During World War II the top income tax rate was raised to 91%, and the corporate rate to 50%. They stayed there for 20 years, and back then CEOs led very different lifestyles than they do today. Unfortunately LBJ lowered the top tax rate to 70% in the Revenue Act of 1964 (attempting to maximize government revenue), then Reagan lowered it to 28%, creating our national debt _and_ the problem of "the 1%" dominating the american economy, bullying everyone else, and buying laws. (The graph of the government's debt and the graph of money billionaires have are mirror images of each other. Republican embezzlement caused these problems.)

Lowering taxes is BAD for the economy. The entire "postwar boom" period was heavily taxed. Kansas lowered its taxes and the state's economy crashed, california raised taxes and its economy boomed, and there's a REASON. Taxes make coporations invest the money they don't get to keep: if you get to spend pretax money on R&D or worker training, it's a lot more appealing. Lowering corporate taxes switches incentive to hoarding, squeezing every last dime out of the workers and pocketing it.

The "dumbest idea ever" (that corporations have an obligation to maximize profits for their shareholders) was invented by Milton Friedman in the 1970's. It does not predate the Boomers, and can die with them.

Adding --color to grep is a somewhat intrustive redo of the -o logic, and as long as I was there I went through the grep_test list and fixed the things that were broken, some of which implied other corner cases I should test. This means I finally broke down and made "grep -e one -e two" not boil down internally to grep -e "one|two", because when you're _not_ using an extended regex there's no portable way to do that. (It logically should be \| but that didn't work in musl and I haven't even tried bionic. Oh, and Rich was made that \1 could be wrong because the glued together versions would have the same set of numbers for parentheticals and each -e should have its own numbers, but I was upset about having to traverse a long string multiple times and thrash the cache instead of one regex looking for complex but cache local things...)

So now I've got a for loop assembling a linked list of struct regex_t, and another traversing that list and filling out regmatch_t, adjusting them as we advance through a previous match and figuring out which is the "best" match, and so on. Complexity I wanted to avoid, but this would appear to be the portable way to do it...

And then I hit -w. Specifically, echo | grep -w "" matches the blank line, echo "hello" | grep -w "" doesn't match, and echo "one two" | grep -w "" (with _two_ spaces) matches again. Did I mentinon posix hasn't got -w?

Hmmm, my grep code splits patterns at \n, on the theory it's reading a line at a time so we can't include \n _in_ a pattern, which doesn't come up with normal patterns but DOES with null terminated input lines...

Travel day, Fade and I flying to Austin, with dog and waaaaay too much luggage. The "I hurt myself" load dragged out of Milwaukee has been broken down into smaller chunks, and Fade and I each are taking 2 suitcases (which southwest lets you do free).

Multiple times I've written a long email reply to something, gone "No, I shouldn't send that", decided it's better on my blog (it's off-topic in whatever thread it is), left the window open to copy it there later... And then my laptop battery dies (this one's terrible), and thunderbird of course doesn't save anything as a draft anymore. Dunno why, that broke a while ago.

Elliott and I went back and forth on the list about strace and sort -V. Still trying to get grep --color finished. Todo list gets longer even while I'm taking time off to work on it.

Broke down and tried to edit the wikipedia[citation needed] page for public domain equivalent licenses, since the 0BSD section is still hilariously wrong a month after I pointed out to them that it was wrong. But the McDonalds wifi I'm working from is banned from editing for 48 hours due to troll edits. (The universe telling me not to do that.)

I got grep passing all the tests in the test suite, and then added another test and made it pass it. I should do a thorough test suite scrub at some point and make sure there's a test for every corner case I noticed and implemented in the design, but that's "review as if it was in pending" level of polishing analysis/focus, and I don't have the spare time/effort right now. Gotta do stuff. Android isn't using toybox grep yet because it doesn't support --color, so I need to add that. (When did I last open this can of worms? It's probably been a year.)

Elliott pointed out that strace is of interest, and somebody asked for sudo on github a few days ago. It's all in the roadmap, it's a qustion of prioritization.

Lots of naps and laying prone. My back is now... less unhappy.

Carrying GIANT HEAVY BAG yesterday (you can fit an awful lot in a zipped up sleeping bag!) was kind of stupid. I should have thrown out more stuff, I just didn't have time to sort. My fault for leaving my decision to the last minute. As usual...

Cleaned out the apartment and told them I was moving out. Mailed boxes on saturday, threw out a lot of stuff I wanted to keep on Sunday, eventually got it all down to an overfull backpack, and overstuffed suitcase, and a zipped-up sleeping bag full of clothes and such, which was heavy enough I could only carry it about 100 or so feet at a time and which left bruises on my shoulder.

At Fade's. My back is unhappy with me. But I have closure, with the _option_ to come back in January but not the _requirement_ looming over me. I can relax.

Stressed to the point of chest pain and shortness of breath, and not sleeping particularly well. Lack of closure, gotta make a decision.

Ok: Bumped my bus ticket back to sunday so I can clean out the apartment. Dunno if I'm coming back in january or not, but I don't want to feel pressured into coming back just to clean out the apartment if I'm not recovered enough after a month to resume work in january. It's been a year, If I come back I can get a new apartment.

Wikipedia[citation needed] still hasn't fixed their mention of 0BSD even though I informed them that OSI corrected itself a month ago. Please stop deadnaming my license.

I got email from system76 about the new "meerkat" laptop, which... I feel bad about never buying another system from them again, but the second one is even bigger than the first one and I don't have much use for a laptop designed to stop a charging rhinocerous with a single blow to the snout.

I'm aware that companies have returns policies so a single bad experience _doesn't_ lose them customers permanently, but it was my fault for not realizing what the dimensions they listed on their website would translate to in reality, so I'm not going to ask them to eat a $2k sale because of that. And yet it also means I'm never buying anothing from then again unless maybe I can stop by their headquarters in person (Colorado?) and see the thing before buying, because I don't trust anything they sell not to require me to employ a laptop caddy to carry it around for me (presumably banging two empty halves of a coconut together).

The attraction of the "meerkat" is supposedly that it's smaller, which is damning with faint praise. Having having been burned twice already, I don't believe them. I could really use a replacement netbook because the 2013 laptop I'm using (also from them!) is waaaay too big (why it was a spare system for so long) and the battery's old, and between the two of them I don't bother taking it with me to lunch because 15 minutes work at a fast food table just isn't worth it with this thing.

(Sigh, this is the same kind of catch-22 dating in my 20's was like. "If I wasn't me, or if I don't work the way I do, I'd know exactly how to do this...")

Worked out the grep -bB thing: I can realloc()! Which most of the time isn't going to copy the data: there's usually extra space on the end because getline() is increasig the buffer in chunks as it reads. (I can't _rely_ on that, but it's the common case.) And I don't have to strlen because getline() already returned the length of the string, which was saved in a variable I can use.

Next up is making embedded NUL bytes work, which I did the hard part of already for sed (the old ghostwheel() function back when I saw "pattern" enough times I threw a "logrus" in as self-defense, and my naming convention sort of spiraled from there. Since cleaned up: it's regexec0() now).

So it was _matching_ after one or more NUL bytes in a string (you can't have a NUL byte in the pattern without rewriting the libc regex stuff, but the use case here is grepping an ELF binary or similar), but the outpt was displayed with printf(), which is going to stop at the first NUL. So it would show a line... but not show the match when it was after the first NUL byte (except maybe for -o).

The fix is to use fwrite() intead of printf(), but the problem is the length of the match was "-1" when we wanted to show the whole line, and even if strlen() wasn't expensive you can't measure the length of a line with a NUL byte in it after the fact.

Luckily, I know the length of the lines (returned from getline()), and just did the realloc() thing to stick an offset at the end of the prefixed lines, so I can make that unconditional (not just for -b), add 8 bytes instead of 4, and stick a length after the offset.

And it's in. I've got one more failing test (multiple patterns, one of them blank? What's going on here?) But hopefully soon I can chmod +x grep and thus add this test to the "make tests" regression test set once it all passes. (Where a failure means I broke something, so it can't fail normally.)

The MacOS stuff continues, but doesn't need a whole lot of input from me at the moment. I got mktemp fixed up again, and now I'm tackling grep.

The lack of --color support is what's kept Android from using toybox grep (and maybe performance concerns, but primarily --color), and the --color logic is more or less the -o logic with "different colors of output" instead of "enabled or suppressed". So I need to do a pass genericizing that a bit so the same tests can serve two masters. I did most of this work a long time ago, and it got buried (and lost) in a stacked pile of interruptions.

But what's been stopping me from working on it _recently_ is that grep doesn't pass its existing tests, and I should fix those up first. Combining "-b" with "-B" doesn't print the offset of the leading context lines, because we don't save the offset value on the linked list of previous lines. And there isn't really an obvious place to put it, not unless I want to create a new structure, and dlist_add() is allocating the old structure so switching that would unshare code.

It's a doubly linked list, so in theory I could sacrifice the ->prev pointer to store the offset, but I'm using the dlist so it can add in order (instead of reverse) without re-traversing the list each time (which is slow). And I could insert another structure in between (so dlist->data->line) but then there's _three_ mallocs which need to be freed (each in its own cache line)... ugh.

But if I collate it into a single allocation, I have to strlen() (if not strcpy()) an arbitrarily long string. This string is returned by getline(), I can't have that allocate extra data or fill out part of a structure, I don't know the length of what it's reading so it has to allocate it for me, and it returns a string.

That's why I didn't do this before. It's expensive for what it's worth. Obviously I can fix this, but I'm not happy about any of the approaches.

Oh good grief, the problem isn't where the jobs went. The problem is not a shortage of jobs. The problem is that a tiny fraction of the population is capturing all the gains. The problem is that 1% of the population ISN'T taxed enough to support the rest of the population (in indolence and idleness if that's what they want to do).

And no, it's not that we need the "brilliant work" of people like Bill Gates. His technology was utterly terrible, and people regularly wanted to KEEP USING the older stuff rather than be forced to upgrade to new (more broken) crap. What he did was CORNER THE MARKET. Android is based on Linux, which is open source, I.E. what people do when they have the free time to work on stuff they _want_ to do.

We need Basic Income, which is the same as an end to Bullshit Jobs. We need to end the idea that your day job is your identity, source of self-worth, and entire reason for existing. We got rid of the idea of nobility (my family owes fealty to Count Whatsisface who's owned by King Shinypants). We need to get rid of the capitalist nonsense that replaced it.

Technology advances until stuff gets too cheap to meter, as Star Trek predicted. (Star Trek also predicted a global war to kill off the billionaires, the same way we had various revolutions to get rid of kings.) Solar Panels and batteries do that to electricity, app-summonable self-driving car services do that to transportation, the internet does that to copying and transmitting information (from entertainment to weather reports to education)... Khan Academy is a thing somebody did for free, and then arranged to be able to do more of it because he _wanted_ to do it. Habitat for Humanity is 2 million people/year volunteering to build houses _now_ when they have day jobs to work around. Detroit rotted in 2008 because we wouldn't let anybody move into the abandoned houses until they collapsed, and "The Big Short" showed entire subdivisions abandoned and unmaintained. The Green Revolution _quadrupled_ the planet's food production and 40% of all the food we currently produce is wasted before it's ever _offered_ to anybody to eat. Capitalism is regulating scarcity that DOES NOT EXIST, so capitalism is CREATING SCARCITY. Mere subsistence is basically _free_ now, we just limit access to it.

Billionaires get rich by cornering the market. Clowns like @jack created twitter _before_ they knew it would make them rich. All the "we need to let people become rich to encourage them to create stuff" ignores the fact that they can only ever become rich AFTER they did the thing they did that made them rich. J.K. Rowling wrote Harry Potter on the UK version of welfare. The Xerox Parc guys who created modern computer designs (GUI, LAN, object oriented proramming, etc) never particularly profited from it.

And rich bastards also say that poor people need to stay starving to encourage them to work! But rich people need to go from "I have $3 million in the bank, at even 3% interest I can spend $90k/year literally forever, I never have to work again" to "I have enough money to buy Florida" or else they'll take their ball and go home and nobody will get Rearden Metal and that'll show us all (the mad scientist's "fools, I'll show them all" speech except passive-aggressive and based on the assumption you're such a special person the rest of us can't possibly live without you and your big plan is to go off and _sulk_)...

If we invent magic so you can just wave your hand and summon a meal, a shower and a change of clothes, a warm room to sleep in, or a health potion, and nobody ever has to pay for any of those things again... people complaing that it's the end of the world becuase "how will I get a job, nobody needs to hire me!" are INSANE! It's the WRONG QUESTION, it's an IMAGINARY PROBLEM. Our society suffering from a SHORTAGE OF JOBS is like the indian caste system and european nobility, it's NOT REAL. It's a social construct.

Seriously, when the Boomers die, their assumptions about how the world works need to die with them. It does not match reality. The problem is grandpa refuses to stop driving despite endless crashes. The boomers have a death-grip on power, and continue to fight the last war. They have become the problem.

An "economy of ideas" is basically people getting data from and putting it into the internet, we _know_ that's most efficiently and effectively done as open source. (If it's not published and peer reviewed, it's not science.) The "decline of the manufacturing econmy" is because we can make 100 times as much stuff with 1% as many people, not because we're making or using fewer refrigerators or t-shirts. We don't repair things anymore, we throw them away and get a new one because it's cheaper. When Paul Krugman says "the long-term trajectory of our economy... is moving steadily away from making physical stuff and toward providing services" he doesn't mean we have less stuff, he means if we can now make 100 times what we can use, then we only need 1% as many people making it to meet our needs. In a Star Trek future with replicators, you don't need anybody sitting at a spinning wheel all day every day. "Telephone Operator" used to be a real job (dial zero to speak to an operator), it's not anymore.

The problem here is capitalism. The old solution has become the new problem, and Clay Shirky gave a Ted Talk on what happens when an instutition realizes they have become the problem: the institution goes through the khubler-ross stages of grief. (It's a really good talk.)

I am amazingly burned out at work. I'm sitting in a cubicle, I'm overweight and too emotionally drained _by_ sitting in a cubicle to exercise or spend much effort on meal preparation (breakfast was two clearance protein bars, lunch was an avacado, eaten straight, dinner last night was an entire rotisserie chicken, which I regretted afterwards... oh and a pistachio croisant and large strawberry/coconut "pink drink" from starbucks).

I'm alone in a city hundreds of miles from anyone I know socially, and the project I was working on that engaged my interest enough to move here... more or less finished up last month.

The money's still great and they want me to stay another 6 months. I'm trying to hold off on making a decision about that... but I'm worrying about my health. And I'm also starting to do the "I know what I need to do next but can't bring myself to do it" mental block thing that says I'm REALLY burned out...

One more week until my month off, the question is should I just cut ties rather than leaving them dangling? What's the professional thing to do here?

The big laptop's battery is old. It didn't get used much as a server, but it's wearing down really fast when I'm out and about with it. The problem is, the battery estimator isn't reliable anymore, so it hard powered off when it said it had 6% left, and now it's just hard powered off when it said it had 40% left.

I should get a new netbook, but dunno what to look for.

I removed CFG_SORT_BIG. Back when I left busybox micromanaging the commands so you could switch off parts to build smaller versions seemed like a good idea, but it's not what toybox does. When you sit down at a toybox system, if it has "sort" you should know how that behaves. Having multiple possible sets of behaviors per command... not good. I've been trimming that sort of thing from ls and such, sort was a holdout I just noticed.

I noticed because I'm doing the FLAG() macros, and it's not as obvious a win as I was hoping. I've converted a half dozen files and the result is... mostly better? But I'm nowhere near eliminating directy use of toys.optflags because there's lots of places where I'm doing bit shifts or exact comparisons.

It's one of those things that's small enough it's hard to see the right thing. The macro is basically just:

#define FLAG(x) (toys.optflags & FLAG_##x)

I haven't worked out how to do a FLAG(x, y, z...) sanely, so the "toys.optflags&(FLAG_x|FLAG_y)" become FLAG(x) || FLAG(y) and I hope the optimizer's smart enough to turn that back?

I think it's better for the:

if (FLAG(x)) {

case, but... Sigh.

I am really, really, really burned out at work. It's a good job, in that it pays well, is not unethical, is easyish for me to do, and they value me. But it's boring and I sit in a cubicle all day living in a city I don't know anybody in, and now that the project I came here to do has finished up, it's really hard to get enthused about the new project they want me to do without novelty to get me over the hump.

The new project is yocto with systemd, on x86 and arm. The userspace is still a giant multithreaded .NET app running under mono (with enough "native" code extensions that it's about half of what's running), which implements "build control systems" (I.E. really high end thermostats that also measure air quality and such).

I'm hoping the month off will help me recover enough to do another 6 months of this? The money would be useful. Unfortunately, nothing about the position OTHER than the money is at all interesting, and my entire career has been based on interesting work over money.

Finished the test.c rewrite and got it checked in. My version passed all the tests I generated yesterday, but I've since realized it should have been "! ( ! x ) -y ! ( ! x ) -y ! ( ! x ) -y ! ( ! x )".

The ability to speed up audio I'm listening to helps me pay attention to it. It varies per speaker, but something like Rachel Maddow (who literally repeats each piece of information she gives about 6 times in a row) at normal speed my attention wanders and then I go "wait, I just missed 30 seconds"... But if it's sped up I _have_ to pay attention, and I get enough information per time period to keep my attention. That's why when I get linked to something like this where I'm very interested in the topic and want to hear the material... I try to dig it up on youtube where I can go speed->2.0 in the pulldown.

(Rachel's a great speaker, this is a style thing she's chosen to do on her show. I think it's because her audience is mostly retirees? Dunno.)

I have to implement priority handling in test.c, so I'm writing a program to emit all possible combinations of:

( x ) -y ( x ) -y ( x ) -y ( x )

Where x is true/false, -y is and/or, and each parentheses may be there or not be there. First pass said that's 15 bits worth of things that toggle, or 32768 possible test strings.

But parentheses come in pairs: there are 4 ( position and 4 ) positions, and count must match. On each side there's 1 "0 set" pattern (0), 4 "1 set" patterns (1 2 4 8), 6 "2 set" patterns (3 5 6 9 10 12), 4 "3 set" patterns (7 11 13 14... basically ~ the 1-set pattern where it's the zero moving instead of the 1), and 1 "4 set" pattern (15). And then those 4 bits for the left parentheses need to match each same-count set of right parentheses, so 1*1 + 4*4 + 6*6 + 4*4 + 1*1 = 1 + 16 + 36 + 16 + 1 = 70 total valid patterns. And 7 bits of other true/false or and/or variance, so 70*128 = 8960 tests.

EXCEPT, I didn't eliminate "left parentheses must come before right parentheses" invalid combinations? Hmmm... I can _test_ for them, but I'm not sure how to _sort_ them out. Eh, program says 27 of the 70 are illegal parenthetical sequences. And as long as I'm filtering, I might as well just iterate through all 32768 and filter out the bad ones. (I'm writing this in C.)

I was thinking of trying to do a shell script to make all these sequences in tests/test.test (there's a name), but iterating through 30k nontrivial function calls is slow in shell, and I remember Divya's chmod tests. (I cut the number down, and 90% of what that was checking was whether the _kernel_ was getting the permissions right, but still...) The more recent bc tests taking way too long to run are a similar cautionary tale.

George Bush Sr. died. I remember Dave Barry fake-quoting him about "The United... whatchamacallem... states. Barbara and I have a summer home there." He was the ex-head of the CIA (who was sad he wasn't allowed to stay on when Jimmy Carter became president), and his foreign policy consisted of having numerous foreign heads of state on speed-dial ever since then because he knew them all personally. His domestic policy was basically nonexistent, which means he did less damage than his republican colleagues. he _merely_ presided over the savings and loan crisis and a prolonged recession lasting his entire term.

(Aside: the 1929 stock market crash and decade-long great depression led to a bunch of New Deal banking regulation under FDR that prevented the problem from happening again for 50 years. Then Reagan dismantled it all due to the fish filter fallacy, and we had the S&L crisis around 1991, the dot-com bust around 2001, and the mortgage crisis of 2008. Yeah there'll be another one along soon, we're overdue.)

Bush Sr. built on the GOP's legacy of racism (Willie Hornton, war on drugs, etc), but that was continuing the policies invented by Barry Goldwater and first implemented by Richard Nixon. Bush Sr. invented the term "Voodoo Economics" when campaigning against Reagan for the 1980 presidential nomination, but then embraced them as Vice President, and his own administration increased the national debt more in 4 years than Reagan had in 8. He instructed the IRS not to ever audit millionaires (luckily they ignored him), and his "no new taxes" pledge was all about increasing inequality in the USA to basically rebuild the aristocracy at the expense of the poor. His main achievement was a war with a much smaller opponent, the highlights of which were basically war crimes. (But that's war for you.)

And yet by modern GOP standards, he was a paragon of virtue. Everyone we've had since has been a mustache-twirling cartoon villain. (In the case of Darth Cheney, standing behind an affable buffoon who explains why H.W. liked Dan Quayle: must have reminded him of his son.) And the two _before_ that were Ronald Reagan, who lowered the top tax rate from 70% to 28% simultaneously bankrupting the country and creating "The 1%" problem and then only avoided impeachment because exculpatory altzheimers turned the Iran Contra affair into "I don't remember all those crimes", and Richard M. "yes totally a crook" Nixon.

In that police lineup, yeah he's the least worst. I suppose that's a legacy. H.W. racked up more debt in 4 years than Reagan did in 8, but the soviet union collapsed on his watch so you could imagine a point to it if you tried really hard, and it wasn't yet so bad Bill Clinton couldn't balance the budget again afterwards and pay down some of the debt. (Until his son Dubyah screwed it up again.)

Got a bus to Fade's for the 14th. Accidentally got it for the 20th and had to go to greyhound and get it fixed.

Trying the Starbucks just south of work, which is open until 9. Working from Tiny Apartment is just too depressing, but I tend to work until I'm tired and then autopilot my way home. (Well, grocery store and then home.) The starbucks on the way back to the apartment from work closes at 7, this is an improvement.

Elliott has decided to make toybox work on macosx, and sent me many patches. He's gotten a largeish set of commands to build:

basename cat chmod cmp comm cut dirname dos2unix du echo egrep false fgrep file grep head help hostname id ln md5sum mkdir mktemp od paste patch pwd readlink realpath rm rmdir sed setsid sha1sum sleep sort tee timeout true uname uniq unix2dos wc whoami xargs xxd

Cool.

Yes, for once I'm happy more work is coming for me out of left field interrupting my work on test.c, because A) it's in service of the hermetic build stuff which is moving Android closer to self-hosting, B) this is work I meant to do when I had the SEI mac laptop, and never did because I couldn't get the development environment working, because I'd somehow borked my apple store login in a way that even Jeff couldn't figure out how to fix.

I break everything. I do Linux because I can strip and field service the OS down to the source code, which isn't an achievement so much as scar tissue from years of battles with everything everywhere constantly breaking. (The ability to stick a printf into the source code helps you debug, you don't necessarily need to modify the resulting system once you've found you can LD_PRELOAD=thwim.so to stub out 5 different sync() functions so vi doesn't gratuitously hang for 5 seconds every 100 characters when the disk is loaded, or that you can chattr +i /etc/resolv.conf to make it USE THE NAMESERVER I GAVE YOU DARN IT, and so on.)

I use close to stock OS images because the alternative would be constructed entirely out of spackle and anger and be a 24/7 job keeping it running. Ubuntu 14.04 behaves because I know where to hit it. I refuse to open the systemd can of worms because it smells just like Windows, a rickety pile of bad assumptions with hair gel in the shape of a White Dude standing in front of it saying "trust me". It would be a full time job finding bug workarounds. That's why I'm probably going to devuan next laptop.

Working on test.c. The contributed one had a lot of duplicated code such as all the -eq an such strings listed twice in two different functions, which sets my teeth on edge because the same thing in 2 places can get out of sync. (The two hard problems in computer science are naming things, cache invalidation, and off by one errors. The underlying generic issue with cache invalidation is "the same thing in two places can get out of sync". And pile up and waste space.)

The reason it's been in pending so long is "test" is an annoying design. It wasn't so bad before -a and -o got added, but now parsing it requires careful sequencing. (Remember: 0 is true in shell return values.)

$ test ; echo $?
1
$ test == ; echo $?
0
$ test -e == ; echo $?
1
$ test -e == -e ; echo $?
0
$ test \( == \) ; echo $?
1
$ test \( == \( ; echo $?
0

You have to look at the number of arguments to see what kind of test you're running. For two arguments, the first is special. For three arguments the second is special. Note how the parentheses above don't group arguments but are strings that == is comparing (otherwise the first parenthetical would be "true" because the one argument string is nonzero, and the second would be a syntax error because the parentheses don't match. In theory "-e == -a -d \<=" could be a valid test if you're saying a file called "==" exists and a file called "<=" is a directory, but the parser sees "-e == -a" as three arguments and then -d is neither -a nor -o: boing. Too many arguments. Of course you can use parentheses to force it to interpret it the other way, and functional parentheses vs "it's just a string argument" is yet _more_ contextual interpretation.

Then there's:

$ test -a ; echo $?
0
$ test -a -a ; echo $?
1
$ test -a -a -a ; echo $?
0
$ test -a -a -a -a ; echo $?
bash: test: argument expected
2

I.E. "one argument is a non-empty string", two is "file '-a' exists", three seems to be non-empty string and non-empty string, and four is "non empty string" and "non empty string" and... argument expected. Hmmm, how about:

$ test -e == -a -n
bash: test: too many arguments
$ test -e blah -a -n
bash: test: argument expected
$ test -e blah -a x ; echo $?
$ echo $?
1
$ test -e blah -a bork bork bork
bash: test: too many arguments
$ test -e blah -o bork bork bork
bash: test: too many arguments

So "-e == -a -n" is not "a file named == exists, and -n is a nonzero string". But "-e blah -a x" reads that way. But -a continues parsing in case of a later -o... and -o looking for a later -a...

$ test 1 -a 1 -o "" -a "" ; echo $?
0

And this is basically the logic I implemented for "find". Which was very much not easily genericizeable. (And required two parsing passes, although that was mostly because it was applying the same command line to multiple files.)

Don't get me started on "-e == -a -o -d != -o", and "\( "x" \) -a \) == \)". Technically "-e == -a -d \<=" could be a valid test if you're saying a file called "==" exists and a file called "<=" is a directory.

Oddly enough, "test !" would be 0 if interpreted _either_ as not followed by no arguments, or as nonempty string. And how about test \( \( "" \) -a "" \) -a ""

At Fade's, my sister Kris plans to pick us up to drive to a thanksgiving thing at 2.

My turtle board finally died. There were like 3 small things wrong with the initial round of prototypes and the most serious one was that the USB mini-A connector (which both powers the board and provides the serial console) was only held on by the data connections, it didn't have a clip to prevent the solder from breaking when you plugged and unpluged it. A raspberry pi case tends to hold it still, but I had to take the case off to reflash it (which means pressing one of the buttons to bring the onboard flash utility up), and... the connector broke. Sadness. I was trying to flash the j32 bitstream on it (I got one for the EVB working, and Niishi-san sent me a Turtle build) so I could maybe poke at kernel porting trying to get the new mmu working in Linux. (The sh4 mmu implementation turns out to be hugely overcomplicated and would never fit in an LX9 FPGA, so we did something simpler that should work fine but needs different kernel code to drive it. So right now it's still booting nommu kernels.)

Sigh, Thalheim broke the mcm binary toolchains. Ahem, I mean he "improved" them into decision paralysis, where users who've never used them before are confronted with a menu of choices they have to make before they have enough information to make them. Each decision required at the start reduces user uptake by some percent (they're not invested yet and can leave it on the todo list until "later"), and faced with a Starbucks menu of options when I've never had coffee before (and it won't work if I get it wrong, and I won't know why) I'd just walk away, so how can I expect anybody else not to? I need a generic "compiler" that they can use for each target. Asking "do you want to try mips, powerpc, or m68k" is already Enough Decisions.

Gimme a damn hammer. I'll work it out from there. (I remember volunteering at Habitat for Humanity when I was like 25. I brought a small hammer from home I'd written "Mine!" on in sharpie, which I used for all my nail whacking, and they said that's a finishing hammer meant for drywall nails and I should use some other hammer, and I didn't because it's a hammer and the nails went in and I was not a professional carpenter. I was still learning not to split the wood or bend the nails, and tool I was familiar with beat tool I was not familiar with. I was _not_ up for keeping multiple kinds of hammer straight, thank you. If I was still at it I'm sure I'd have graduated to hammer knowledge, but they stopped letting me be on site in flip flops so I stopped voluteering and took a tae kwon do course at UT instead. I earned a yellow belt with white stripe, the most flattering rank ever, which means I'm qualified to run away.)

Introduced to a new coworker as "the toybox guy" and I assumed she wouldn't have heard of toybox, but she had. (Then again said coworker, when her badge was flipped, was the original author of the shadow utils package, who I last saw at a LUG meeting in Austin in the 90's. So I'm not sure how representative that is.)

Bus to Fade's for thanksgiving, so short day at work.

Mimi Zohar pinged me about the cpio xattr/initramfs stuff again. I should find some cycles for that...

I taught the toybox test suite that "make test" should only run tests with the executable bit set, so I can have "pending tests". (You can still run "make test_commandname" individually, or TEST_ALL=1 to override it, it's just the test everything mode that cares.)

I've needed this forever because the test suite is full of known failures. Now I'm going through and chmoding the command.test files where all the tests _do_ pass, and cleaning up a few low hanging fruit fixes while I'm at it. An auditing whack-a-mole session, as it were.

People regularly ask how they can help with toybox development and I suck at providing good answers to that. (The cleanup.html page explaining what I _do_ was one attempt. The pending directory is me trying to get out of their way and let code go in without being up to my standards yet.) And way back when I pointed them at the test suite as some place they couldn't do much damage, and I wound up with a bunch of tests that didn't test anything obvious, tested system calls rather than toybox's implementation (as in a half-dozen variants of the same command that go through the exact same toybox code path and only vary in what they feed to a kernel syscall), tests that took 15 minutes to run, and so on. There are tests toybox doesn't pass because whitespace in output changed, or an error message got rephrased. Some tests pass for toybox but not for other implementations (in which case what are they testing exactly, TEST_HOST has to work too or it's not a good test)...

Plenty of it's my fault too. I add tests to the test suite as todo items, tests that fail now in toybox but not ubuntu, to remind me to change the code so toybox does a thing right. (Some I check in, some I don't...) My auditing pass is hitting low hanging fruit like grep.c where I can implement some small todo items in the command and _then_ the test passes, but I should finish cleaning up test.c first. Too many dangling wires I've lost track of what they connect to, gotta close tabs...

Bug report from someone using toybox under systemd in yocto. That is a combination I did not expect.

The first yocto cleanup patch touched fsck.c, getty.c, test.c, tftp.c, and tftpd.c all out of pending, which means somebody at yocto is using those commands in pending. (With -Werror flag that broke the build.)

Of those, test.c looks like the lowest hanging fruit and I've made stabs at cleaning it up before. The problem is A) test is kinda badly specified (behaving differently depending on the number of arguments _but_ in a sort of recursive way), B) this implementation has a lot of duplicate code that should collapse together if I can figure out how to order it right.

Weekend, got some decent coding in, half of which was tackling the backlog of patch submissions and bug reports.

OSI did the thing! 0BSD is now Zero Clause BSD everywhere! Woo!

(Ok, in some places it's hyphenated, and in some places the words are in a different order, but that's standards bodies for you.)

Of course the one guy who disagreed did not lose gracefully, he had to post his "BUT I STILL DON"T AGREE" to the list and then made their new page on it start with a "Note: Despite losing, I was right" disclaimer paragraph, that's... sigh, really?

But if that's the best he can do now to sow confusion and retard adoption of the license it's still way less than his first attempt at jamming up the works, which _did_ do a lot of damage. (99% of the people out there couldn't care less about OSI, it's been replaced by SPDX. So the _details_ of what OSI says don't matter, the important thing is the headline is no longer "opinions differ on shape of planet".)

I should update the toybox license page's end notes.

I got a llvm build script from the hexagon guys, which... doesn't work for me because I didn't have cmake installed, then because the cmake in Ubuntu 14.04 is too old. (Not hexagon's fault, the llvm build is insanely brittle. They need the latest C++ spec and the latest cmake. Not a tool you want to bootstrap systems with.)

I'm sympathetic with the desire to move off of make, my own make rant was spread across two posts to the old Aboriginal Linux mailing list. (It's one of many things I should probably turn into a proper essay at some point, but I don't because who would read it?)

Meanwhile toybox's scripts/make.sh is now detecting that if you set "LDFLAGS=--static" and rerun the build, it should re-link, but "make chrt" is going "nope, chrt binary is up to date". Because in my own project I implemented the make logic as a shell script (which is at least vaguely portable), but provided a conventional "make" wrapper for build UI compatability reasons (people are used to configure/make/install), and it's hard to keep that a simple thin wrapper. The make wrapper is Doing A Thing it probably shouldn't be, and it's hard to get it _not_ to remotely efficiently.

I share the pain of this message on the OSI license approval list. "the last thing I'd want... would be to get involuntarily tied up with a prior, controversial license that I didn't write, different in many crucial respects from my own..." Yeah, conflating is a thing OSI does. "As for me and my house, I'm no longer interested in OSI approval." Is anybody? SPDX replaced them.

I just want them to stop spreading an incorrect name for the license I got SPDX approval for. To me, de-approving the license and taking the bad page down is 100% equivalent to changing the name. Either way they would stop spreading bad information.

The bus back to Milwaukee was delayed 5 and 1/2 hours. If they'd told us this at the outset we could have made other plans and come back, but first it was delayed half an hour and then there were NO UPDATES (and 6 people waiting at the St. Paul pickup). The local greyhound guy had no info. The minneapolis greyhound location wouldn't answer the phone even when _he_ called. The national greyhound 1-800 number couldn't find _anything_ in their computer (the lady in Bangalore couldn't confirm this bus route existed, couldn't look me up by confirmation number or name...)

Finally the bus showed up 5 and 1/2 hours later. The story that the original bus had died and they had to wait for the next bus on this route to come in so they could send it back out again immediately instead of doing its normal maintenance. (They couldn't put us all on the next scheduled bus, which was already full.)

Not happy about the lack of communication here.

I missed my bus to visit Fade, or more accurately got on the right line number but heading to Chicago instead of Minneapolis. By the time I found out, my bus had left, and the next one's not until 6:40 (and gets in somewhere around 2am, I think).

Once again this administration's primary goal (literally the first project it pushed for after the inauguration) has hit a snag. Remember, every pipeline they don't get up and running ASAP it won't have enough total oil run through it to pay off the mortgage on the pipeline they took out to construct it. Oil go bye-bye in a single digit number of years as everything goes electric. It's _already_ cheaper to generate electricity with solar and wind, and storing it in batteries is on its own exponential cost curve running about 5 years behind solar. (Even the naysayers expect $50/kwh around 2025, despite all the headwinds the GOP can throw at it. Cell phones pump billions of dollars into battery R&D no matter what.)

All the fossil fuel discovery, extraction, transportion, refining, and consumption infrastructure that exists today is basically a stranded asset at this point. It costs more to repair than to replace, and soon it'll cost less to _run_ than to replace. The case for investing to install _more_ in it just isn't there, despite enormous political will from dinosaurs that profit from selling this stuff. All the back room deals, fire sales, and regulatory capture in the world won't hide stark business reality for too much longer.

I should reply to this post about old architectures in the kernel and there was a twitter thread about GPDR I've misplaced the link to. The thing is, patents expiring makes old hardware new again. Technology doesn't advance when patents are granted, it advances when they EXPIRE. The cell phone was invented April 3, 1973, and 20 years later it took off because all the PATENTS had expired. The great economic engine of the USA in the 1800's was ignoring european patents. China rapidly technologically catching up with the US in the 1980's and 90's was similarly powered by completely ignoring western IP claims. Patents are a giant drag on the economy, we just tend to ignore the control groups.

Patents were invented to prevent artisans from taking secrets to their graves, but that's a lot less of a problem these days. We know exactly what the 11 herbs and spices are, the Coca-cola recipe is public (it was appendix A of the book "For God, Country, and Coca-Cola" produced for the hundredth anniversiry in 1987, along with the three _different_ methods by which he got it)...) We can reverse engineer all sorts of stuff with modern laboratory analysis. The limiting factor is intellectual property law.

Jeff has 6 patents on SEI's core technology, and defending them internationally (not from lawsuits, just filing and re-filing all the paperwork to keep them active internationally, without which any Fortune 500 can set up shop in whatever country isn't covered) costs as much as a full-time engineer. I'd much rather we could do more work instead, but large predatory corporations want to steal everything they can, and when they prove to themselves they can't do what we do (yet again), they're too proud to come crawling back to us to use our expertise to do the work, so they abandon the project instead. (Thanks 3M and Siemens. Thanks a lot.)

Engineer at work talking about how they want to move away from OpenSSL to "WolfSSL" specifically because they want something proprietary that they can pay for "timely fixes".

I didn't even try to explain that "closed source" plus "crypto" should be treated about like "I put the lid back on that jar of sphagetti sauce but didn't stick it in the fridge, it should be ok, it's only been out at room temperature for 3 days, go on have some."

I did mention how post-heartbleed the open source community moved away from OpenSSL (which had disgraced itself after Theo successfully chased away all external reviewers) to other solutions, Google maintained BoringSSL, the FSF loonies did LibreSSL-free-gpl-free-copyleft-all-hail-stallman, the embedded world has BearSSL, and that yes this opened a window of opportunity for a proprietary company to market themselves, but as the industry re-consolidates around a preferred solution I expect WolfSSL's market to dry up and then you'll be paying to keep them alive until you move back to an open source solution. But hey, this won't be a problem for maybe 5 years which is long past management's decision horizon.

Looking at the sh4 syscall generation rewrite from Firoz Kahn, but gmail strikes again where multiple copies of the same message with different headers are dupe-killed and you CAN'T STOP IT, and since he cc'd both linux-sh and linux-kernel, the 0/3 message describing the series wound up in my linux-sh folder and the actual 3 patches wound up in linux-kernel.

The kernel's "patch inline" thing combines badly with both mime encoding of email (gratuitous = escapes) and html archives (even if they have have the raw data in <pre> blocks they replace > and such with &gt; and when they don't have <pre> wrappers there's a <br> on the end of each line...) And you can't just cut and paste something that's tab intended and expect the whitespace to survive. (Sometimes it does, sometimes it doesn't. But this is one more reason toybox uses spaces instead of tabs for its indents: I can cut and paste patches without worrying so much.)

Whitespace is important in patch files: each context line starts with a space (meaning blank context lines are one space followed by a newline), and if your line is tab indented it's space then tabs. Context lines have to match exactly or the patch hunk fails. (There's an ignore whitespace flag but then the indentation you're _applying_ is likely messed up and you wind up with mixed tabs and spaces in a file, which becomes obvious when you change tab stop. Again, two spaces means I get a lot less mixed tabs and spaces in submissions. They _happen_, but you've generally got to be four indentations deep before it even comes up and even then it's seldom an even number of them so it's a lot less likely to trigger.)

Fade's flight to Austin was cancelled. Never fly Frontier for something important.

(She was trying to fly home to vote since voter suppression ate her absentee ballot. If you don't think the GOP doesn't voter suppress white people too ask why a college ID doesn't count as voting ID but a concealed carry permit does. There are entire books about how racism is a tool the rich have used to distract gullible poor whites from their slum lords bleeding them dry with payday loans and such for _centuries_. Even when the billionaires internalize the racism and misogyny they're _primarily_ tools of oppression by rich bastards dividing and conquering their slaves, their renters, and their harem. France had the right idea, guillotine the lot of them.)

Spent all day too stressed to get much done.

Didn't canvas last night (closest campaign HQ that might have an address list and/or door hangers was a 5 mile bike ride away and daylight savings time means it was dark by the time I left work), but I did get up early to stick a "Vote today" sign on the front door of my apartment building. When I got home somebody had stuck much _better_ signs (with the poll location and everything) on both the front door and the elevator. Yay us!

Mixed results, but... net positive? We didn't get any of the longshot rock star candidates (in texas at least, yay AOC), but the solid trudging work mostly went for us. The senate was _designed_ to let cows vote, so we weren't going to win that one this time around.

Finally got the x32 target working in mkroot. That's probably enough to cut a release, I should do that. (Dust off the mailing list, or do it on the toybox list? Hmmm...)

Now I'm trying to add native compiler support, which means packing up the native compiler directories, which means I taught the build scripts to call mksquashfs on the native compiler directories. And while I'm there, unless NO_CLEANUP is set have it delete the native compiler directory afterwards, and blank "build" before each build (so the musl-cross-make work directories don't accumulate). Some of my builds are in a VM with limited space, its disk keeps filling up...

Poking at the the 4.20-rc1 kernel (which I once again didn't get any commits into the merge window for), and the CONFIG_UNWINDER_ORC bug is back. Switch to the frame pointer unwinder that doesn't need libelf installed, and the Makefile still barfs demanding libelf be installed before it'll continue. (I.E. the kernel developers are not regression testing on a system that DOESN'T have every possible build dependency. I'm weird for caring about the forest of unknown crap sucked into the build to produce who knows what.)

I worked around it by deselecting CONFIG_STACK_VALIDATION entirely, which meant deselecting RETPOLINE first because that was pinning it on, and then it went:

CHK include/generated/compile.h
make[1]: *** No rule to make target `init/main.o', needed by `init/built-in.a'. Stop.

So yeah, that's an -rc1 all right.

Yup, I didn't send in a talk proposal to scale either. Thought about doing the 3 waves thing but... I gave that talk at Flourish, was well-prepared and proud of the result, and the recording was never posted. I did the work, and _they_ didn't do the work (again!), and I should really just do both sides myself which means podcasts. But without an externally imposed deadline (and an audience that shows up to hear me and will be disappointed if I don't perform), I never seem to bother...

I could do something on hermetic android builds, on mkroot, lessons from the j-core experience, on 0BSD, an update on toybox, a redo of the "simplest possible linux system" thing (now with less jetlag!)... but who really wants to hear from yet another white male in 2018?

Also didn't go out for Halloween, because it's raining. Well, I went to Wendy's and worked on the toybox release notes.

Toybox 0.7.8 is out.

The Scale CFP (Southern California Linux Expo Call For Papers) expires tomorrow, I should send in a talk proposal. Working on the toybox release instead.

At my table at Wendy's. The boil water restriction got lifted yesterday so I've got soda refills again, and... my laptop's mouse froze a minute or so after I opened it. (It was working, and then _stopped_. Suspend and resume didn't fix it.)

Navigating from window to window with mouse commands, closing everything so I can reboot instead of getting work done. (I use open windows as notes-to-self about what I'm in the middle of, so when it's time to reboot the box I need to write them all down so I don't lose track of my place. Yes, "collate todo lists" is a perpetual todo item of mine...)

And after a reboot, the mouse is back and working fine. Linux on the desktop! There's a very long list of reasons it didn't take off. (Sigh, doing that used up the battery. It lasts about an hour and a half. The netbook's battery lasted at least 3 hours.)

Flew to Austin on Frontier Airlines. They would appear to be to Southwest what Southwest is to Delta. The plane was almost full yet I had a row to myself (and the guy across the aisle had the row to himself, although somebody moved into one of those seats after takeoff.) I dunno if this is a "give first timers good impressions" thing in their computer, or an artifact of being the last row in "zone 4" and the computer filling it in sequence? Either way, more enjoyable flight than I expected. (And it was like $30 plus airport fee, really cheap. Ok, there was no free in-flight beverage service (tempted to pay $3 for a soda, but didn't) but I knew what I was signing up for. Can't complain.)

Went straight to fiesta, voted (Go Beto!), came home, and collapsed. (It was only a 3 hour flight and I didn't get up that early, but travel remains exhausting nonetheless.) Then walked with Fuzzy to pick up George's medication (she has an enlarged thyroid and is in a hilarous cone-of-shame to stop her licking a sore, but doing reasonably well for a 15 year old cat), and they built a Rudy's just south of the vet so we had dinner there. Wound up watching The Big Short and talking until 5 am.

It's good to be home.

Fade gave me a half pill of her modafinil prescription to see if it helps with my seasonal affective disorder. (It's like 50 mg, half of the smallest pill they make for it.) Dunno if it's the placebo effect, but I concentrated _really_well_ and got a lot of programming done. :)

I made it about halfway through the toybox release todo list in a couple hours, I'd probably have made it through the other half too but Fade got hockey tickets. Women's ice hockey: university of minnesota wave caps vs... somebody else. It was fun. Fade's getting season tickets. (If she wants to date any of the hockey players she has my enthusiastic encouragement, but she's been way too busy to have a girlfriend. Grad school's like that...)

Spent today shoveling through stuff at work, trying to get ready to get on my bus to Fade's. (Spoiler: I made it.)

The 9.0.7 OS version on the N40 (I.E. the sh4 box's port from WinCE to Linux, which they're pretending is a dot-upgrade from "9.0.6" to "9.0.7" instead of a complete new category of operating system running the old WinCE code under Mono) is just about ready to ship (alpha test versions are in the field), but there's a couple teething issues left which boil down to "when a box boots up, sometimes you don't know what IP address it got, and thus how to connect to it with the GUI tools, unless you connect to the serial console". Which is not something we ever want people to have to do in the field.

The first reason is our recent domain migration broke the DHCP "update the DNS server with this box's name" functionality because it wants a fully qualified domain name now, but DHCP is what _returns_ the fully qualified domain name so now I have to call it twice and/or remember what it said last time, again defeating the purpose of DHCP and adding a worry about "what if it changes and my cached data is obsolete, how do I know?"... Anyway, if we're seeing this customers might too so I fixed it.

The second reason is I added zeroconf support to the box as a fallback if DHCP didn't work, and there are hiccups. This is not the same as assigning a static IP at install time, that you presumably know. But if you tell it "get a DHCP address" and it can't, it gives itself a zcip address so it has _something_. (The dhcp server really should do this for us, but doesn't. I have a toybox todo item to add it to mine, but dhcp is still in pending.)

The bug is, when it's got a zeroconf address the snmp trap announcing the board doesn't fire because the binary-only module they bought but never got the source to (written in .net and now running under mono) isn't calling the "enable broadcast packets" ioctl() on the port it opens (I guess WinCE didn't need to?), and for some reason it works in the DHCP address range but doesn't in the zeroconf address range. (I.E. it's not that I don't understand why this fails, it's that I don't know how it _ever_ works in the cases it _does_. It's not supposed to.)

But address categorization in the kernel is a horrible brittle mess of historical reasons that should probably be fixed. (If it was we'd reclaim tens if not hundreds of millions of IPv4 addresses, as described yesterday. Just reclaiming "class E" is more addresses than the entire populations of japan, australia, new zealand, taiwan, and korea put together)

Anyway, I can't fix their binary only .net module, and I fiddled with the idea of a kernel patch to disable the check or an LD_PRELOAD library to intercept the bind() and try to recognize this specific one and add the ioctl() call for it, but... ew.

So now I'm trying to write my own SNMP trap generator. It's a single UDP packet that's basically a transport for a string (containing 4 keyword=value pairs, comma separated I think), with about 40 bytes of header. Except I grabbed some broadcast packets to port 162 and displayed them through hexdump -C (the engineering department's full of these boxes, I get a packet every 5 minutes or so) and it doesn't seem to match any sort of standard I can tell? Hmmm. It's _tiny_, but messy nonsense. And all the code I can find to deal with this stuff is bureaucratic nightmare corporate garbage intentionally obfuscating itself. I'm guessing there's money in this protocol, even thought he protocol is trivial, so they try to make it look like a lot more work than it really is. Sigh.

Did not manage to get it sorted before it was time to head to bus. (Gotta catch a plane back to Austin to vote, and I might as well fly from Minnesota as from here...) Handed the SNMP thing off to Rich Pennington.

Multicast is useless can _also_ be sung to "every sperm is sacred".

I'm on a mailing list with some people who are trying to reclaim a couple hundred million unused IPv4 addresses and add them to the usable pool. There's some small debris like the fact that 0.*.*.* and 127.*.*.* are each 16.7 million addresses when they _need_ one. (Or in the case of 127.x two because ubuntu stupidly put its loopback nameserver on an address _other_ than 127.0.0.1, for no obvious reason. Then again, they can fix that themselves...)

But that's a rounding error compared to the fact that the last 1/16th of the entire IPv4 address space (268 million addresses!) is "Class E Experimental" reserved for future use, and since we all say we're living in the future we might as well use them now.

Another 1/16th of the 4 billion total addresses is reserved for multicast, which was never remotely as popular as it was expected to be, and should be reclaimed.

The reason multicast failed is data compression: napster is _exactly_ the sort of thing they expected Multicast to be needed for and instead it used breakthroughs in data compression that could squeeze a whole CD into 1/11th the space, and then MP4 did the same for video, and the _problem_ with those is if you miss a single packet the whole thing's useless. These superior data compression formats do _not_ degrade gracefully, you lose a single packet you're hosed until at least the next keyframe.

And this means youtube, netflix, hulu, prime, spotify, snapchat, whatever google hangouts is calling itself this week... NONE of them are using multicast. Realaudio died for a reason. Multicast is tying up huge amounts of IPv4 address space for approximately the penetration of "gopher".

I'm 99% certain that with an authority handing out addresses to all remaining multicast users worldwide (if there are any outside of hotel television systems which are LAN-local anyway) would comfortably fit in a /16 address range, although two of them (one /16 for public, one /16 for private) might be politically easier. But doing that still gives us 268 million addresses addresses back, that's 128k addresses out of 268.4 million. (The 1 in that 128k lines up with the 4, even a _generous_ reserve for actual remaining usage would be a complete rounding error to the allocated address range.)

People keep talking about how IPv4 "knew it would be osbolete someday" and NO IT DIDN'T. In 1980 that was more addresses than there were people on the planet and you didn't give an address to a PERSON, you gave it to a machine which meant a minicomputer shared dozens of users. You gave an address to a _household_, possibly even to an entire apartment building, and that's before Pauline Middlink invented IP masquerading so your router could provide an entire coffee shop full of people connections from a single IPv4 address. IPv4 _is_ enough for everybody, that's why the switch to IPv6 has been going on for 30 years and shows no signs of accelerating. IPv6 was overengineered and overcomplicated and nobody _wants_ to use it.

I researched this area in the first place because some 20 year old piece of code at work is using multicast for time synchronization. It turns out it's in the NTP and SNTP RFCs from back in the 90's, and they never removed it when they reprinted them. This box is from the 90's, so of course they did that and they don't want to _lose_ this feature even though it makes no SENSE in 2018.

But even in that use, it doesn't route outside your LAN so a /24 multicast address range (256 addresses) would be _plenty_. It would be massive overkill. It goes off less than once a minute, it should almost certainly just use broadcast packets. But back in the late 1990's they decided to use multicast, so...

Doing the guilt loop about patreon again, where it's been long enough since I've updated them that I want to update but think it should be about something _big_, which means I go do work and then don't finish it so can't announce _progress_ yet...

I got a large mkroot pull request with over a dozen individual commits, which I've looked at the titles of but haven't downloaded and looked at yet. I still want to merge mkroot into toybox, which is a simplified stripped-down version. This takes mkroot in the opposite direction. I kind of want to encourage the guy to fork it, because it sounds like he has plans I don't? But then there's the "simple reference implementation" issue, and I recently posted on the toybox list about copying libraries out of cross compiler toolchains (to create a dynamic build from an arbitrary toolchain) being hard to get right.

Working on the toybox release, and my tree is DEEPLY into the weeds, as waiters say (which I guess they got from golfers maybe?). A bunch of files have three different conflicting changes applied to various parts of them, and "git commit --patch" is very nice but then I need to pull it over to a clean tree and see if the result works in isolation, and if not do "git reset HEAD^1" and try again. And a lot of them were things like "the top setup in ps.c is switched to use start_redraw(), but that function's only in watch.c right now so I need to move it to lib/lib.c before that command builds again, except it should go in lib/interestingtimes.c and I _really_ need to rename that to something shorter (it's my not-curses which is apparently very impolite in china, according to Terry Pratchett who mentioned in an interview that his Discworld book of that name had issues selling there because of the name)...

Wanna make all the changes, gotta sequence them. Finding a loose thread to pull on an unravel the knot becomes an issue. As always, it's a symptom of too many interrupted work sessions I never got back to finish before I'd forgotten where I let off and I have to discover it again when I trip over it later...

Right now I'm tring to finish up adding a "%" type to lib/args.c that fills out a long with milliseconds using xparsetime(). This touches scripts/mkflags.c, lib/args, I changed xparsetime() while I wsa there, and now I'm changing every caller. And then when I test it the build breaks because it's trying to "#define FLAG_%" which says mkflags.c needs more work...

So much thread. Today I did NOT send this to the OSI list:

> So I'll try to avoid adding too much more here. :)

P.S. If you want a text version of the "why public domain equivalent licensing" spiel, it's here. And here's data about the growth of "no license specified". (There's more recent numbers but that's the link I could dig up easily. The trend line continued slowly downwards for at least the next couple years.)

That's why I started pushing my license, before you ever heard of it.

If you want what I was trying to do with zero clause bsd specifically, I explained that here.

Fuzzy went off to her Epee bout. She beat 3 opponents and came in 74th out of 127, which her coach says is great for your first national bout. (Her coach was here because he's competing in his own event.)

I hung out at Starbucks and put together release notes for the toybox release, which turned into a big todo list of half-finished stuff I forgot I hadn't tied off, and I wound up delaying the release again.

In the evening she made Scones with the Goat Keffir she found at the grocery store. (It's liquid yougurt made from goat milk and we used it as a buttermilk substitute.) They're very good.

Finaly got the watch.c rewrite checked in, although there's more to do on it. (It doesn't handle ascii escapes at all right now, just drops the ascii 27 so it prints out [35m and such.)

The crunch_str() plumbing was written when I thought combining characters were _before_ the character they applied to, not after. So it needs to keep printing until it finds the next character it could _not_ display. The tricky part is it consumes an input string and returns how much input that was and how many columns were advanced, except if combining characters are _after_ the characters they apply to, "columns advanced" and "we're done" are conceptually decoupled. It can go "yes, that was 80 columns" and still need to evaluate more input before confirming it's _done_.

This also means that a terminal displaying this has to redraw characters it's already done to _add_ combining characters, which may change the color of the character (complete redraw) and so on.

I.E. utf8 is very clever; Unicode is really stupid. Need to figure out how I want to handle that... (Even conerting to a "number of characters left" metric, it needs a "1/2 a character left" state to say it's advanced the cursor all the way but may still need to output trailing combining characters from the next batch of input. I think it would need to go from 0 to -1 to indicate that with integers. Great. I need to change every caller, _and_ add test cases for the edge case of "trailing combining characters at the right edge of the screen with disjointed input".)

Interesting: "more" exits when it runs out of input, so if you pipe less than a screenfull of stuff to more it acts just like cat. My test case was, of course:

for i in 1 2 3 4 5 6 7 8 9; do echo $i; sleep 1; done | more

But less _doesn't_, even just "echo hello | less" clears the screen and waits for you to hit "q".

But when "git diff" produces less than a screen full of output, it doesn't pipe it to less. It only pipes the output to less when output it produces would scroll off the screen otherwise.

I thought less had to be smart (less, screen, and vi all use the infrastructure I'm building for watch), but instead git's being complicated.

Andy (one of my 3 bosses at work) asked if I want to extend the contract another 6 months. I'm torn. Pittsburgh's not a bad city but I don't know anyone here, my job isn't exactly world-changing (it's not _useless_, just not my idea of particularly important; it's basically very fancy expensive thermostats), the average age of my coworkers is probably 60, and I'm in a cubicle in an open floor plan. That said, the money's quite good and I should probably work on that "retirement" thing a bit while I have the chance.

My toybox work is never going to pay the bills (Google's given me their "Open Source Award" with a $250 gift card twice, which was nice of them), and although Jeff says he's going to have money in the new year to hire me back give how often he's said that before I'd give it a 20% chance of actually happening.

Andy pointed out that I could take a full month off (second half of December and first half of January) before coming back, which moved the idea from "I don't think I'm up for it" into "tempting" territory. Huh...

So NDK R18 is out, and I should probably test with it... 2 hours to download with my phone. Lovely.

Meanwhile, fixing up the x32 symlink issue in mkroot (tail end of the mcm-buildall rename support), and fixing a build break in the timeout command for the x32 target due to the posix committee doing an insane "hallucinate a new bespoke type" thing...

Ok, building toybox with the NDK seems fixed now except for needing llvm-cc symlinked, and the liblog thing (only liblog.so, no liblog.a, so --static breaks the build). And the second of those I can fix up in portability.h (Elliott says the version in the NDK is just a stub anyway).

(I'm told building a kernel with the NDK is unlikely to happen any time soon. I need a lot more space cleared before opening that can of worms. Might get mkroot's userspace to build, though...)

Cycling back to toybox release prep, tried to build toybox with the x32 compiler and "timeout.c" broke because the microseconds field of struct timespec is not a long. They introduced a gratuitous suseconds_t type for NO OBVIOUS REASON. (It's microseconds. Goes up to a million. Fits in a 32 bit int, but here it's "long long".)

If this was time_t I'd just change the type, but it's NOT time_t. Why is it not time_t? If you're going to bitch about alignment constraints (can't have the struct be 12 bytes, it must be a power of 2!) then why have a second gratuitous type just for the subseconds field? Struct timespec (measuring nanoseconds, going up to 1 billion) uses long and struct timeval (measuring microseconds, going up to 1 million) makes up its own type. What is WRONG with the posix committee? (Other than "Jorg Schilling continues to exist", I mean.) Look, 1 million will NOT fit in a short, fits in an int. 1 billion will ALSO fit in an int. Making these different sizes is CRAZY...

I'm sorry, I'm not humoring this.

Humans can just about perceive milliseconds (60mhz screen refresh is about every 17ms, MIT study found an image needed to be flashed for 13ms for a human to process it), anything below that's noise to us. Computers are clocked in nanoseconds (1ghz = 1 per nanosecond but they generally take multiple clock cycles to do anything with a bunch of jitter). So time functions based in each of those units has reason to exist. Time functions in microseconds, such as adjtime(), suit neither use case and just clutter up the place. (I know they wanted a wider range in 32 bit numbers than nanoseconds would give them but it's still just + or - 35 minutes, go ahead and use 64 bits already.)

Made another attempt to get qemu to run a Hello World kernel, and it's being stroppy. In theory qemu's -kernel loader can handle ELF files (that's what vmlinux is), and you can write to the serial port with a two line loop, so if your entry point is a loop writing characters to the serial port...

In reality, I haven't gotten it to work yet. I've tried:

echo "void _start(void) { *(volatile unsigned char *)0x81093025 = 42; for(;;); }" > bork.c
powerpc-linux-musl-cc bork.c --static -nostartfiles -nostdlib -Wl,-Ttext=0xc0000000
qemu-system-ppc -M g3beige -m 256 -nographic -kernel a.out

I should see an asterisk (writing ascii 42 to the register), but I'm not seeing an asterisk.

I'm using ppc because that's the mkroot target I've got using a vmlinux kernel already. That boots and runs. But this doesn't. I think there's linker magic afoot. (Or else the virtual serial port needs more setup?) I want to get it working on qemu before trying with j-core on the numato board, but it's not woring and it's hard to see why. (And of course at some point qemu -s broke so it doesn't start suspended, but runs off into la-la land before you can ever attach gdb. Why?)

Haven't got my email set back up yet, but I've got the files copied over and more or less sorted. I'm using the giant System76 laptop from 2013 (the one I've been doing toolchain builds and such on, since it's 16 gigs ram, 8xSMP, and has a terabyte disk, it's just physically ginormous. The _new_ ginormous System76 box still has a distro with systemd on it, and is thus effectively a brick.)

Back poking at watch.c: I'm implementing the cursor tracking logic, and the question "what does a tab do if the xterm width isn't divisible by 8"?

First thing is xfce's "terminal" does not display an accurate width when the window is small: it says 10 and it takes 16 characters to wrap. (Bravo. I think changing the font size confused it?) So expand it to 14 (actually 19), and echo -e 'a\tb\tc\td' produces "a b c d".

Huh, with 80 column lines (after printing out 8 groups of 0-9 to confirm it _is_ 80 characters), if I repeatedly print "a\tb" the first tab advances 7 characters, the second and later advance 6, and then the last one advances 5.

I went to linux text console (ctrl-alt-f1) which measures 170 columns for some reason (framebuffer, I guess), and doing "\tx' over and over ends with two consecutive x characters. Which makes sense: 170/8 is 21, 21*8=168, but tab advances you to the start of the _next_ block of 8 characters, so the last tab is position 169. And then if you tab from there you wind up at 170, I.E. the tab is ignored. And you can tab multiple times at the right edge of the screen, it'll never advance you to the next line.

Tabs are fiddly.

Next problem: when I wrote the crunch_str plumbing I thought combining characters went _before_ the printing characters, which would be the sane thing to do so you immediately know when you've hit the end of the line. Alas, utf8 and unicode were not done by the same people: unicode combining characters come after the printing characters, which means you don't know when you've filled up a line until you've read _more_ data than will fit in the line. So you have to overshoot and parse at least one character more than you can print to know when a line's done. (It also means that when rendering, you redraw the same characters multiple times when data comes in slowly.)

(Again, this isn't _just_ to make watch work. I need to write vi and screen and such too.)

Slight hitch in the plan to ship a toybox release before today: my netbook died over the weekend while I was in Minneapolis visiting Fade. It suspended before lunch on saturday and resumed to a black screen after lunch, wouldn't re-suspend when the lid closed so it wasn't the backlight bulb burning out.

I had a professional frown at it, and got an official professional "dunno". The bios isn't coming up, which makes further debugging difficult.

I'm back in Milwaukee now where my other machines are, and a few weeks ago I asked Fade for a usb hard drive cable (she's the keeper of the amazon prime account), and I'm using it to copy the files off the netbook's drive. I got lazy, my most recent backup is 3 weeks old, so of course it happens now.

And friday. Where did the week go?

I proposed to OSI renaming the free public license 0BSD, and today it's gotten positive responses from 3 different people, including, inexplicably, Bruce Perens.

...I'll take it?

I composed, but did not send, the following response. On the theory that "selling past the close" is a bad idea. (If things are going your way, _let_them_.) That said, posting it here:

On 09/27/2018 09:07 PM, Rick Moen wrote:
> Please pardon a bit of legal pedantry, but 0BSD is not 'equivalent to
> placing code in the public domain' (as you say on
> https://landley.net/toybox/license.html and elsewhere).

I usually say "roughly equivalent" but link to the wikipedia page called "public domain equivalent license". I'm aware reality is insanely complicated, and am trying to present a simple story I can use to encourage individuals to license their code in a way corporations don't immediately reject.

> Copyright title
> continues to exist in the abstract ownable property in question from the
> date of creation to when copyright expires -- whereas the defining trait
> of actual PD is ownable title having expired or been expunged.

And in europe there are inalienable moral authorship rights...

> You might feel that this distinction doesn't matter: We would all hope this
> is the case.

I feel it confuses the story to try to explain the difference between copyright, contract, trademark, patent, trade secret law, why you want to avoid the eastern district of texas, the horror that is the "United states Court of Appeals for the Federal Circuit", let alone international jurisdictional issues...

I am entirely aware I am not a lawyer, let alone an IP laywer (at least as much domain expertise there as kernel vs GUI vs crypto), but I could give quite a long prep talk to someone about to meet a good IP lawyer.

> The assurance third-party reusers have that they aren't
> committing copyright torts is that they're doing so in good-faith
> reliance on a copyright notice with permissions grant, knowing they're
> either inside the copyright runtime but exercising that grant or past
> the copyright runtime in which case the work is _truly_ public domain.

The need for a license is because you can't just "public domain it", yes.

> And another reason this isn't equivalent to PD is that the legal
> obligation[1] to retain the copyright notice (e.g., the 'Copyright (C)
> 2006 by Rob Landley ' example on your toybox page)
> until copyright expires -- which for reasons mentioned above is for
> everyone's benefit.

The Berne convention says works are copyrighted even without explicit notice, and the Apple vs Franklin lawsuit extended that to cover binary computer code in 1983. The requirement to keep specific notices was the half-sentence I removed.

The internet is really good at finding plagiarism, and between github and archive.org proving you published first and the thing is yours isn't usually a huge deal. The larger cultural issue is "attribution" vs "ownership", but legally demanding attribution leads to visible watermarks on photos and such, when what you really want is not plagiarising:

The difference is pretty obvious to the younguns

> Yes, we get that coders would like to magick all of this hassle away --
> but wishing doesn't make it so. Your licence, CC0, MIT License,
> ISC License, and Fair License are IMO about the closest one can safely
> get that I've so far seen.

That's kinda what I'm going for, but MIT and ISC are public domain _adjacent_ licenses that have the "stuttering problem" leading to multiple duplicated copies of license text.

(I asked and they said "the copyright dates changed, and a strict reading of the license requirements said... I've used that specific file as an example often enough they changed it in master, but the problem's all over the place. The kindle paperwhite had 300 pages of this nonsense in its about->license pulldown.)

That said, half the law is precedent and common usage. Here is a very minor change to a license in widespread international use for decades already, which explicitly claims kinship with one of the most widely known license families, just about the only one other than "the GPL" with instant name recognition even among newbie programmers, and the one which the "GPL vs BSD licensing" axis is named for.

Rob

The In Nomine game was cancelled this week, so I get an unexpected programming night.

I need to print a known number of unicode columns for watch to display text without unexpectedly scrolling the screen. This is a similar problem to the top command, but that escaped all nonprintable (combining) characters to provide an unambiguous representation, and here I want them to do their thing.

This led to a thread on the list with me asking how combining characters work. (They go after the character they apply to, not before.) And that seems doable. I still need to escape nonprintable ones (invalid sequence and unknown code point).

The fiddly part is low ascii characters, which aren't standard. The xfce terminal displays some of them as boxes with a hex number in (unprintable codepoint), with width 1. (Well, it advances the cursor by 1 but displays a little wider, and they visually overlap with the next character; close enough). The Linux built-in text mode (ctrl-alt-f1) doesn't display anything below 32 (space) and doesn't advance the cursor either. Most of them are just ignored. (Everybody ignores NUL, although I'm told it's because commands have a surprising number of off by one errors and output null terminators all the time. Toybox doesn't. :)

I need to beat crunch_str() or similar into something that can display this, but... tabs advance the cursor by a variable amount based on the current X position. The \b character goes BACKWARDS. \n moves down a line, \r jumps to the left edge... the behavior isn't just "how many columns do we advance by". And then there's ANSI escape sequences...

I need to teach the crunch_str escape callback to return -1 meaning stop early, and let the caller deal with it.

Another busy week at work. They're all busy these days. Money's good though. (Slowly paying down the large home equity loan I took out to buy another year riding down SEI, after multiple consecutive "flood of the century" cleanups wiped out my savings.)

I dug up an old Numato board to send to Johann-Tobias Schaeg, who plans to port the Tron t-kernel to it. (A Japanese RTOS.) This is something I generally want to encourage, so was worth a few cycles digging my numato board out of the closet (I have 3 but only brought one to milwaukee; I still have the turtle board prototype and an EVB to run j-core stuff on), get it reflashed and booting Linux.

Took a while. I'm a bit rusty on j-core board setup it seems. :)

No, the reason stack measurement was using signed is it can grow in either direction and thus produce a negative result, which you have to take the absolute value of. We care about the _distance_ between the two, and signed twos complement math gets that right even if it wraps around.

All those "oh noes, the math wraps, let's optimize out the entire if() statement!" C++ bigots can get hit with another -fstop-doing-that compiler flag. (The correct response to most "undefined behavior" is to DEFINE it. If your optimizer can't handle it, then I don't get that optimization. I already _know_ my code could be slightly faster if I "did it another way", such as rewriting it from scratch in assembly. Don't care, I'm balancing several other factors to produce the design I want to produce. Bring me your portability arguments when your theoretical new toolchain can _also_ build the Linux kernel, I spent 3 years trying to extend tinycc to do that and I think I have a reasonable grasp of the problem space thank you. "The compiler can do two's complement signed integer math" is already a capability required to be present by posix, why do you think they're going to implement _more_ than one way to do _subtraction_?)

Cycling back around to toybox watch.c. It's hard to stay motivated with politics (the oil companies, boomers, patriarchy, white people, and possibly capitalism are all on the downswing but their death throes are trying _realy_hard_ to take the country and possibly planet with them), the EU Copyright directive is a disaster, the C++ people are doing attack du jour on C. (I stopped paying attention after Stroustrup declared C obsolete, which was about the same time he declared linked lists obsolete. Projecting much?)

Anyway: watch.c. The header it generates includes ctime() output, and the man page says ctime doesn't internationalize so length is always 24, but I'm not quite comfortable trusting multiple libc variants to be consistent about that. For one thing, the string it returns ends with a newline. (Why?) This is probably heavily specified and I _could_ confirm it's stable, but the strlen is cheap (stays within a cache line if it's aligned) so... eh.

Fluffing up the roadmap, which needs more work. I want to cut a release at the end of this month, and I've only got a couple weeks left to do that.

Rich reminded me to take another look at the stacktop logic. I think the reason I didn't make it unsigned the first time is that userspace can't put the stack in the upper half of the virtual address space (which is reserved for the kernel), and if you add two "unsigned"s to the typecast it doesn't fit in 80 characters. (And even if it did cross the midpoint doing two's complement math it'd get the answer right _anyway_, it's comparing two pointers on the same stack to see how much stack space has been used, the prorammer knows something the compiler apparently doesn't and the typecast is to get out-of-control optimizer to stop messing with stuff it doesn't understand.)

But yeah, technically Rich is correct. So I went in and added the unsigneds, and then verifying those lines were correct... if stacktop is zero we _don't_ return early (and thus call the local function), but stacktop being zero means we vforked(), in which case we _shouldn't_ recurse, so...? (Answer: xexec() in lib/ is also checking stacktop, this check here is so that on our first run we don't spuriously think we're out of stack.)

The problem is I'm not putting enough work in on toybox to keep it all loaded into my head, so when I go back and poke at stuff I wind up spending a lot of mental effort to re-create my original reasoning. It's a bit like maintaining minimal airspeed to stay airborne. Below a certain amount of development I spend all my time reverse engineering my own work, at least when trying to complete partially implemented commands or poking at the core infrastructure, anyway.

In this case, looking at the rest of the stacktop checks I _think_ the _exit() at end of xexec() should be testing if (!stacktop) instead of if (!CFG_TOYBOX_VFORK), but... was it an oversight or was I thinking of something subtle I didn't document? I don't _think_ I was, and if I didn't I should %*(#%&# add a comment now. But it's been long enough that I don't remember why I did this. It looks like a mistake. I think it was a mistake. It's only in an error path, so probably never got much testing, and the effect is subtle (mostly missing fflush(0) and possibly sigatexit() cleanup not getting called... toysh didn't get implemeted enough for shell builtins to matter yet)...

Sigh. Back at LinuxWorld Expo in 2000 or 2001, I bought the boxed set of recordings of every single panel that year (and damaged my hearing listening to them on the 36 hour drive back to Austin: the recording quality was terrible so I turned them way up to hear what people were saying over the road noise...) Anyway, one of the great talks I remember was Jon "Maddog" Hall talking about a minicomputer backup job that went to reel-to-reel tape, and consumed a dozen spools each job (manually switching them when each one filled up so an operator had to stand by and babysit the process), until he fixed it to write data out in 4k blocks intead of a byte at a time. (Because the tape machine would spin up to speed skipping/erasing tape as it did, write a start-of-record, write the byte, write the end-of-record, and spin back down again skipping more tape.) And how just getting the blocking size right made the performance SO MUCH BETTER that it all fit on one reel and an operator didn't have to babysit it, just change it once a day.

Somebody publicly hits that same general problem every couple months, and I want to refer them to that talk but of course the recordings were never posted online. But right now, I'm hitting that same problem in toybox development, and it sucks. I need to work in bigger chunks, and I haven't got them, so I'm swap thrashing. (And my limiting factor isn't exactly time, it's energy. I need to rest, commute, cook dinner, deal with email and such, and do toybox all in the same downtime.)

Posted a DMAEngine status report to the kernel list, documenting where I left off for posterity. (I'd happily try to push it upstream except I haven't got a spare 5 years to uncircle the kernel clique's wagons again.) Alas I only got the controller side working before time ran out and we moved on to other stuff...

Meanwhile, toybox! Where did I leave off... "git diff" shows I was partway through genericing microcom plumbing, which has its own variants of set_terminal(), and serial speed setting code also present in stty and gettty. Except this is doing both at once, and set_terminal() doesn't take serial speed. What I could do is add it to set_terminal and have xset_terminal() supply a 0 parameter. The question is who uses set_terminal() directly (and would need to be converted)? Let's see, lib/password.c is (which looks like a bug). And top is, which... I'm not sure about? In ubuntu 14.04, "cat | top" says tty get failed and then hangs with no further output but not exiting either. (Bravo.) Busybox top prints one thing of output (at the wrong screen size, seems to assume 80x25). And current toybox top... gets it right. So I should preserve that. Except that if you ctrl-c to exit it doesn't re-enable the cursor display.

The tcsetattr() plumbing is saving the old settings for stdin and the serial device into TT, which prevents the plumbing from being fully genericized. (The save states would have to go into "toys" instead, lib hasn't got access to any speciic command's TT.) In theory I should be able ot get it to use tty_sigreset() anyway, because A) do we really care about restoring the serial context to what it was before, B) do we really need stdin to be what it was before, or is reseting it to a sane "cooked" state sufficient?

But tty_sigreset() outputs ansi escape sequences (to reset the cursor visibility and such), and that's not right for microcom, where you may legitimately want to "microcom blah > filename" and have the exact output preserved.

I was asked to modify the LED driver to create nodes writeable by a specific group. This means the nodes A) need to have a hardwired Group ID, B) need to be mode 775 instead of 755.

This is the same LED driver I fixed platform data for last month, so I have some familiarity with the code, but this driver isn't currently setting either of these parameters, which means they're being inherited from generic code somewhere.

In leds-pca955x.c the function pca955x_probe() gets an argument struct i2c_client *client from ambient plumbing (fully formed from the forehead of zeus), and eventually calling devm_led_classdeev_register(), the first argument to which is client->dev. That winds up in of_led_classdev_register() (in led-class.c), which calls device_create_with_groups()... Hmmm, what does it mean when type is set and class is NULL...

I googled and a random stackoverflow page suggested I basically should do sysfs_get_dirent(kobj->sd, name)->mode |= 0770; (Only on 3 lines with more error checking.) I.E. don't try to get it to supply the right mode at creation time, but basically chmod() it after the fact inside the driver.

At which point I might as well just do this in userspace in an init script. This part of the kernel has ridiculous levels of indirection. It's just _silly_.

I've had to bookmark the DMA stuff and rotate back to other stuff, now that I've reached a reasonable stopping point. (As in the controller part works now, and next up I have to tackle the driver side of things which I might as well do later since it's a new can of worms.)

One bit of other stuff is that I found out part of the board's instability is CONFIG_PM, which is marked "experimental" for sh4? Huh. Lots of unrelated stuff is going through PM calls, the problematic part appears to be taking/freeing locks? I can also disable the userspace callbacks, and a bunch of unneeded config symbols... (Yay miniconfig.)

Back on the JCI n40 sh7760 sh4 hardware, and the actual physical DMA controller hooked up to DMAEngine is working with dmatest! Woo! (Which means the DMAEngine plumbing is telling the DMA controller to copy from memory to elsewhere in memory, so not hugely exciting, but the hardware is DOING something. Correctly! Driven by newly created platform data telling it how!)

I tested qemu from 2.11.0 back to 2.4.0, then 2.3.0 paniced the mainstone kernel before init so I bisected it to qemu git fc1891c74ae1 (June 15, 2015) which fixes a bug supposedly introduced in 2f0d8631b7 (May 13, 2014) except the old version doesn't work for me either.

I don't think this ever worked in qemu? They just said it did? The driver was tested on real mainstone hardware, and QEMU probably only ever _tested_ the PIO codepath. Hmmm. Be nice if there was a human I could ask...

Took a break from DMA yesterday and this morning to work on other stuff (note to self: toybox's dhcpc and zcip binaries should be the same darn thing because making the one fall back to the other is hard otherwise), but I'm back poking at it now.

AHA! That conversion commit (halfway genericizing a pxa driver to use DMAEngine) went into v4.4-rc1, so 4.4 has to work if anything ever did, so rolling back to v4.1 and still getting the same failure means the regression is in QEMU.

Ok, what do I know: ethernet card on arm platform is SMC91X with PL330_DMA, and drivers/net/ethernet/smsc/snc91x.[hc] has #ifdef CONFIG_ARCH_PXA. Meanwhile qemu has smc91c111_init() in hw/arm in the files gumstix.c and mainstone.c. And mainstone uses pxa2xx.c, and there are build instructions for that.

But "make ARCH=arm mainstone_defconfig" does not enable CONFIG_DMADEVICES which CONFIG_PXA_DMA is under. So it _can't_ work? So something is wrong here. How do I reproduce the old presumably known working environment before trying to map this forward to current code (and see where the stuff I'm _trying_ to debug broke it)...

The smc91x.c DMA code was only ever halfway translated to DMAEngine, the DMA enabling code is still under #ifdef CONFIG_ARCH_PXA today, and uses PXA constants inside those #ifdefs.

What is a PXA? Back around Y2K Intel bought an ARM fork called StrongARM, demonstrated their utter lack of understanding of anything non-x86 for several years, and then sold the corpse to Marvell. PXA was the ARM chipset they put out in the meantime.

(Aside: back when I wrote for The Motley Fool I got to phone interiew an Intel vice president. I remember I taking the call in my office at BoxxTech and asking the guy what they were doing with Arm, and him being surprised at my interest. The answer was basically it wasn't x86 so Intel had an allergic reaction to it because "we do x86" was still Intel's corporate identity. See the "you can't get from stage 2 to stage 3 until you've given up the concept of having a core product/business" part of the three waves stuff, they were _hugely_ stuck on that the way microsoft was stuck on being "the people who do windows" back when they were trying and failing to diversify into msnbc and xbox and so on.)

Anyway, the smc91x driver conversion says it was tested under "mainstone", and that's one of the arm boards qemu emulates. The QEMU mainstone board claims to emulate both the DMA and the ethernet card in question, so let's try to set it up.

This sh7760 DMA thing is hard.

So, back before my week in Austin I was trying to get DMAEngine to work on the sh7760.

The smc91x.c ethernet driver has DMA support, but it was initially only for an old ARM chipset (PL330_DMA in PXA chipsets). It was sort of halfway converted to DMAEngine, but the code's still full of #ifdefs for a specific arm board. So I'm reading the dmaengine documentation (and watching youtube presentations about how it's supposed to work). There's two halves of this plumbing: hooking up a DMA controller so you can ask it to do things in a generic manner (a DMAEngine controller driver), and then teaching a device how to use DMAEngine.

Working on the first half at the moment. In theory there's plumbing for this already. It's not currently hooked up on this board. I hope it hasn't bit-rotted too badly...

Back at work in Milwaukee. My eyes are really easily tired, still, ever since they got dilated. (And I suppose from a week of Austin pollen. And my glasses are still chipped and scratched and bent and so on. Fade ordered new ones but it's Zenni optical so they come from china in a container ship. Takes a while.)

Eye doctor appointment today, got my eyes dilated and my retinas photographed. She didn't see macualar degeneration, but the blind spot in my left eye _is_ three times the size of my right eye (something I noticed a couple years ago), because something something extreme myopia something something prescription something something optic nerve. She wants to see me again in 6 months.

Dilation made catching up on the toybox patches I didn't see due to spam filtering a bit more difficult.

The reason I didn't see Elliott's message on the mailing list is because gmail's decided all emails from him go into the spam filter. I'm calling that an own goal on Google's part.

And it's not one message I missed, it's several messages going back over a week. Lovely.

Blah, end of the month is the Ohio LinuxFest call for papers deadline. I could submit something. It's not far from Milwaukee.

I suspect I'm not going to, though.

Blah, end of the month is the Ohio LinuxFest call for papers deadline. I could submit something. It's not far from Milwaukee.

I suspect I'm not going to, though.

I've been doing a cleanup pass on the deflate code from toys/pending/compress.c with the intent of moving it to lib/deflate.c. This is hard because it uses TT and toybuf and command flags and such, which were fine in a command but not in lib.

For things like "tar" I could just ahve everything shell out to gzip/gunzip and use it as a pipe filter, but not for zip/unzip, and if I get around to doing a read-only git clone that needs zlib compression (which is deflate with a different header and crc algorithm), plus rsync -z...

So I'm converting it to lib/deflate.c, but my only current user doesn't provide a good model for what that API should look like. I should probably tackle "zip" next.

Back in Austin! Yesterday was finishing stuff at work (well, failing to) and travel, bit of a blur. (Fade's back from her 6 weeks in Rome, so we're back home with Fuzzy and the cats. And the dog, which Fuzzy's been taken care of but Fade is reacquiring custody of.)

Toybox! Been basically ignoring it for a week because tired. What's accumulated...

Blah. Went off caffeine last friday and I've done nothing but work since. I get home and go to sleep and wake up to get ready for work. Probably good for me in the long run...

On the way home yesterday I went to the doctor about weirdly persistent short bursts of pain on my left side (only lasting a second or so, but 3-4 times and day and always in the same place for the past week). He ordered a urine test, and it says "kidney stone". So I have that to look forward to. (I wouldn't think that could seem like it was at the bottom edge of one's rib cage, but I guess kidneys are weird.) And now I need to find a "primary beverage" _other_ than the cheap "60 cents a quart" gatorade knockoffs, since that's probably what's doing it. (All those mineral salts have to go somewhere...)

I've spent this week trying to add DMA support to the sm91x driver on an sh7760 board. In theory smc91x supports DMAEngine and sh7760 has a DMA controller. In _practice_ smc91x.c and smc91x.h have #ifdef ARM around all their DMA support for historical reasons, and even when I enable the sh4 on-chip dma controller DMA Engine driver (there is one, CONFIG_SH_DMAE), it only ever calls the _init function and never calls _probe, and tracing through it looks like it wants platform data describing the DMA controller. (Or a board converted to device tree.)

I don't know what such platform data should look like, but I found arch/mips/jz4740/platform.c setting up a DMA controller via platform data, so maybe I can crib from that?

For extra fun, the smc91x.c fallback code when DMA _doesn't_ happen appears to be performing the PIO in interrupt context, not even handing it off to a tasklet. When it can't set up the DMA, it does a for loop right then and there in the "else" case. No wonder the board's unhappy. (That's generic code for you: we're doign DMA, everything should support DMA, the PIO case is just a fallback nobody should be using, don't bother optimizing it...)

The big project I've been working on at $DAYJOB, since the start of the year, boils down to porting an sh7760 embedded system with 128 megs of ram and 128 megs of NOR flash from Windows CE to Linux.

The new system is buildroot (sadly with glibc) running a giant application binary (a .NET thing which other people got running on mono). I started out with general plumbing (things like writing a boot executable busybox init can call during statup to read bootloader variables and configure the network and such based on what they say), but the box needs to handle some realtime tasks (mostly to do with responding to serial data within some number of milliseconds), and a lot of what I've done has been helping squeeze latency issues out of the base OS to help the app hit its realtime performance goals.

In theory only certain threads need to be realtime, and those are running native code (from shared libraries). Getting them to respond reliably by deadline has involved identifying and removing a lot of different source of latency.

Initially they thought "run the linux-rt patch and adjust the priorities", but the realtime patches actually made things worse and raising priorities starved kernel threads.

There was of course some driver work: mostly somebody else's job although I dug through it to help them figure out what needed changing; the serial FIFO is enabled now and the tty layer remains creepy (notifying the tty char device that new data is available so poll() returns requires taking a lock, so can't be done from the serial interrupt context, it has to schedule a tasklet, and that tasklet is lower priority than "realtime" threads.)

But a lot of what's needed adjusting is userspace. I mentioned the switch from fork() to vfork. We also had to mlock() a lot of stuff (we don't have enough memory for mlockall(), the OOM killer immediately triggers, so we're parsing /proc/self/maps and replacing malloc() with global variables so we don't have to lock the heap).

Now we've reached the point where the ethernet card's lack of DMA support is a big bottleneck. I remember my 75 mhz 486dx reading 300k/second from its ATA hard drive when in PIO mode, and doing 10 megabytes/second once I got DMA enabled. Here a 200 mhz CPU with a 67mhz external bus is attempting to copy 11 megabytes/second from a 16 bit ethernet port via PIO. Needless to say the CPU is pegged at 100% without getting full speed even when we're not adding ssl or gzip to the protocol.

As an aside, I once benchmarked 10baseT at 1.1 megabytes/sec, 100baseT at 11 megabytes/second, and last I checked gigabit ethernet was maybe 80 megs/sec on a good day with a tailwind, although they've probably fixed it closer to its theoretical ~110 m/s now. (I don't own a hard drive that fast so it doesn't come up much, gigabit is "plenty".) There's been 10gigE hardware out in the wild for a decade now, and mostly nobody's cared.

With mp4 compression DVD-quality video is somewhere around 150k/second and even an HDTV stream pretty much fits in 10baseT. What people really use the faster speeds for is copying large files and mounting network filesystems.

The jump from USB 1.1's theoretical max of 1.5m/sec to USB 2's 60m/sec to USB3's 640m/s is significant because we're plugging in hard drives. Filling up a 64 gig USB stick at 60m/sec would take around 20 minutes, and filling a terabyte disk through USB 2 is well over 4 hours. Network resources we usually just stream the data we're interested in, and 100baseT is plenty fast to access a user's stuff or even a small number of people sharing. Gigabit comes in when you connect a floor of a building. 10GigE comes in when you connect entire buildings to each other, which isn't something end users do much.

Anyway, this box is trying to keep up with 100baseT and failing, so it needs DMA.

Caffeine deprivation. Long naps today.

I need to redo route, I need to do a proper toysh. These are the blockers for removing busybox from mkroot, which is the blocker for integrating mkroot into the toybox repo.

Thalheim got me a login to a fast machine running alpine linux, but attempting to build musl-cross-make failed because the patches it's applying require "fuzz", which busybox patch doesn't have because they're using an old fork of toybox patch, and that doesn't have fuzz support.

So I need to add fuzz to toybox patch. It's on the todo list...

But what have I been doing today? I want a short output mode in ifconfig that shows me:

  lo      00:00:00:00:00:00 127.0.0.1 ::127
  eth0    12:34:56:78:9A:BC 1.2.3.4

And when poking at toybox ifconfig.c to add that, I did cleanups to the existing code. (Yes, the code I use as the example of how many cleanup passes I can do on a command! It's never done, it's only shipped.)

Sigh. My eyes hae been weird forever. Back in high school, I had some sort of funky double vision where the large fuzzy circles bright lights did with my glasses off had a dome on the top, and when looking at things like an LED clock there'd be a ghost image above it. That turned out to be the glasses I was wearing or some such? In any case it went away again eventually. Most things seem to, but I'm getting older...

My right eye's been a little different from my left for years. They were identical growing up, but I noticed the colors were very slightly more green in my right eye and red in my left eye while working at WebOffice (good grief, 17 years ago now). It's barely perceptible, but it's there. Hasn't gotten any worse since I noticed it, though.

Now I've noticed that vertical lines in my right eye are bowed very slightly to the right. It's again barely noticeable. I googled and they said "macular degeneration". (Never google a medical symptom.) I'm _hoping_ it's another "your glasses are terrible and the effect persists a while after you take them off" thing...

My right eye's been the one that's had stabbing pains behind it, and feels like something's been pushing against it, for 10 years now. Which I got multiple cat scans and MRIs about (and none of the places I got them from could give me a copy of my records years later, they hadn't retained them, what wonderful US medicine)...

I poured my energy drink down the sink anyway. Going cold turkey off them. (I already know they're not good for my vision.)

Don't expect a productive week.

I've been redoing the ps help text so "ps -o help" is generated from a new field in the typos[] array (which is the "-o TYPE" data, yes my names tend to be horrible during development but I try to clean them up before checking stuff in, remind me to replace the "DREAD" help text to say "Disk Read" instead of "Pirate Roberts" before committing this).

Copying the data from the old manually formatted text to the new automatically generated columns is problematic. Some fields are a full line, or even multiple lines.

Fun formatting bits where I cheated with the manual layout because the left column had 5 spaces for field names and the right column had 7 spaces, and the left description had 33 spaces and the right has 29... and of course the "command line -o fields" section doesn't line up with the "process attribute -o fields" section at all, it just LOOKS like it does at first glance.

But the real problem is I have fields that it knows how to display, but aren't in the help text. (Many of these are in the default output of top and iotop.) And the thing is, I don't necessarily remember quite what they're displaying at this point either, it's been a while. And sometimes the kernel itself is unclear or has serious version skew. This is a perennial problem.

For example, "RSS" is displaying SLOT_rss and "RES" is displaying SLOT_rss2. Meaning I have SLOT_rss and SLOT_rss2 and I've forgotten the difference between them. I traced through the code to see that rss is read from /proc/$PID/stat (as described in table 1-4 of Documentation/filesystems/proc.txt) and rss2 is read from /proc/$PID/statm which means the kernel is returning "rss" from two different sources, and I retained them because they DON'T MATCH. (Thank you kernel.)

I vaguely recall rss2 happened because of either top or iotop, which needed the data in the statm version, not the stat one. But what was the difference? Might be time to read the kernel source again if I didn't leave myself a note in the source, on the mailing list, or in this blog... (Or twitter, but that's mostly cries of pain/frustration.)

Sigh. if I google "what is the difference between rss in stat and statm" and wind up getting pointed to something I wrote a year or two back I'll be annoyed.

Alright, last day of this. But I'm almost caught up! (Who knew it would be hard to compress over 10 years of my life into a concise narrative?)

When I returned to IBM in the second half of 1998, I sprayed the whole place down with Linux.

What happened was, I wound up back on the 3rd floor of building 903, 4 doors down from my old office... and bored out of my skull. I was there to provide support for big OS/2 customers like Ford's assembly lines, in _case_ they had a problem. There was no new development planned on OS/2, and the existing OS/2 department was largely trying to find other jobs in the company like I'd done with JavaOS a year earlier. (You may notice a theme here of me being 6 months to a year ahead of my coworkers. They were just then seriously looking to leave.)

I'd left IBM in late 1997 because I was all excited about Java and couldn't get IBM to let me do java, and during my absence IBM had taken on Java as a religion and wanted to do Java everywhere... but I was rapidly cooling on Java because Java 1.2 came out in December 1998 and it redid the GUI based on a horrible model/view/controller monstrosity called "swing" that was a _bad_idea_. Java 1.1 had the lightweight AWT that was elegant, there were some holes (like the lack of truncate() I'd pointed out) but 1.2 turned the GUI into a bloated nightmare and introduced "deprecation" meaning they didn't plan to be serious about backwards compatibility like C. For me Java peaked somewhere between 1.1 and 1.2, I did _not_ like the direction the langauge was going in, and Sun's refusal to provide a JDK for Linux had revealed that they didn't want to destroy microsoft's monopoly, they wanted to capture it intact. Nobody wanted to substitute microsoft's leash for Sun's noose.

Sun was very clearly threatened by Linux, but the Linux community had written Sun off as too dumb to live after this post from a sun engineer, and the too dumb to live part was in reply to a long technical explanation of how Linux was better than Solaris which also explained why Sun had invented threading. Threading happened because Solaris' process switching was utterly terrible. Linux's process switching was faster than Sun's _thread_ switching, and Java's refusal to provide things like poll() or select() and instead make you spawn threads just so they could block waiting for I/O was _silly_. (Linux would shortly have its own performance issue trying to replicate sun's model, but they also rapidly _fixed_ it. "It never occurred to us to optimize for such a stupid use case. We've now done so. Next.") By the time Sun screwed over Blackdown, The Linux community as a whole wrote Java off as a bad idea that might need legacy support the same way Cobol did, but nobody sane should write anything _new_ in it.

Linux famously grew 212% in 1998, that was all the Java developers switching over. Netscape had collected the "anything but microsoft" crowd under a single flag, and then poured them into Linux by elevating Linux to a Tier 1 platform and pointing to Linux as the model that convinced them to release the netscape source. You still needed an OS to run Java under, and the Java userbase switched en masse to Linux and patiently waited for proper Java support from Sun, learning native Linux development in the meantime... and then Sun's bad behavior convinced them to stop waiting and stick with native Linux development.

I was about 6 months ahead of this curve because I'd been keeping tabs on Linux since the SLS disks, and had already been looking to move _off_ of OS/2 with Linux as the obvious next step (because "it's not going away" was a unique value proposition among non-microsoft operating systems, at least before Steve Jobs returned to Apple). My main problem was that Linux had never been able to make XFree86 work with any graphics hardware I could actually obtain, and even installing the OS/2 port of XFree86 to tinker with it hadn't solved the problem for me. Linux was fine, XFree86 was terrible. And after OS/2, I wasn't going back to a text mode only OS.

So I bought some new hardware and made another attempt at putting together a Linux system. I didn't have a CD burner at Quest or at home, so I downloaded the 25 debian floppies and tried to install them, but my serial port was at a nonstandard IRQ and I didn't know I needed to use the "setserial" command to tell Linux (hard to pull up a man page when you don't know the name of the command)...

And here's where I asked that question on the list, which marked me finally getting a usable Linux system installed at home, albeit in a slightly less obvious way than the question implies. After posting, I read all the list traffic, including a message from somebody who had an easier time getting Red Hat installed, a reply to my post from someone who had the same problem but no solution, and then I hit this message replying to someone comparing Debian and Red Hat, where the Debian user said he was "really grateful for not having people like that trying Debian" and that "Debian is not the right distribution for them" and I went "Ah-ha, there's my problem! I shouldn't be using Debian! It's full of assholes like that guy. He says he's glad for everybody like me to go use Red Hat and leave his thing to die, I should take this advice and let network effects do their work!"

So I downloaded the Red Hat install disk, figured out how to get setserial to teach the kernel to use my modem (either that or it probed the sucker correctly, there was a kernel config entry for that way back when that Debian didn't enable because it was "dangerous"; it poked the card to generate an interrupt and assumed the next interrupt was the card's interrupt, in theory a spurious other interrupt could come in but in reality it worked fine). And I was a happy Red Hat user for years after that (happy until the inexcusable introduction of "kgcc", and continued to use it via inertia through Red Hat 9, then Fedora Core 1 dropped support for the processor my machine was still using because they built the kernel "optimized" for a newer chip and it wouldn't boot on my machine. At which point I switched to Knoppix, and from there to Ubuntu. I refused to touch Debian without thick gloves between me and it until quite recently, all because of one asshole at a formative moment...)

This poking at Debian and Red Hat was the context in which I returned to IBM in the second half of 1998, and sat in my office bored... so I tried to install a Linux partition on my workstation so I could poke at Linux at work as well as at home. Except that Red Hat's network install wouldn't work through the socks/proxy firewall, but the department at IBM had a CD burner and a stack of blank CDs in the supply closet and I remembered Debian's download directory had contained ISOs, so I downloaded the Debian ISO image and burned it to a CD and... found out IBM had given all the leftover unsellable Micro-channel bus PS/2 systems to employees for desktop machines, and Debian didn't support micro-channel yet. (Red Hat said it did, but Red Hat had no ISO downloads (to encourage CD sales) and couldn't install through IBM's firewall...)

So I had a useless Debian CD on my desk and mentioned it to a coworker in the usual office chit-chat, and the coworker said he had a PCI bus machine (because the micro-channel ones were old and creaky by that point, and being replaced), so I gave him the Debian CD (legacy AIX knowledge was common in the department so a PC unix was of interest), and then other co-workers wanted debian CDs so I burned more and passed them around the department...

So I got my bored co-workers excited about Linux, but I couldn't get an official IBM position on Linux. I cornered an executive in an elevator and asked him what IBM's plans for Linux were, and he said the lawyers were uncomfortable with the GPL and until they signed off on it, IBM wouldn't touch it. So I let my contract expire at the end of the 6 months, and went to work somewhere else.

The next year all those old OS/2 guys founded the "IBM Linux Technology center" (and eventually got a bigger space under the cafeteria in the building next door), and when Lou Gerstner left he gave his successor Sam Palmisano a todo list starting with "spend a billion dollars per year on Linux for the next 5 years"...

The first time I left IBM, it was because I couldn't do Java. Then they got Java as a religion. The second time I left IBM, it was because I couldn't do Linux. Then they got Linux as a religion. I spent the first half of the 2000s thinking I should have stayed at IBM and waited for them to catch up with me...

A year and change later I attended the disaster that was Kansas City Linuxfest (which was _so_ bad several of us huddled around a table in the debris and talked about how to run a convention right, and I took notes, and a couple years later that became Penguicon). In the aftermath of KC LinuxFest there were boxes and boxes of the June 2000 issue of Linux Journal which they'd meant to give away as schwag at their booth, but there were like 1/10th the attendees as expected, so they were piled on a pallet to ship back to Seattle, and I grabbed a couple boxes and drove them home and left them in the entryway of the IBM cafeteria with "Free: Take One" written on the flap. That was the "who's who in Linux" issue with interviews with the top 50(?) contributors, including lots of important people like Pauline Middelink (inventor of network address translation) who you don't hear from today, because driving brilliant pioneering women away from the boys club via endless harassment is how a lot of white dudes keep themselves employed. (They can't compete, so they harass...)

I've poked at Linux as a hobby ever since. I taught more night courses at ACC around this time ("intro to unix" passing on the Linux stuff I was learning, and "intro to operating systems" which was a survey covering mainframes and such) I really enjoyed teaching, but after a break when I looked into it again the paperwork requirements had grown beyond my interest, and teaching as adjunct faculty paid less than writing for the Motley Fool did. And I wrote about Linux several times for the Motley Fool (another "for fun" thing, although for a year or so it turned into a half-time telecommuting position that let me take time off from programming while my mother was sick). Writing or The Fool tapered off at the end of 2000 when the dot-com bust happened, that's its own story, ask me about that sometime if you're curious...

I'm not sure if "coming back to babysit Feature Install" was my first Bullshit Job, since I did in fact have the skills to do it and was ready to if the need had arisen. But if not the next one on my resume, Trilogy, definitely was. Trilogy was more money than 27 year old me had _ever earned ($50/hour! About $75 adjusted for inflation.) doing a crazy java thing which was _also_ IBM politics. Two divisions of IBM had fought over a project until management took it away and outsourced it to stop the fighting. Of course Trilogy had sold them a useless home-grown Java framework as part of the deal that I was nominally there to work on, but I don't think I ever added a link of code to it that mattered? The real job was coordinating work done by competing bits of IBM in diferent cities, which Trilogy was not in a position to do. (But we were at least considered impartial, and thus distained by both sides.) So I spent all my time on phone calls getting bits of IBM to talk to other bits of IBM, and eventually figured out that what had screwed up the project was a specific manager in Dallas who had figured out how to advance his career by screwing up projects. He'd take a project still in the planning stage and go "why isn't this being implemented?" (Because the design's not done?) But he'd make noise and implementation would start with an unfinished design. Then he'd go "Why haven't we started testing?" (Because it's still being implemented?) And enough noise later testing of the unfinished thing would start. The result was a nightmare, but "Mike gets things done!" This wasn't happening before he acted, and now it's happening! The fact it _shouldn't_ be happening and is heading for a cliff wasn't to become apparent until he was promoted away.

The project outsourced to Trilogy was some sort of billing system for IBM mainframe customers, which some IBMers confessed to me _had_ no fixed prices for anything because the sales process was entirely about figuring out what the customer would pay and charging them every penny of it. My main recreational activity on the job was fishing through the bug database for a request from IBM to do a thing and a request from IBM to undo that thing (or do the opposite), pair them up, and bring them up in the daily conference calls. The project got so bad that the IBM WorldWide Integration and Test facility in Australia signed off that it had "passed" (some random version of the constantly changing spec and some random version of the constantly changing code matched some random version of the constantly changing test profile, or at least they caimed it did) and closed out their budget and said "sorry, we're done, we can't work on this anymore". Of course there were pending design changes (of a fundamental "it should work _this_ way" type) still being entered into the defect tracker, but it was brownian motion rather than progress. The test department washed their hands of it so no further progress could be made, and that was just one political crisis du jour. I was paid to be on conference calls with France and Boblingen Germany, WWIT in Australia, and Poughkipse New York. Note the three wildly different time zones? 6am call, 8pm call, or 3am call (Austin time) depending on who had to be on the phone. I would sometimes have four of them in a row (all three for the day and another in the morning), and was encouraged by my boss to sleep under my desk. It was one of those dot-com companies with a free food room, and not cheap stuff either: roast beef and powerbars and so on.

At the end of the 6 months I turned down a 50% raise ($75/hour! In 1999 money! For a 27 year old!) to continue, because I was developing stress-related health symptoms. (And my mother was dying of cancer, but that's another story...)

Trilogy taught me to pursue fun work rather than the most lucrative thing I could currently be doing. At my next job (Boxxech) I tried out a management role (and hated it), then took a huge pay cut to join a startup I found at the local Linux User's Group meeting, because I wanted to do Linux full-time. That was WebOffice, which I've already written about, and which led to Aboriginal Linux and mkroot and busybox and toybox. There, all caught up.

Day 3, still working towards my switch to Linux, we're up to about the start of 1997.

OS/2 4's two selling points when it shipped were Java and voice recognition (which didn't have an easter egg for "Tea, Earl Grey, Hot", and I'm still sad about that). OS/2 was crammed to the gills with Java, which was uniting the "anything but microsoft" crowd. (Back when I still got along with Eric Raymond I put this part of the history in the art of unix programming, which I added so much material to he almost made me a co-author. It's a pity he ossified into a loon. What is it with old unix guys? When I first met Eric, he said back in the 1980's Richard Stallman was sane and he and Eric used to be friends, but Richard got progressively crazier as the years went on. Eric was worried he'd go crazy the same way as he got older, but instead he found a whole new axis of crazy to go down. I am sad. Anyway...)

I was intrigued by Java because my old "platform independent binary excutable code" (PIBEC is not a useful acronym) project was my proposed graduate project if I'd gone to grad school instead of trying to pay down my student loans immediately. (It's also what got me into compiler development, I first read large chunks of the GCC source code trying to figure out how to add a new "backend" generating my bytecode. I'd also need to have two kinds of function pointers, one for native code and one for bytecode, but I never got that far. I also tracked down a copy of Small-C and read that when gcc circa 1993 proved intractable, my later participation in tinycc was probably inevitable.)

But I hadn't been able to go to Hursley (where IBM's JDK team worked) due to the Austin site consolidation, and Sun didn't get back to me in time before I'd committed to move to Austin, so at the end of 1996 when Warp 4 finally shipped (a year too late to matter) I finally did transfer to IBM's JavaOS team... as a tester. (There were no developer seats open, but I wanted in and I can test. And it meant I was doing Java. Now instead of working on the 3rd floor of building 903, I was on the 6th floor. Oooh.)

I thought JavaOS meant "port the JRE to DOS using expanded memory and green threads with an SVGA AWT". That was the obvious approach to me: boot from a floppy, run well in 1 meg of ram, it wouldn't do SMP but you can do upgraded versions later, NCSA telnet was providing an IPv4 stack for DOS back in 1986. This would handle all those kiosk use cases they were talking about with ease and could be scaled up in later releases.

But IBM was porting Sun's JavaOS to PowerPC, and what Sun did was take the Solaris kernel (there is no recovering from this point in the sentence, we're already doomed) and run the JDK as the init task on PID 1. Solaris had no device drivers for non-Sun hardware, so they allowed userspace device drivers to be written in Java, meaning the OS was calling out to Java to interact with the hardware. Of course they didn't hava device drivers ready in java either, for esoteric things like IDE hard drives, so they had to dhcp/bootp (which intel later renamed PXE boot but this was 1997 so Intel hadn't taken credit for it yet) to bring the system up, and ran from a 32 megabyte ramdisk, with another 32 megs for Solaris, or a total of 64 megabytes of memory _IN_1997_. This was an INSANE amount of memory at the time. OS/2 could run in 4, run comfortably in 8, and 16 megabytes was posh, and OS/2's biggest criticism is it was a memory hog. It quickly became clear to me that this was another powerpc tangent due to IBM's hardware side dragging the software side away from anything any customer would ever want.

This is also the system where the animated screensaver paused noticeably every few seconds to run the garbage collector, which got me thinking about improvements. But at the time what this meant was on top of everything else, the desktop is not reliably interactive.

This time the crippled powerpc Boondoggle IBM was doing got cross compiled from PowerPC AIX machines, so I was learning some new stuff, but I wasn't really getting a lot of Java coding in. (It was during this period I wrote a deflate implementation in Java, in the evenings on my home machine. If work wasn't going to teach me java, I'd do it myself...) The JavaOS work wasn't as stressful as the OS/2 development had been, but it seemed like such a waste of time. And as a tester, I wasn't allowed to fix anything that was wrong with it. And they didn't _need_ testers to know that 64 megs of ram with no hardware drivers and a glitchy desktop was not a recipe for success. I worked on JavaOS for 6 months, and then I too looked outside IBM.

I answered a classified ad for a Java developer, nailed the phone interview, and got a job at Quest Multimedia writing 45 "interactive diagrams" for the CD in the back of a McGraw-Hill math textbook. I'd been at IBM long enough I didn't have to pay back my relocation money, so I took the plunge.

Working at my first start-up was fun, but still fairly exhausting. Jere Confrey (who is apparently at NC State University now) and her husband Alan Maloney were professors at UT who were good at navigating grant money bureacracy, but were sick of 2/3 of it immediately vanishing when they ran it through their university (charging them exorbitant rent for office space and $50 per folding chair and so on; the university viewed such grant money as supporting the university more than any specific professor or project). So they set up their own little company to run the grant money through, and got their office space in a strip mall off of 183 and Anderson Mill (northwest of town).

They had a textbook contract that involved writing precalculus tutorial exercises as Java applets. Jere would come up with an idea, Alan would do a macintosh "hypercard stack" with a sort of storyboard, print it out on paper (usually 5-10 pages, mostly fake screenshots), and I'd implement it. They had a graphic artist to feed me backgrounds and gifs to animate (and to make the web page each applet would embed into), and the four of us were the entire company.

I worked there for 6 months, and am still proud of a lot of the work I did. I made an auto-resizing graph (figured out where the graph lines should be in the X and Y range you told it to display, trying 1, 2, 3, and 5 until it got between 7 and 10 lines) which you could feed a string equation to and it would plot it on the graph, and if the equation was an inequality it would fill the appropriate side of the graph (with a dotted or solid line depending on whether it was > or >=), and for one of them I even colored how many layers of overlapping inequality you had. And of course moving a point along the curve... Under the covers I wrote a function that would parse a string and perform the equation it said (with parentheses and correct priorities and everything, pushing and popping operator and operand stacks), and the way it plotted curves is I'd stick an X in there as a variable and then string.replaceAll() to substitute in the number and repeat that for each point I wanted to plot, and it was fast enough on a 486dx75 to do at least 4 or 5 frames per second even on the complicated ones. (The trick is never allocate objects after startup, which is expensive because it memsets their contents to zero, and also means the garbage collector never runs.)

Another fun thing about working for professors is that when I decided I wanted to teach night courses at the local community college, and write for an online publication (the Motley Fool), they were supportive. (IBM made it clear that full-time employees were serfs and they owned ALL YOUR TIME, so anything you did outside of work belonged to them or was somehow stolen from them. Quest didn't care as long as I got the work done, and with two professors running the place "busy writing and teaching when not doing work for you" made them feel right at home, like I was a grad student.) I started by teaching an "Intro to Java" course at the ACC campus closest to work, because nobody knew much about it then and my year and change of poking at it, plus the day job doing it, made me an expert.

The desktop system Quest bought me took a while to set up. I wanted to run Linux on it, but "there's no JDK for Linux" was the #1 bug on Sun's "Java Developer's Connection". The main page of java.sun.com, where all the java documentation lived, showed you the top 5 bugs, and if you logged in you could vote for which bug they should fix next. This bug had more votes than the next 4 bugs combined, and it stayed that way for 11 months... until they changed the page to not show any bugs. So my desktop had a Linux partition on it, and I played with Linux on my home system Xfree86 on my Western Digital graphics card was still all sparkly and vertical tearing because they updated the screen while it was drawing rather than waiting for the vertical retrace interrupt. On the new SiS motherboard that Quest got me, the framebuffer driver had an endianness issue so every 4 bytes were reversed, and the software drawn mouse cursor read the framebuffer data in one endianness and wrote it in the other, so moving the mouse cursor across the screen left a smear. And of course finding the XFree86 devs to talk to them was basically impossible, and nobody could get access to their source control without being an "approved committer", so I could never get any of it fixed unless I tracked it down myself and submitted a patch to the distribution maintainer. (Maybe they could get it upstream into the package? Not my problem at that point.)

So I put an OS/2 partition on the Quest machine, and did all my Java development on that. And then they'd test it on their MacOS 7 browser (both netscape and internet explorer for MacOS 7), and test it on a Windows 95 IE machine (which had _different_ bugs than IE for MacOS, I remember IE got nested lightweight component positions wrong, instead of traversing down each canvas and adding up the offset of each parent component to figure out where to place the upper left corner of this one, it would multiply your component's offset in this canvas by the number of parents it had. Took a while to figure that out because it was usually off the bottom right of the screen. Worked fine on everyting else, but not on IE. This meant I couldn't use nested canvases but had to position every component manually in a single canvas, because microsoft. There were so many other bugs like that, Java really earned the "write once, debug everywhere" moniker.

But I wanted very much to add Linux to the pile, because JavaOS clearly wasn't it. There was an open source thing called Jos people were playing with that I hoped might turn into a thing, but I'd been following it most of a year and it had already stalled by that point. (This is where I learned that talk begets talk and code begets code: they'd started with design discussions instead of a prototype. Their discussion quickly expanded to the point where no intiial implementation could do the vast design justice, and there wasn't a cannonical implementation for everybody to attach their work to or coordinate development via... A year in they had a page of desktop graphics and that was it. I used Jos as an example in my Prototype and the Fan Club talk.)

But it wasn't clear what Linux I should install. The SLS disks I'd tried years earlier were long gone. I remembered mention of "debian" and found that but their page pointed me to the FSF which had a don't call it Linux, it's GNU GNU GNU we take credit for EVERYTHING ALL HAIL STALLMAN! screed, which was historicaly inaccurate but I didn't know better at the time and this led to a certain amount of embarassment. And in any case: no JDK. I couldn't build and test the Java applets I was writing on Linux.

A couple months later (Feb 1998) Netscape announced it was releasing its source code and elevating Linux to a Tier 1 platform, and I went "oh good, everything should get fixed now" and expected Linux to get proper third party support from places like Sun. Instead Sun recoiled and hissed at the threat. Remind me to write up the "Sun Civil War" someday.

And I still couldn't manage make a Java development workstation out of Linux at any point during my time at Quest, so I did all my Java work in OS/2, which I viewed as increasingly dead. (For example hard drives were coming out that were too big for it to format, ala the ATA-1 128/137 gigabyte limit, with no fix in sight at the time...)

This is also where I got my first laptop, I mail-ordered a used IBM "butterfly keyboard" laptop and installed OS/2 on it to do Java development. I loved that machine. Sadly when I moved out of Austin in 1999 I leaned the laptop bag against the side of my car while packing, forgot it was there, and ran over it backing out). But I replaced it with another laptop, and have used laptops instead of desktops as my primary development system ever since.

But mostly, while I was there, I wrote java GUI programs. Lots and lots. I wrote a 4x4x4 3D tic-tac toe game that played against you with varying difficulty levels (the trick to beating it on the highest difficulty level was it would try to extend its longest line or block your longest line with the same weighting, but you could fork it by making structures you could complete along more than one axis...) I had to relearn _so_ much trigonometry to do the "point on a circle you drag around with the mouse" stuff that showed you the X and Y coordinates and the angle and you could set any of them by highlighting and typing into the appropriate display field and it would move the others and the point for you... Plus you could set its rotation in radians per second and it would move it for you...

Anyway, I _loved_ that job. The money wasn't great (they'd matched my IBM salary, which was better than flipping burgers but not twice what flipping burgers paid; add in student loan debt and a car payment and there wasn't much left each month). But after about 6 months I got a sinus infection and decided I really _needed_ health insurance, and oddly enough IBM was advertising my old position doing Feature Install for OS/2, and I went "I am literally more qualified for that than anyone else on earth". And as a contractor they'd have to pay me by the hour (at a better rate than I'd been earning as an employee), and either they wouldn't work me 90 hours/week or they would PAY ME FOR ALL THOSE HOURS. Either way, I'd come out ahead.

And that's when I got serious about Linux.

Continuing from yesterday, and trying to work towards when I finally switched to Linux as my desktop system.

In 1995 I graduated and took a job at IBM, moving to Boca Raton to work on IBM's OS/2 port to the PowerPC. It was the place the PC had been invented, Lou Gerstner had brought the company back from the dead and was running it _very_ well, and I was using OS/2 as my desktop system which seemed to have an actual chance in the marketplace that I wanted to help along in any way I could.

This was a learning experience. I worked on the last 4 months of "OS/2 for the Power PC", and basically watched it die, then got moved to Austin and worked 90 hours a week for a year on something that was already too late.

OS/2 for the PowerPC made sense when they started the project, but not when I worked on it. The powerPC was one of the late 80's explosion of RISC systems, because CISC was clearly on its last legs. I blathered about Risc vs Cisc long ago, but the point is everybody knew a RISC system would unseat CISC and thus x86, they just didn't know which one. And then x86 redesigned itself to be RISC under the covers and took the wind out of the sails of mips, sparc, powerpc, alpha, and so on because it could run the same software at the higher speeds. (Backwards compatiblity means you carry along your existing customer base to the new iteration, and snowball to dominance via network effects. People thought a massive performance advantage would overcome that, but didn't realize Intel could capture that performance advantage while retaining binary compatibility by sticking an instruction translation pipeline on the front of its chip that re-wrote CISC instructions into RISC instructions on the fly, I.E. the Pentium.)

The Pentium was introduced in 1993, doing the under-the-covers RISC stuff. So by 1995 the PowerPC hardware IBM was porting OS/2 onto, which was supposed to outperform everything, ran slower than then-current x86 chips. The effort was clearly doomed, but IBM had spent years on it and were going to finish it rather than cancel it and have to explain the write-off to their investors. Unfortunately sucking away years of development from OS/2 at a fairly crucial time, to work on some cul-de-sac that had been politically important to IBM's hardware side but not important to any customers, essentially doomed OS/2 by giving Windows time to entrench itself. (Although what _really_ doomed OS/2 was that when they _did_ start to have decent retail uptake, they plugged it into the standard IBM tech support phone system that cost them $35 to field each tech support call on a box of software that retailed for $49.95. Two calls to the 1-800 number and they lost money. IBM upper management _did_ actively sabotage OS/2 starting around 1993, and that's why.)

Anyway, we got OS/2 for the powerpc running by the end of 1995. (Even though it couldn't natively compile itself, everything was cross compiled from x86 OS/2 via a special watcom cross compiler; I showed them EMX, the GCC port to OS/2 I'd been using as a hobbyist for years, but they weren't interested because it wasn't "professional".) OS/2 for the PowerPC "shipped to a shelf", meaning you could _technically_ order it but only by knowing its catalog number, which they never told anybody, because if anybody HAD bought it they'd have had to train and staff a tech support line which would be exorbitantly expensive and they were desperate not to.

As we were finishing up OS/2 for the PowerPC, IBM announced a site consolidation: the Boca Raton facility where the PC had been created was closing down and being sold (it eventually became a retirement community), and we could either leave IBM or move to Austin Texas. They tried REALLY hard to get everyone to move, taking us on what I called the "bribe trip" to see "Austin as you will never be able to afford seeing it again" (boats on the lake, ranches way out in the boonies, parties on the top floor of a skyscraper!), and paying large relocation bonuses (I used mine as the down payment on my first condo). And it _also_ meant we couldn't go anywhere _else_ within IBM, it was Austin or quit. (Which meant I couldn't go to the Hursley England site where all the Java development happened. I applied for a job at Sun Microsystems, but by the time they called back I'd already signed the "yes I will go to Austin" contract and was too young to stand up and break it. The contract said I had to give back the relocation money if I didn't stay a year; lots of my co-workers were locked in for 2 years, although they got more people than they expected to move and paid people to retire early on _top_ of the relocation bonus. My team lead Pete Rodriguez (who is ungooglable because there's like 30 of him) spent the last week at the company a year or so later staring at the ceiling and laughing every few minutes, he apparently got a _good_ deal...)

I arrived in Austin in February 1996 to work on OS/2 Warp 4.0, the x86 release we _should_ have all been working on at least 2 years earlier. There were all sorts of fundamentally political compromises in the code, such as the fact the main filesystem driver (HPFS, "High Performance Filesystem") was 16-bit 286 code and thus ran slowly. Why? Because the 32-bit version was full of Microsoft copyrights and cost them extra (royalties) to deploy. But because a 32 bit version already existed, they wouldn't spend money developing a _new_ 32 bit version that didn't have any microsoft copyrights in it and which could be part of the base OS rather than an extra expensive add-on nobody ever bought. (IBM had a mindset that code that cost money to build was worth that money, and thus removing code was WASTING MONEY. It was insane. We did our best to work around it.)

I'd worked on something called Feature Install on the PPC version (inheriting it from contractors whose contracts weren't renewed, and who literally used variable names like ldkopqvzc and ldkopgvzc and yes differing only by a q and a g in the middle of word salad is a real expample). They'd basically programmed for job security and _dared_ IBM to fire them, and their bluff was called and I inherited the mess. And they wanted to port this to the x86 version. I was 23 and had been programming since age 10: I took a flamethrower to it, unintimidated. My team lead Pete did his best to shield me from management's notice.

On the one hand, Feature Install was a package management system, a brand new idea at the time (Linux distros have rpm and dpkg but other operating systems generally _didn't_, you extrated zip files and extracted new zip files over them when you got an update). Having a package management system is great!

On the other hand, Feature Install was part of their object oriented desktop code (the "workplace shell", based on IBM's System Object Model), and the original idea had been to subclass a file folder so that when you dragged and dropped it from removable media to the desktop, it would install the package! And then they hit the problem that most of what they wanted to install didn't fit on a single floppy, so had to be split up into multiple disks, and from then on the implementation was fighting against its original design idea. And now they wanted to use it to install the operating system, which meant we had to bring the desktop up before we could install the OS, and the desktop code was NOT designed to work that way...

It was a mess. I had my hands full making it work at _all_. I still remember staying late and coming in on weekends doing a massive cleanup/rewrite of some of the fundamental plumbing (because what was there DID NOT WORK), and having a big flag day rewrite replacing over 1/3 of the codebase with 1/10th as much replacement code (making it table driven, this affected the GUI that edited the fields and the code that used the fields and the save/load logic and made it all consistent and happen via a common codepath)... and my team lead telling me to check it in on a sunday so management couldn't object...

And then it turned out my manager was there on that sunday, and walked by asking me what I was doing, and then told me not to do it. And then TWO DAYS LATER the testing results of the old code came in and failed spectacularly (one metric took 20 minutes to do a thing my code could do in 3 seconds) and the manager went "oh, we have a performance patch" and told me to check in my code... and took credit for 3 months of my work as a "perormance tweak" he'd done in response to the bad test results.

That manager's name was Kip Harris. I _almost_ quit right there. He caused 50% of my department to quit (they couldn't transfer to other parts of IBM due to the relocation, so they left IBM. This is back when that basically didn't happen). He was then demoted out of management, and the new manager (Jim Segapelli) had a reconciliation meeting with what was left of the department and gave me the biggest raise he could (something like 15%, although as a recent graduate it was 15% of $36k/year so not _that_ big) after the previous manager had given me the lowest performance evaluation ("more is expected", although that was partly because IBM had a quota system for rankings, copying Microsoft's stack ranking which started as a way of doing stealth layoffs and then got retained permanently for a decade, which IBM copied because microsoft was doing it).

1996 was not a fun year for me. They gave me a pager, and used it between midnight and 4am three times in the same week. And I did not get paid overtime.

Anyway, IBM had us heads down doing OS/2 4.0, which kept us too busy to notice it was too late for whatever we shipped to matter. In November 1995, Windows 95 shipped, which was still terrible but _less_ terrible than the 3.x line. Instead of crashing hourly it crashed daily, which meant it was approximately usable, which made it the first version of Windows that its entire customer base _wasn't_ looking to actively replace on a daily basis. I.E. the market window for OS/2 to become the dominant PC OS had closed, but we didn't let ourselves acknowledge that until we shipped our own thing most of a year later.

When OS/2 4 shipped, we were finally allowed to interview elsewhere in the company to look for another project to work on. I interviewed for an AIX position working on X11 (because it sounded like fun and would make me learn graphics and the guts of that X11 stuff I hadn't been able to get to work properly under Linux), and the manager told me to my face that I was too young and shouldn't work on this dead-end Unix stuff because it was dying. I brought up Linux... and he hadn't heard of it.

I knew he was clueless. I was still following Linux development, or at least I pulled up the web archives of the Linux development newsgroup from time to time and conirmed it was chugging along. But I couldn't get it to run on any hardware I had. I installed the OS/2 port of XFree86, which is where I first encountered the sparkling/tearing problem XFree86 had with western digital graphics cards. I was thinking maybe I could learn enough to fix it... but the manager dissuaded me from taking the position. Even though I thought IBM was wrong about X11 being dead, if IBM thought it was a dead technology then IBM would sabotage its own version of it and I'd had enough of fighting IBM management to learn and do things.

Tomorrow: My Java years.

We're coming up on the 20th anniversary of my return to Linux. I'm in a reminiscient mood.

Summer of 1982, between 4th and 5th grade for me, my family moved from Kwajalein in the Marshall Islands to Medford, New Jersey. (Talk about culture shock... Urgh.) Every summer on kwaj we'd spend a month visiting the mainland USA, traveling around, and before settling in New Jersey we did our usual round of visiting relatives, which means over that summer I saw grandmother's Atari 800 with a basic cartridge and floppy drive, and I was HOOKED. (My father put the floppy in 90 degrees off from where it should be and I figured that out by the scratches the read/write head left on the sleeve.) When we got home I wrote basic code in a notebook with pen and paper, writing a program that was a series of print statements explaining how basic worked. (I was 10. My fascination got a bit meta.)

I pestered incessantly for us to get a computer, and Christmas 1982 my father got the family a Commodore 64. (Which was not an Atari 800 but I had no say in the matter and Sears was having a sale.) By new year's the "family" computer lived in my room. Probably for my birthday in February I got Zork I, although mostly I played pirated games copied from friends. (I was a kid with no money, I wasn't exactly costing them a sale and I knew it. I bought what I could, which wasn't much.) A couple years later I started dialing in to a local bulletin board system "The Realm of the Dragon II", running Dragonfire BBS software which its sysop had written in C128 Basic. (The computer came with a 300 baud modem that plugged into the cartridge port, and a card for some free hours of compuserve, but I never used them because I couldn't pay for online time and didn't want to use up a resource I couldn't replace. A friend borrowed the modem for a week a few months later, and from them I learned about local bulletin board systems we could call for free. It took a while to convince my parents to get a second telephone line, but tying up the first one a lot eventually did it.

I wrote my own terminal program (in basic, compiled with "Blitz!" so it could keep up with my 300 baud modem) I called "junkterm" because I knew I wasn't much good at programming yet. My online handle was "Greeny", spelled in orange using C64 graphics. (I needed a cable from the sound output to the modem to do touch tone dialing, so I taught it "pulse" dialing by rapidly hanging up and picking up the phone, timed with for loops.) And then following up on that I wrote 3 different bulletin board systems in C64 basic, but didn't really run them except briefly for testing, because I only had one computer and didn't want to tie it up (the system had 65536 bytes total address space and 39811 basic bytes free when the OS ROM was masked in and running. Multitasking was not a thing in that context)/

I did everything in BASIC. I dabbled with assembly language a bit but had a really hard time debugging it. (Really machine code, my "assembler" was typed in out of a magazine and wrote to memory directly with hardcoded offsets for all the jumps... I think that's a "machine code monitor" but it's easy to be unclear on the distinctions when you've never encountered all the different options.)

My friend Chip (he picked it to replace "Lawrence") had other computers (he'd started on a TI 99/4a and moved on to a 16 bit PC running DOS), and in 1988 that friend ran a WWIV bulletin board system on a 16 bit XT clone that I helped him apply "mod files" to. WWIV was shareware that gave you full source code when you registered (which you would then compile with a pirated copy of Turbo C, and later Borland C), and a user community that had never heard of the patch command explained to each other in mailing list postings about the cool new change they did and how you could make THIS part of the code look like THIS instead... Yes I learned C by modifying WWIV source, and got a book called "From Basic to C" out of the library to try to understand what I was doing, then in 1990 I borrowed a friend's book called "How to Program in Turbo C" by Howard Schildt and _memorized_ it (to Roxanne's "Look Sharp" album, on repeat) to get full coverage and connect everything up. (Yes, I'm aware Schildt is considered a terrible programmer, but you've got to start somewhere and "the textbook is often wrong" is an important lesson for everyone. I also found a compiler bug in Turbo C my first year where an increment would get optimized away -- not in the assembly output -- unless I suck in a printf() to check its value, in which case it came back. I turned it into "varable = variable + 1;" which was not optimized away for some reason... And I was PISSED when I upgraded Borland C++ from 2.0 to 3.0 and suddenly my throw() function was now a reserved keyword. But I digress...)

In 1989 I skipped my senior year of high school and went straight on to Burlington County College (I still regret not skiping the first 3 years). I graduating from BCC at the end of 1991, and started at Rutgers spring semester of 1992. They had a computer lab full of Sun workstations in the process of switching over from SunOS to Solaris: very expensive and very unixy and yay C but the compiler on them was DEEPLY broken (it returned a "char *" from malloc(), what?) and I mentally categorized them as "big iron" that the PC would steamroller.

Meanwhile at home, I took years of christmas/birthday money plus everything I earned tutoring other students at BCC and bought my first 386 PC (pile of parts out of Computer Shopper), and put together my own DOS box and wrote my first bulletin board in C (including serial interrupt routines! Fancy. I got 8250 ones out of a book then looked up how to enable the 16550a receive buffer. Transmit still happened spinning a character at a time, because that didn't drop characters.) That one was called "chamelyn" (filename had to fit in 8 letters) which could theoretically emulate the UI of all the other bulletin board systems I'd used.

The way it worked was I created a very simple scripting language that was more or less an assembler for an 8-bit assembly language and the C had an array of 256 function pointers where it would for (;;) execute[code[position++]](); and then wrote the actual BBS part in the scripting langauge. And yes this means I'm one of like 30 people to independently reinvent "bytecode", and when I found out about Java later I (A) realized I'd been working on it about 6 months longer than Sun had, (B) pointed out they'd missed truncate() and couldn't shorten an existing file without deleting it. (Sun Engineer "Mark English" replied to my email that I was right and I'd just missed the Java 1.0 cutoff but he'd add it to 1.1.) I then spent a lot of evenings implementing a java version of the "deflate" algorithm from info-zip, and had just started work on the decompression side when 1.1 came out with zlib bindings in the standard library working about 10 times faster than my java native version. :P

Anyway, in 1992 I found out about C++ in my language survey course at Rutgers (which coverd Lithp, prolog, and C++), so I started another bulletin board system from scratch in C++ called "xblat" and I got THAT one connected up to fidonet. (Wrote my own fidonet tosser/packer, but used binkleyterm+zmodem as the network front end. Somewhere I have a random 9999 messages from the fidonet SF echo from the early 90's archived, because the message database cycled itself and that's where I moved and unplugged it, and never bothered to set it up in the new place because I got dialup internet through flash.net.)

The other change between chamelyn and xblat was I got a copy of Desqview (finally, multitasking! Run the BBS while using the computer!) which didn't work with my serial interrupt routines (Desqview had simple round robin scheduling and if the serial drivers were within an image then the interrupt was blocked when that image wasn't running, so there were multi-milisecond gaps with no serial port service, and characters got dropped all over the place), so I had to get FOSSIL drivers working for the serial port and teach my BBS to use them. (Binkleyterm already knew, and could do the baud rate setup and such.) The trick was the FOSSIL drivers ran _before_ desqview, so the interrupt routine wasn't managed by the multitasker, and the "gimme the data you've collected" call still worked from within a managed DOS instance...

This is the context in which I encountered Linux: in 1993 the 4 sls floppies came across fidonet, and I went "this is very interesting, this is a whole operating system distributed the way binkleyterm and zmodem are, it's like WWIV except you don't have to buy access to the base you apply all your local changes to meaning it's hobbyists all the way down. But... why would anyone want to clone a Sun workstation? It can't run DOS programs, so I can't run my BBS or games or any other existing program I have under it.

I took it over to Chip's house and we tried to install it on one of his PCs and xfree86 WOULD NOT WORK with his Diamond Stealth Multimedia graphics card (well, mcga 320x200 mode was the highest resolution it could do). I dug into the Linux mailing list archives to see if I could work out how to get it to work, and they said manufacturers wouldn't release programming information and that this would probably never get fixed. (Years later "trying to get X11 working on any graphics hardware I had" would be the main blocker to switching to Linux as a desktop on both western digital and SiS chipsets. If the X.org fork had happened 20 years earlier maybe Linux on the Desktop would have too... But I digress.)

Circa 1993 my professors at Rutgers were talking about how the big thing coming up was SMP (because linear CPU speed improvements had to end _sometime_), which would require threading to take advantage of, and I dug into the archives for that too... and found that none of them had SMP hardware (too expensive) and a long post by Alan Cox (#2 guy in Linux) explaining that threading was a bad idea and shouldn't _be_ supported...

But meanwhile, DOS was clearly on its last legs. The 640k barrier and lack of built-in multitasking were a PROBLEM. (I eventually worked out exactly what was going on and co-authored a paper about it, but at the time it was just the "smell of death" anyone who grew up on an 8 bit system learned to sense around a platform it was time to move off of.) Meaning I wanted to move from DOS to _something_ else... and Windows 3.x was BAD CODE, being too unstable for words was just the SURFACE problem. And Microsoft had gone full-blown evil trying to shove it down everyone's throats, to the point I installed IBM PC-DOS then DR-DOS to not be running Microsoft's version.

I wisted about Desqview X but never found a copy, and when OS/2 3.0 came out (the 32-bit 386 rewrite) I _bought_ a copy (real, actual, legitimate with money!) and installed that. It supported SMP and threading, IBM was seriously pushing it for home users (with the "nuns" televison ads and so on), it had good backwards support for DOS with a cleanish 32 bit migration path...

This is long, I should cut it here and pick up later.

My mcm-buildall.sh script currently builds an i686 cross compiler (dynamically linked, with lots of optional features like thread support disabled), and then builds _another_ i686 cross compiler statically linked with the options switched on.

I thought I could simplify this script so you can go "~/mcm-buildall.sh m68k::" and it would build the -host cross compiler and then build the static m68k cross compiler with that. I tried it. The m68k cross compiler build broke in the gmp build because one of the components segfaulted when built directly with simpler cross compiler.

Bravo, gcc developers. You know my rant about how a compiler isn't fundamentally different than a docbook to pdf converter? While that remains true, the FSF is fundamentally terrible at software and screw up everything they even peripherally touch. (Also, cross compiling sucks because the number of hosts is _multipled_ by the number of targets when working out how many different codepaths need testing. And documenting _why_ you need to do these elaborate rube goldberg build setups is kinda horrible. My motivation here was to simplify the build so I didn't have to _explain_ it...)

On that note, glibc is appalling.

I've previously mentioned my corollary to moore's law, that 50% of what you know about programming is obsolete every 18 months. And that the reason for the longevity of unix is that it's mostly been the same 50% cycling out over and over ever since midnight, January 1, 1970.

The flattening of moore's law's S-curve has slowed but not stopped this cycle, and I'm waiting for systemd to go the way of devfsd and hald. Some bits do flake off over time (sccs->cvs->svn->git), but the portions of unix that are "old but still relevant" are as close to universal constants as we get in programming, and yes I throw the C language (but NOT C++) in that pile.

*shrug* Time will tell, but fragmentation seems less likely to form a new plateau. If lua had been the browser language instead of javascript the world would have moved to a new baseline already. (C is from the people who made Unix. Go is from the people who made Plan 9. Sure, they're the same people, but context _matters_...)

(C is a portable assembly language, however much the C++ folks shriek Luke Skywalker's "No, that's not true, that's impossible!" line every time somebody points out the obvious. And strive to sabotage compilers with endless Undefined Behavior to try to screw it up. Just admit that signed math is two's complement and make LP64 part of C20 already...)

I have a dozen things queued up to do and what do I spend time on? Fixing ping. And not even the improved error reporting suggested on the list, but reviewing the code to see where that should fit in and then testing corner cases shows me the error reporting stutters (division by zero error in the summary display logic when no packets have been returned, which triggers a signal that... calls the summary display logic again at exit.) And -c isn't working when you ping a site that doesn't reply (it's limiting returned packets, not sent packets), and it looks like -w isn't working (haven't tested yet, just reading the code and seeing a dodgy "else" handoff)...

Upstream Linux broke LED platform data on sh4 when they converted it to device tree. They still allow platform data to pass in a pointer, but they changed the structure type that pointer dereferences, and the definition of the structure is local to the driver consuming it so the platform data CAN'T provide it. (There's still a generic structure providing all the info in a driver-agnostic way, but the device tree conversion changed the code to no longer _use_ it.)

And of COURSE the device tree guys' response is "convert everything to device tree, you have no other choice, we're part of systemd now" or some such...

*shrug* I have a local patch that makes LED platform data work again without breaking device tree (I think, haven't got a device tree thing to test but they obviously never tested platform data so...), and I posted that patch, and I'm using that patch. If they don't want the patch, vanilla can stay broken. As usual, add a todo list item to poke 'em again in a year to see if whoever was objecting has died yet.

Looking at watch again: "watch ls --color" prints "^[01;34mandroid^[0m" and similar. And "watch 'date ; sleep 5'" produces no output for 5 seconds, and then updates the display every 5 seconds.

That's... fairly simple behavior to implement. It's also wrong, I want toybox to do _better_ than upstream here.

(Sigh. I always get screwed up by singular/plural in directory names. Is the url "download" or "downloads"? Is it "tests/files" or "test/file". I _fairly_ consistently use plural, but not quite enough to get it right, and lots of directories in URLS and packages and such aren't mine anyway.)

And how does "quoting arguments" work? According to strace, "watch echo 'one  two'" becomes exec("sh", "-c", "echo one  two"), which results in an output with just one space between the one and the two. (The downside of washing the command line through sh -c, passing through argument quoting becomes darn near AI-complete. Looks like they basically don't try to get it right.)

Day off today, poked at toybox stuff. Didn't get as much done as I wanted, but then I seldom do. Looming end of available time limits what I'm willing to start on, most ratholes are too deep to go down, I have to leave off halfway and then next time I've made new work for myself rebuilding the context where I left off. It's not bad if I know I can get back to it the next day, but I usually can't. So I pick at things with little or no momentum.

I emailed to ask if Google might be interested in buying a "toybox support contract", and heard back that although it's not the decision of any of the people I regularly communicate with (they're all engineers)... the answer is basically no. Google doesn't like to use vendors, and the corporate side doesn't see the point in investing in a command line that already does what they think they need.

*shrug* Can't say I'm surprised. I got turned down when I first asked if they just wanted to _use_ it. If I was easily discouraged I wouldn't have gotten this far. But having to keep digging away in my own time to solve these problems in spite of the large institutions is painfully slow, it would go so much faster if I didn't have to spend most of my time and energy working on something else entirely to support a family.

Large institutions have a different mindset than individuals. I need to do a proper version of my 3 waves talk. (I'm still sad that when I _did_ a proper version, which I was pretty happy with right after I'd given it, Chicago's Flourish conerence never posted the recording. Sigh. I could propose it as a talk at another conference, but I haven't been doing that this year because I'm sick of hearing from white guys and don't want to take up a slot that should go to someone else. Really I should just record the talk myself and put it on youtube where it's not _my_ fault it's completely ignored. (Yes, I say that with the original article series on The Motley Fool having gotten something like 17 million views the first year, being reprinted in their "popular articles" section years later, getting third party commentary, etc. My brain does not accept this.)

The problem is without an externally imposed deadline and an audience/editor who would be disappointed if I didn't finish by deadline, it stays on the todo list forever. (I put "podcast" as a patreon stretch goal in hopes of creating a deadline/audience without having to scrape up several hundred dollars to travel at my own expense to some hotel in some city where half the time they WON'T RECORD IT ANYWAY. Grrr...)

Went to the big protest in front of the courthouse, somebody was handing out posterboard squares and letting you use a market to make your own sign, so I did "ICE at the poles, not at the borders". (Something somebody said on twitter.) I brought a powerade bottle, but it wasn't enough.

It was nice, the only dip was when they had a grey haired old white guy speak (we've had enough of that, thanks, and I say that as one), but that was just one speaker out of a dozen or so.

Next time: sunscreen, two water bottles, make a sign ahead of time.

Politics is horrible and draining and relentless, but while doing a version march from 3.18 to 4.14 at work I hit "Temporary per-CPU NMI log buffer size" showing up as a new config option under the RCU Subsystem and still had a burst of rage wanting to smack the linux-kernel development community at large for unnecessary overcomplication.

So there's that.

(A version march is where you try the release versions in sequence because the config and board support patches need so much manual adjusting every couple releases a bisect is crazy to try to get anything coherent out of. I stopped at 4.14 because 4.15 has some intermittent flash/jffs2 corruption bug that eats the filesystem if you write enough to it, but is intermittent _enough_ it's annoying to track down. No idea why it's happening. Stopped at 4.14 or the moment, that's relatively recent.

Got the new m68k toolchain and target built in mkroot. Needs vivier's qemu-m68k, but otherwise pretty much works.

Work's got a giant multithreaded application that's trying to do realtime tasks, and they have a shared library that calls system() and popen() to run subtasks. Naturally: that doesn't work right.

If you fork() from a thread, the normal copy-on-write semantics don't apply and the new process has to copy all the process's memory to the new PID, which can easily be dozens of megabytes. (The problem is threading is already sharing it, and having separate processes _and_ threads share the memory would require each page to have _two_ reference counts to keep track of the different categories of sharing, which would bloat the page tables.) Forking takes locks that block system calls and page faults in the parent process, which is not a problem when a single-threaded parent process hasn't returned the fork() system call yet, but it means other threads of that process can't do it either. Copying lots of memory (dozens of megabytes) takes a long time and happens under these locks. Between the two of them this manifests as a large latency spike in the existing process's other threads while the fork() is happening. We were measuring 70 miliseconds on an embedded board.

The fix is to use vfork() instead, which only blocks the parent _thread_, not the parent processs. It doesn't copy memory (the parent and child use all the same mappings, even the stack, with the parent blocked until the child calls exec() or _exit(). Of course using vfork() properly is tricky (mostly because very few people these days seem to understand what it _does_), so I wrote a vsystem() and vpopen() based on vfork that they can replace all their system() and popen() calls with. (Luckily, I'd already done most of that for toybox so I could crib from my own work.)

Way back when I wrote up a couple pages on vfork() for the busybox FAQ, but after I left the busybox guys crapped a long busybox-specific digression into the middle of it (something about configuring and building the busybox binary?) that made it useless for explaining vfork to people. I should dig up my old text from before they broke it and put it in the toybox FAQ so when I want to explain "why vfork" I can link them to an existing thing. (It's on the todo list. I don't have as many half-finished FAQ entries as blog entries, but it's a similar category of problem.)

Swapped out my phone battery with a new mail-ordered one. The new one lasts MUCH longer.

Making another try at an m68k toolchain build. It's being stroppy.

Editing another batch of blog entries. I got up to an entry that trails off halfway through a technical explanation, which is where "editing" turns into large amounts of "new authorship". But it's a writeup of what I was thinking at the time, so it counts. (Autobiography is seldom written live as it happens.)

(Then again I left myself a "Did my old realtime Java GC idea ever get implemented?" note as the whole blog entry a week ago, and that's a longish writeup to properly explain, which I haven't done yet. My blog has significant technical debt.)

Cut a toybox release yesterday, so today I'm tying off some things I bumped until after the release. One of them is splitting up the ps help text so "ps -o help" (or any unknown -o field) shows the field list, and the normal ps --help is the other half.

This of course led me to thinking that if I'm breaking this into a function anyway, what I should REALLY do is move the help text snippets for each option into the option array, and just have a function traverse it to generate help output. It would have to make a couple passes because I have the six different variants of "show the command name/line" broken into their own sections, but that's easy to detect/categorize in the table so sure.

Except there are two multiline entries in the table, and it might make sense to move those into a third section. And this led me into looking at the list of "S" field output, which has a bunch of magic letters that mean stuff, and what the heck is "wakekill" anyway, so I went down the rathole of trying to figure out where I got that from, and... it doesn't seem to be in current kernels?

Table 1-4 in Documentation/filesystems/proc.txt just says "state (R is running, S is sleeping, D is sleeping in an uninterruptible wait, Z is zombie, T is traced or stopped)" which isn't all of them, so I dug down into the source until I got to fs/proc/array.c task_state_array (line 140-ish) and that's got RSDTtXZPI but still no K... Ah, here's wakekill.

I have no been away from the cats long enough to miss them a little. Not enough to want to have any up here in Milwaukee, but enough that I am not Actively Overcatted.

(I wanted to upgrade from pets to kids a decade ago, but that's not how my life turned out. Oh well. Cats started registering as "this is what you got to have in your life instead of children" and it got old.)

A really good tweet led me to read about the history of dinosaur extinction research, and science mainly advancing when old fogies die remains very true. As does the observation that when someone's conclusions remain constant but their professed reasons for coming to that conclusion constantly change (same result, different justification), something is wrong.

I remember one of my school teachers had a newspaper article about this, back around 1990, which was quite detailed about the decades of research the local Yucatan oil drillers had put into examining this big crater they'd found, and their inability to get rich white english speakers' attention, and then I'd tell people for over a decade afterwards "oh yeah, they found the dinosaur impact site, it's just off the bumpy bit south of panama" and nobody believed me.

Same with the story of the guy going "no seriously, H. Pylori bacteria cause ulcers, you can treat this chronic disease with antobiotics" and the rest of the medical profession ignoring him despite that one being really easy to test. (Everybody thought ulcers were caused by stress, it was a plot point in the 1979 Disney movie "The North Avenue Irregulars" I had on videotape growing up, but in 1984 he ran an experiment proving it couldn't _not_ be a bacterial infection no matter how much senior medical professionals insisted it was stress because it had always been stress, and he got the Nobel prize for proving them wrong 20 year later after enough of the old fogies he'd offended by having the facts on his side finally died.)

The interesting part of the long wikipedia history of the extinction research above is how many different ways it got confirmed with people still flipping out. (Lots and lots of "3/4 of all species go extinct at the K/T boundary layer" and "irridium at the K/T boundary layer consistent with asteroid impact found at dozens of sites around the world" and "there's this K/T boundary layer AROUND THE ENTIRE WORLD, seems like a thing" being responded to with "crater site or it didn't happen". I grew up with that question for about 10 years and then "Oh look oil drillers found a HUMONGOUS CRATER the same age as the K/T boundary, they found it years ago but they were poor spanish-speaking brown people in south america who sent letters to white dudes that the white dudes never opened, and presented at conferences that no white dudes attended. Funny that."

The important part is that the people insisting the world was the way it wasn't, and who had seniority and power and could make everybody ACT like the sky wasn't blue, finally died. And then of course painfully obvious truths wer acknowledged; the emperor can have no clothes once he's in the casket.

(Today's "huh" is actually the diabetes vaccine, although a decade ago they thought injecting capsaicin into the pancreas would shock it back to normal so who knows. "Everybody's wrong about this" doesn't mean they _are_. Just that you have to remain open to new evidence.)

We're seeing garbage collection latency spikes in the mono app at $DAYJOB, which is frustrating because I designed a realtime garbage collector back in the 1990's. I wonder if it ever got implemented?

Hmmm, I'm not finding a writeup of my old idea to link to. (The blog I had at the time was on flash.net, it's not even in the wayback machine and I lost those files when a Zip disk got the Click Of Death.)

Back in 1997 I was thinking about both java GC latency spikes (the screen saver in IBM's powerpc port of JavaOS would freeze visibly every couple seconds), and how to extend java references to 64 bits (because clearly that was coming after we dealt with all the Y2K bugs). The obvious way to do the second was to have the actual reference be an index into an array of pointers (and when I later described this idea to a professor at UT he said that's called an "object table"), and it occurred to me doing that could make garbage collection fully asynchronous.

Garbage collection needs to know "is this reference still used or not", which is one bit of information. So have a bitfield with one bit per array index, and at the start of garbage collection memset the bitfield to zero. Then as your garbage collector walks the global variables and down each thread's stack, set the bit of each reference you find (basically "thingy[index>>3] = 1<<(index&7);").

The trick is during garbage collection, any java "assign a reference" bytecode should set the bit for the reference it just assigned to a new location, because otherwise it could move it "behind the back" of the garbage collector to somewhere it's already checked. That way you don't have to freeze all the threads to run your garbage collector: the still-in-use bit gets set either way.

There are of course implementation details. You might need as many as _three_ bitmaps to do this (one an allocation bitmap when creating new references, which becomes the "output" of the gc process, a second for the references your GC has confirmed are still used, and a third for the ones you've found but not recursively looked at the dependent objects for yet).

And of course there's a bit of fiddlyness to make sure you don't walk off the end of the stack of a thread as it returns from functions, but as long as you bounds check the values as not off the end of the object table, worst case scenario of reading garbage that you think is a reference is you just misidentify a lost object as still used until next gc run. That could create a false positive but never a false negative. (And if you do implement a bit of locking at the end of a single thread's stack to avoid that, it's still _bounded_ latency. You only ever need to freeze one thread at a time for the duration of examining _one_ function's local variables, which might be easier to implement as "when the GC in progress flag is on, returning from a function checks if the GC is in this function, and if so waits until the GC has left this function before returning").

Doing this would even let you pack your memory: copy an object's memory to a new memory location, then move its reference in the object table. All the scattered references to it in objects and local variables and such are just an index into the table, which doesn't change. The actual memory location's in one place that can be changed atomically. You can use mprotect() to yank the write bit from the old copy to pause threads that try to write to the old copy during the move and it's still bounded latency. Heck, you could fixup the write attempts in the fault handler _during_ the move if you wanted to, so the size of the object being copied doesn't affect the latency.)

Of course this was back before JIT, when bytecodes were interpreted in a loop, so switching between "doesn't set the bit" to "sets the bit" modes when you start a GC pass would be easy. (Setting the bit _all_ the time would be impolite to L1 cache.) And it's a realtime thing rather than an actual optimization (avoids latency spikes, but slower over all due to the extra dereference in every object access, although modern processors with large caches and deep pipelines honestly might not care).

But I got busy and didn't pursue it. (After all, if I could think of that obviously other people doing all the fancy JIT stuff had to be way ahead of me...) I did describe the idea to that UT professor I mentioned (well, adjunct faculty? I think he was the assistant for the Data Mining class I was taking), but I got busy and didn't sign up for any classes next semester and didn't see him again for months. (Which means it must have been fall of 1996 when I spoke to him, that's when I applied to grad school and took one Data Mining class, then stopped for several years before trying again.)

Amusingly, I bumped into the professor again (I forget his name; white guy, dark beard, barely older than me and I was in my mid-20's) a year or so later at one of the Mensa Thursday night dinners at HEB Central Market's little restaurant thing, and he said IBM (which I'd stopped working for by that point) had offered a low-five-figure grant to sponsor work on my realtime garbage collection idea... but I'd never responded to the email he'd sent to my student email address (which I'd never asked the university to give me, and had never logged into; I had a home email address, he hadn't used it). I assumed at the time IBM had implemented it themselves if they thought it was worth doing, and got on with my life.

And now 20 years later I'm hitting a problem I designed a solution for right after I graduated from college. Frustrating. (Obviously I'm not personally rewriting Mono's garbage collector from scratch any time soon. But dude, how is this not obvious?)

Back from austin. Jetlagged by the redeye flight.

Finishing up fmt.c.

Thursday already. Tempus fugerunt. Far enough into my week away from $DAYJOB that I start culling my todo list because it ain't gonna get done this time.

Last night I opened one too many chrome tabs and my netbook ran out of memory to the "can't move the mouse cursor for half an hour because it's swapping" level. Once upon a time Linux used to have this thing called the out-of-memory killer but people complained it might kill the wrong process, so now when it would have triggered it instead hangs, to the point it won't recover if you leave it running overnight, and you need to reboot and lose all open windows/tabs on all 8 desktops instead. About what I've come to expect from Linux on the Desktop: lateral progress. Stuff that used to work no longer does. (A companion to "Something must be done, this is something, therefore we must do it." The old Linux Luddites podcast motto: "Not all change is progress.") Anyway, rebooted and trying to figure out what I was doing now all my context's gone. And my blog serves its original purpose! (Notes-to-self: what was I doing again?)

Checked in the do_lines() semantics change with changes to sed and cut, found a bug in cut (regression from October where adding a pedantic posix compliance corner case broke real use elsewhere), complained on the list and fixed it. Still haven't gone back and redone fmt yet.

Spent most of the day watching Fade play Revenant Kingdom, my biggest achievement before dinnertime was getting out to the credit union to activate my new debit card. But I should get a couple hours programming in, so headed out to wendy's with fully charged netbook and phone. Let's see...

I want to switch fmt to use loopfile_lines(), and it cares about the end of files so I want to change the do_lines/loopfile_lines shared infrastructure in lib to pass a NULL line to flush, and it currently doesn't which means I need to adjust existing users.

Context: do_lines() takes a filehandle and calls readline() in a loop, passing each string to the callback function. The callback gets a char ** so it can NULL it out if it wants to keep it (caller frees the string otherwise), or it can assign (char*)1 to it to skip the rest of the file. Then loopfile_lines() takes a list of files (null terminated char *[]) and iterates over them, calling do_lines() on each.

The implementation requires a glue function between loopfiles_rw() to translate argument syntax (loopfiles_callback(int fd, char *filename) to do_lines_callback(char **str, int len)), and the glue function stores the real callback in a global variable. This is slightly awkward on two levels: 1) and means you can't nest two loopfile_lines() calls (which hasn't come up yet), and 2) the global isn't in struct toy_context. (Keeping it near the user vs keeping them collated so sizeof(toys) shows how much global data we're using, both have downsides, it's one of those things where the stakes are small enough the relative cost of the less-right solution isn't easy to see, so I'd wind up agonizing over minutiae if I tried to fix it. There's a few static vars in lib/*.c, but no toys/*/*.c gets to have any so at least there's a rule.)

Anyway: changing the semantics of do_lines/loopfile_lines to indicate end of file because something cares now, and there are two current users: The "cut" command is using loopfile_lines() and doesn't care about file breaks so I can just add an if (!line) return; at the start of its handler function. But the other user is sed, which is calling do_lines() directly and it _does_ care about file breaks, but only in -i mode. And it's doing explicit do_lines(0, 0) flushes. Hmmm...

I'm in Austin for a week, taking time off from work. (I don't get paid, but I get to recover, see everybody at home, and work on toybox.)

I finished up and promoted ping, and now I'm poking at fmt which logically should use the loopfile_lines() infrastructure, except it doesn't have a flush call at the end of each file. The only current user is cut.c, which means sed is _not_ using it. i should figure out why.

These aren't necessarily the most important commands, these are the ones that are closest to being done. I need to cut a toybox release this week, and I should get kernel patches resubmitted (week 2 of the merge window), and I should integrate mkroot into toybox, and I should figure out what to do about microsoft buying github...

Set my alarm early to pack, wound up lying in bed doing the "I need to get up" but not actually moving thing. (My flight's somewhere around 6:30 pm so leaving work a bit early.)

Trying to finish up everything in the world at work in the meantime...

I get emails. Today I got:

Hi Rob,

I have seen your video regarding building a Linux system. You have been doing an amazing work in the Open Source world.

Quick Intro (myself): I am an Embedded Software Developer, an Operating System enthusiast. Love and Passion for Kernel internals, building easy to use systems.

My Problems: I have tried may open source distributions. None of them would work for me satisfactorily. I run into many issues. Like,

> Sometimes I try to install some drivers (for some hardware) and it won't work. There would be endless compilation errors.
> Sometimes window will crash, some application may crash.
> One day or the other, something would be broken.

There are many more issues, I can not recollect all of them at this time.

I know it is open source software, but still as an independent user (software developer) I struggle a lot.

How about creating an operating system that just works. Idea is to create a Linuxbook (like chromebook) and a whole ecosystem for that. I see a very good market for that.

I have list of features in mind which can bring good from all the worlds in to one system.

I would like to know your thoughts on this.

Which is why I'm trying to find a good introduction to the dunning-kruger effect that ISN'T a pile of smug superiority. So far I've found this.

We all start there. "Thousands of other people have tried, but I haven't yet, how hard could it be?" That's the dunning-kruger effect. The domain expertise necessary to figure out how difficult something is turns out to be exactly the same set of skills necessary to perform the task, meaning anything you have no idea how to do seems easy.

And it's not entirely a bad thing! Linus Torvalds said if he knew what was involved in writing Linux when he started, he'd never have started. Linus dunning-krugered his way into doing Linux and then played "oh, we need to change this" for over twenty years now. He's _developed_ buckets of domain expertise as he went along, but his initial "I can do this" bravado was based on a one semester course followed by copying a toy OS. (Luckily his "I don't see why we need to do this" rejection of microkernel architecture turned out to be right because the ivory tower academics were teaching BS [LINK tanenbaum-torvalds debate], but the newbie being right and the old hand not seeing the forest for the trees was largely coincidence.

(Although the confrontation boiling down to the newbie saying "Ok, explain it to me" and the expert not being _able_ to is an important corrective factor in science. Has been for centuries.)

Last job I spun my wheels in pursuit of world-changing goals. I really wanted to Do The Thing and we never made decent progress largely due to factors beyond our control (politics among the board of directors screwed up our second funding round leading to perpetual understaffing so we were all swap-thrashing between too many tasks, and Jeff always shot down any short-term "funding from operations" approach which was all _I_ knew how to help with, because it wouldn't bring in enough money to matter with the burn rate the company had). That turned into Giant Bundle of Stress, the money dried up and I went into debt to keep doing it, and I'm still recovering from both.

This job I'm well-paid to do a small near-inconsequential thing each day. My Big Project over the past month was a kernel version upgrade. Nobody outside the company is ever likely to notice and even within it only a half-dozen actually understand what a given thing is for. (My last two issues were "show the right mac address in the boot console messages on startup" and "the board should remember the DHCP address it had last time and request it again next boot". The larger project is that we're migrating networked climate control systems for large buildings from Windows CE to Linux because CE on this hardware was end-of-lifed.) But the work gets done. Things are finished, checked in, tested, and we move on. This company has no shortage of money.

Neither was what David Graeber's new book calls a "bullshit job", but each is only half of an ideal job. I'm strongly reminded of an old dilbert cartoon.

I look forward to the baby boomers dying so we can stop considering capitalism as normal, and cash in the past century of economic and technological progrss to finally get universal basic income. (There may be some "let them eat cake" between here and there. I really really hope the current administration means we're getting the Robespierre/Napoleon part out of the way _before_ guillotines come out, but I have no clue what the future holds. The millenials seem to be waiting for the boomers to die before taking any other action. Gen X has been waiting for the boomers to die... since Reagan was elected, I suppose.)

(Yes, I am aware #notallboomers. As with #notallmen, it's not a useful objection. This ain't getting fixed while Racist Grandpa is voting with the dregs of the confederacy. Society advances the same way it always has.)

(No, "people will just breed to consume the resources" turns out not to be true: if you educate women and reduce child mortality rates without providing significant support for child rearing ala subsidies and free daycare and so on, population falls below the replacement level. This has happened in every advanced civilization around the globe, it's a significant problem and part of the reason we have more old people than young people, and only places like finland have been effectively compensating. The average family size is _less_ than 2 children. The US population has only been growing recently due to immigration, and that's rapidly declining because nobody wants to come here anymore. Not even to visit.)

I've been doing cleanup for a toybox release, although I'm flying to Austin on the 8th for a week off from work, so might bump the release back another two weeks so I can work on it while I'm there. Or maybe that should just be the start of the next dev cycle...

Anyway, I'm going through the github pull requests, and I'm finally taking another look at izabera's commit adding unlimited precision support to sort -n. I rejected it the first time as unnecessary for sort (single precision float is fine), but... what with the offered bc implementation turning into a mushroom cloud of politics, plumbing towards that end seems good and doing math as ascii strings (via long division and such) is dog slow but should be reliable and easy to understand?

I should have put a release out last weekend (3 months), but I was exhausted and Fade was visiting and I hadn't noticed the date. So I'm chipping away at it this weekend, which is another variant of "closing tabs". Cleaning up the endless backlog of half-finished stuff, and doing proper writeups.

I'm tempted to move the ps -o field list out of the "ps --help" output and instead have "ps -o help" show the list, but... I'm not sure it's an improvement? It wouldn't put the remaining text down small enough to fit on an 80x25 screen (although I've got a patch getting it a lot closer).

And once I get zcat promoted I plan to compress the help text (at least for non-single builds), and this would move it out of that. (Although when I've got the infrastructure maybe I could add some COMPRESSED_STRING() macro to append large blocks of text to the compressed help field entry and gimme a pointer to it using the same infrastructure... But there's already too much generated/ magic going on. Hmmm.)

Finally got a stable kernel forward-port at work so we're not working on a 4 year old kernel. Not to current (some intermittent bug in the flash code keeps eating the filesystem?) but two releases back (4.14) seems stable.

That was an _exhausting_ 3 weeks

Youtube's strategy is to show way more commercials to annoy people with music playlists into paying for a subscription. (No really, they explicitly said this.) My reply is basically "better dead than red", and I just turn the volume down when it does a 15 second commercial between each song. (There's a dial on my headphones.)

But WOW there are a lot of car commercials (and car insurance commercials) they're spitting at me. Which are hilariously ineffective because I know that both are going away in a few years when google maps adds a "ride" button next to the "directions" button when you selected a destination, and a self-driving waymo thing shows up. That is a service I'll subscribe to. If Google wants my video streaming dollars they can buy netflix or hulu. (I doubt they're going to buy amazon and the video is part of the giant Prime hairball anyway.)

In the meantime the car companies (and car-adjacent companies) are desperately trying to squeeze out a last few dollars before closing time. Self-driving means car sharing which means you need 1/10 as many cars. Yes even at rush hour: there's "be there at 6 am" through "be there at 10am", and at a fairly pathological average of half an hour to get there and half an hour to get back to pick up the next person (current reality's less), that's 4 cycles per car without "surge pricing" leading to the sort of carpooling people already accept for airport shuttles.

There's zillions of bigger issues (turning all those parking lots into extra buildings increases density which makes cities function better, no more gas stations, geico goes away, jiffy lube goes away, no more car loans...)

Sigh. The fix for youtube being stupid is to load music onto my phone via usb (the stuff I had on there went away in the factory reset last year), but they made it stop mounting as a USB stick a while back and I hate setting up the funky magic windows protocol thing they think we should use these days. I suppose I should learn to do it via adb...

I'm overdue for a toybox release! Oops. (Feb 24 + 3 months was last thursday.) As with all toybox releases, the big push of work (after finishing whatever I'm curently working on) is going through the git log to see what I did so I can write up the release notes, which always finds dangling threads I want to tie off before shipping, things I don't really have a proper test for, and stuff I did the first 1/3 of and could probably finish with just another hour or two of work (it's never another hour or two, it's days)...

At work I need an sh4 system booting to initramfs from real hardware, and getting a glibc buildroot to fit in the 4 megs of flash reserved for the kernel in the partition layout ain't happening (the uncompressed /lib directory is 7 megs with just about everything switched off).

So I threw a mkroot cpio at it, and although i got it to boot... the eth0 address is 10.0.2.15, which is the qemu static IP, and I need dhcp. Hmmm. I need to wget stuff, which means it needs to talk to the lan, which means an address in the dhcp range.

I switched buildroot over to musl and am rebuilding, that'll probably be small enough to fit. But collecting data for what an actually _usable_ mkroot needs. Shell, route, and dhcp. Heh, and strace didn't build against musl with buildroot's toolchain. (I have the start of an strace in toybox I should find a few hours to work on...)

Right, switching back to mkroot and adding dhcp there... it builds toybox defconfig so grabbing the dhcp out of toybox pending is awkward, but I'm building busybox with a config file so switching on the _busybox_ one is trivial... ok, try that.

My limiting factor these days is more "energy" than "time". Work pays quite well, but leaves me exhausted. It's still a step up from working for a startup that couldn't reliably pay me, still left me exhausted, and didn't have a defined schedule so I could say when I'm _not_ working and thus focus guilt-free on other stuff. (What I did for them was never _enough_, but I was spinning my wheels inefficiently for a lot of it because I was spread too thin and couldn't rest and recover.)

That said at SEI I was working on potentially world changing stuff there that advanced the _definition_ of the state of the art along multiple major axes, and here at JCI we're coming out with a software update for climate control equipment in large buildings, which already worked. It's worth doing, but it's the programming equivalent of washing dishes. The challenges are of the "oh that's baked on, how do we avoid scratching the no-stick surface" variety. It can require close attention and cleverness, but the problem being solved isn't novel or of much interest to anyone else.

Still, I'm being well paid for work people find useful which is neither unethical nor a legal quagmire. The AT&T set-top boxes we were working on back at Pace (the position before SEI) were supposed to spy on the people who used them (monitoring what shows you watched on Netflix, for example, so AT&T could try to sell its own video streaming services to you. We weren't implementing that bit, but we knew it was coming because they _told_ us). The worst you can say about the JCI boxes is "maybe we could sell them a little cheaper", and the customers aren't complaining. Seem quity happy, and have been for decades: they do what they're supposed to, to the best of our ability to make them do that, and do not act against the customer's interest in any way that I am aware of.

For a Fortune 500 company, that's kind of impressive. (Ok, JCI moved its headquarters to Ireland a few years ago to dodge taxes. There _is_ evil going on. But we're not being directly asked to perpetrate it yet. As capitalism in 2018 goes, that's high praise.)

Memorial day. Fade went back on the bus at 2.

Poking at the hello world on bare metal stuff again, trying to strip down a userspace one first, and building gcc -nostartfiles --static for a version with write(1, "Hello\n", 6); is still linking in hundreds of bytes of useless __libc_disable_asynccancel crap I haven't figured out how to disable.

On the kernel side, trying to do an ELF version of Balau's example without the crazy linker script, you can "gcc -Ttext 0" to move the text segment to location 0 (where the arm reset vector apparently lives, so execution starts there. Yes there's more interrupts that could happen later in the table but for these purposes I'm going to assume they won't and just put the hello world there. (You'd think that the interrupt table would be a list of addresses the processor jumps to, but no. It jumps to a fixed location where you put a branch instruction, and the spacing is enough for "branch instruction plus address". Why? Because the Arm designers thought that was a good idea.)

The kernel has "make savedefconfig" which does something a little like the miniconfig.sh plumbing I have, but the format's different. Miniconfig is the deltas from allnoconfig, and defconfig is the deltas from the symbol default values (many of which are on).

I like miniconfig because _conceptually_ it shows you all the important selected symbols. The ones that if you started from allnoconfig, you'd have to switch on to get this configuration. This "defconfig" stuff starts from an arbitrary base and makes arbitrary changes to it, half the knowledge winds up in each places and the result doesn't necessarily mean anything by itself.

But it's what the kernel guys are using, and it's already merged. Inferior but deployed.

Went off caffeine this weekend. Spent a lot of it sleeping, the rest failing to make any progress on any of the open source stuff I want to do. A bit underclocked just now.

I didn't figure out a way to remove mmap() from the elf parsing stuff in file.c, couldn't find my open windows for the rbtree stuff, forward porting buildroot's nommu arm qemu image from the 4.4 to 4.5 kernels hits the switch to device tree after which there's no console output whatsoever and I haven't figured out why yet...

Currently poking at putting the mkroot script in toybox, and as I'm converting it I realized... there's no need for the airlock script? All I'm compiling is 1) toybox itself (which I already had to be able to compile under the host to _create_ the airlock directory), 2) the Linux kernel. There are no other packages being built, and I'm removing the "modules" stuff that adds extras, hence no real need for the airlock?

I still want to build a kernel under the resulting system, but I need to add a "make" command first, and an awk, both of which are kind of a big ask. But getting a system to boot to a shell prompt should be as simple as possible, and adding native development tools to the result should be a tarball extract or similar. ("Or similar" because if the base system is in initramfs, the development tools may be bigger than the filesystem can hold, hence the symlink script aboriginal had to do that from the squashfs.)

I need to make an archiver that can create/extract squashfs like tar or zip files...

The System76 machine arrived. Nope.

Oh well, I tried.

Got another email about j-core from somebody wanting to participate in its development. I sent back the first paragraph of this:

Unfortunately I haven't had access to the j-core.org or lists.j-core.org domains for months. Ever since the servers moved to cloud hosting my ssh key didn't transfer over, so I can't update the website or fix the mailing list. I also lost personal access to the "not cleared for public release" code at the start of the year, so haven't been able to do any development on that stuff either. (Or track the development being done in japan for proprietary uses.)

I did _not_ send back these paragraphs, which I typed and then cut out:

I arranged weekly calls with the developers for a couple months, and brought these issues up each time, but and each time, but nothing ever changed. Last week I _didn't_ arrange one to see if the project maintainer would notice its absence, and he didn't.

The problem is I don't work for that company anymore, and although I was still trying to participate in the open source side of things there doesn't really seem to be one when it doesn't suit the company's proprietary interests, which it hasn't recently. Maybe that will change in future, but so far anyone not working at the same company as the other developers has been second class citizens, so at best we're looking at something like Android or Sun's OpenOffice or Mozilla under Netscape before Jamie Zawinski resigned.

The classic failure mode of that kind of read-only project is that there's no point in outsiders submitting even bug reports upstream because the version they're using is a year newer than anything you have access to, and the design and development conversations all go through privileged insider-only channels you can't even read, let alone contribute to.

*Shrug* Jeff doesn't see it that way, you can always try asking him. I think [email redacted] still works.

Probably what somebody needs to do is take the last open-source tarball, check it into github, and do an open fork as a real project just ignoring what sei propreitary does. But the chance the private one _might_ release another drop has so far overshadowed the public one enough that nobody's seriously tried to bang on the "stale" version that's out there.

May 13, 2018

May 10, 2018

May 8, 2018

May 7, 2018

May 6, 2018

May 5, 2018

May 2, 2018

May 1, 2018

April 28, 2018

April 17, 2018

April 16, 2018

April 15, 2018

April 14, 2018

April 13, 2018

April 11, 2018

April 9, 2018

April 8, 2018

April 7, 2018