Met mark around lunchtime and got the bank account for the new company open. Now to figure out whether to try to launch it slowly part-time over the next couple years while working a day job (like Mark's doing), or jump in with both feet. The second's probably more likely to work, but as of yet I have no actual customers for it...
Ok, the new patch is in, and that should be a toybox 0.0.4 release. I'm test-building Firmware Linux with it to make sure everything else is happy. (I think I already fixed touch... Yup, checkin 218.) I have one more todo item for the toybox release (the readlink -f stuff Roberto Foglietta's been sending me) but that should be out this evening.
The two other projects I might syncopate a bit, because although scheduled releases are great, having releases of all three of my projects at the same time could get a little inconvenient going forward. And in any case, I need to do a little more banging on both firmware and tinycc.
Heard from the Google recruiter today, who says they might be starting a new branch in Austin. I've heard this rumor before, but it would be nice if it became true someday. The main reason I haven't gone to work for Google is they keep wanting me to move to California. (The job the recruiter contacted me about is, once again, in california. And they want somebody to manage kernel developers, which I could do but which combines two things I've only flirted with before. (Management, and kernel development.) I've done both, but they tend to grab your attention to the exclusion of all else, and I've preferred to remain a generalist. I suppose as I get older, trending towards management is sort of inevitable...
Switching back to a day schedule, mostly offline.
The downside of netflix is the frequency with which you get damaged DVDs, and have to stop watching stuff halfway through until they send you a new disk three days later. (Or, in the case of the first pressing of "The Christmas Invasion", until you get your own box set as a wedding present, three months after they yanked the entire production run but have yet to re-issue it or even comment on when it _might_ become available.)
This is second "House" DVD we've had to stop watching halfway through an episode due to a scratched up disk. They're very nice about replacing them, but it still won't be here before Tuesday at the earliest, and the episode had just gotten to a good bit. Altogether, we've gotten at least a half dozen unwatchable DVDs over the past year, which we've had to send back.
Yet another corner case in "patch". I'd assumed there would always be at least one non-patch line between each patch, so what I was doing was naievely reading each hunk until the first line that didn't start with a "+- " char, and then trimming any extra context lines (more than the starting context line).
This was _really_stupid_, and I honestly don't know what I was thinking. It was small, simple, and wrong. Here's how that breaks when you cat multiple patch files together:
--- filename +++ filename @@ -42,7 +42,7 @@ context context context -line +line context context context --- filename +++ filename ...
The length of the patch is given in the @@ line at the top: the -xx,7 means 7 lines of old content, the +xx,7 means 7 lines of new content. Use that and concatenating patches is no problem (and context line trimming isn't necessary either, so it actually probably winds up smaller.)
My code's even already parsing @@ lines, although I just through it would just be for error reporting purposes. The -42 and +42 are the starting line at which to expect the start of the hunk (in the old and new file, respectively), which is only really useful to tell you at what offset you did find the hunk. I even thought about searching for multiple patch locations and taking the closest one, but traditional patch just locks to the first three matching lines of context and fails the hunk if the rest of the patch doesn't match. (Unless of course you implement "fuzz factor" support, which is often a bad idea.)
There is a corner case where repeated context lines could make it miss a patch location and fail to apply. If the first three context lines are "aab" and the file has "aaab", then matching the first two and failing the third would flush those three already-evaluated lines from the previous supposed match to the output, and they wouldn't be re-scanned for another match later down. This is an "I can fix that, but is it worth enlarging/complicating the algorithm"?
I could make the thing _darn_ reliable (and fuzz factor implementation trivial) by loading the whole file into memory and running each hunk against the in-ram copy. It could apply overlapping or out-of-order hunks, use the closest match in case of duplicate match sites... But it would also eat much more memory at runtime, and I'm trying to do embedded software...
Maybe I could do something with mmap(MAP_PRIVATE)? Eh, worry about it later, get this implementation working first. And _that_ means a rewrite of the hunk reading code.
Hanging out at the Starbucks behind the arboretum. Today they have a woman playing a harp. She's playing "tea for two" on said harp.
Refactoring tinycc. Yeah, I know I should get a release out first but I really can't make much more changes to the code until I clean it up to the point where I understand what the heck it's _doing_. The current organization is haphazard, undocumented, and full of cross-dependencies in the strangest places, uses way too many single character variable names, buckets of global variables with names like "ind", "loc", and "vtop", and instead of breaking up the code into multiple files that could be separately compiled, it #includes other .c files which can't build on their own because they depend on static functions and static global variables from the #including files. Yeah, cleanup time.
Currently changing the names of the functions in i386/gen.c so they all start with a gen_ prefix. (Some already did, some started with just the letter g, and a few had neither.) This is a finite and hopefully somewhat self-contained change, so I know when I'm done. Trying to figure out along the way what each function does (and comment it as I go) which is a bit nonobvious in places. Take gsym() and gsym_addr() for example (now gen_sym() and gen_sym_addr()). The first is a wrapper calling the second with a global variable for its second argument. The second takes two arguments, both integers, but what are they? The first indicates a symbol somehow, and the second is an address, but in what format? They're both integers, should they be longs on a 64-bit host?
Looking at the calls to the function, tcc.c has block() which has an if statement for TOK_WHILE() that ends with a call to gsym_addr(). Of course the variables used in this area of the code are a, b, and d with no further explanation and no comments.
Ok, it looks like what these are doing is _linking_ a symbol. I note that gsym_addr() is probably going to be really unhappy running on a host that traps on access to unaligned integer pointers, but for the moment I'll add a comment and move on. What the heck does the comment "output a symbol" mean in this context? What actually gets output? They're patching addresses that have already been output.
The harp lady is now playing an Evanescence song. The "these wounds won't seem to to heal" one. And singling along to it. Now she's doing a classical piece I recognize but couldn't name. Beethoven, possibly. (I know Trans siberian orchestra covered it, and it's often played around christmas.) Next a Roxette song, "listen to your heart". This is actually a better selection than I expected from harp music.
I wonder what oad() stands for. Output with additional data, perhaps? What it _does_ is gen_inst_le32() so that's what I'll call it, I guess...
Right, renaming pass first, _then_ try to clean up the code. (Gotta do them seperately so when I screw stuff up I have a chance of tracking down _which_ change broke things, and debugging it.)
And patch broke in the uClibc build, because if a hunk is right up against the end of the file there aren't as many trailing context lines as starting context lines. (There may not, in fact, be any.) So fewer ending than starting context lines means match end of file. The start of file case is sort of already handled by the current code; with zero context lines we match immediatey (start of file unless it's not the first hunk, which is a "don't go there" case). Trimming trailing context lines shouldn't hurt too much because the start of the file is easy to find reliably.
Finally collating my todo lists to produce a master todo list. Not _close_ to complete yet, of course. :)
At Epoch I learned that I'm not the first person to do a parallel bzip2 implementation. It's even in Ubuntu's "universe" repository, I just never noticed before. Can't say I'm surprised, it is an obviously parallelizable algorithm (Independent 900k chunks. Ok, the only way I can see they're parallelizing decompression beyond 2-3 threads is via an unreliable heuristic. I haven't looked at the code yet, but I'm just guessing that they're searching ahead in the data for frame headers, hoping there are no false positives in the actual compressed data. I wonder how they do handle false positives?)
Then again, toybox isn't exactly trying to break new ground, is it? Neither was busybox. The point is I can do a _better_ job, or at least a smaller and simpler one. :)
Implemented the rest of patch add/remove in toybox. Comment from 60 seconds ago, "That may actually have worked". Firmware Linux seems to be building with it. Fingers crossed... I still need to implement "move" support, ala git. And I need to go re-read Brahm Cohen's blog (the creator of bittorrent) to see what his new diff algorithm was all about. But hey... Progress.
Broke the option parsing logic of tinycc into a separate file. I know I shouldn't do cleanups until after this release, but the urge to wield a shovel at it is enormous. I have Jack Shang's macro parsing fix and am trying to figure out if it's the correct fix (it works, but the author himself isn't sure it's the right way to fix it), and backing up and tracing through the context of what the token parsing logic is doing, the urge to refactor and clean up is HUGE.
Heh. Now that I'm using the toybox commands during the build, FWL hit the problem that "touch" isn't creating files. I knew about that one from the test suite already, time to fix it...
Merry christmas.
Banging on tinycc a bit, on and off. Ok, mostly off doing other things. (Fade and I spent 2 hours driving around town trying to find an open restaurant for lunch before discovering that the IHOP within walking distance of home was open. The only other candidate was a Furr's in south austin with a line stretching outside the building. The cheesecake pancakes are REALLY GOOD, especially with strawberry syrup.)
I learned from Wikipedia that Sandman:The Dream Hunters was actually based on Pu Songling's Strange Stories From A Chinese Studio, which I thought I ought to read. Will report back.
- Neil Gaiman, author of Sandman: The Dream Hunters
So, after much head-scratching trying to figure out what I screwed up in the bunzip refactoring, I find out that the version I'd added to toybox never worked with the testcase I'm currently trying (a semi-current stable Linux kernel tarball). So I've backed up to my old micro-bunzip.c and I'm debugging against that, but it's reading data in different sized chunks in places which screws up the trivial divergence finding technique of "do a printf of each read starting position and bit length, and diff the output".
On the bright side, all this bashing through two slightly different implementations and head scratching over "what does this bit do?" is refamiliarizing me with intricate subtle code I haven't really looked at closely since I wrote it in 2003. I should write up an explanation of how the darn bunzip algorithm works and send it to the lwn guys or something.
The bunzip problem turned out to be missing curly brackets.
Huh, I appear to be allergic to parsnips.
Fade and I went to see Sweeney Todd at the Alamo Drafthouse today, and of course when the Drafthouse is showing a musical where the protagonists murder people and make meat pies out of the bodies, the drafthouse is going to serve meat pies. I had the rabbit and Fade had the duck. Halfway through the movie I started to get a weird reaction I haven't had for a few years, my neck aching and feeling swollen, and like I'm not getting enough blood to my brain (tired, unable to focus, stuffed up ears). The last time I seriously felt this way I traced it to some only vaguely identified vegetables. This time, the drafthouse's ingredients list for the rabbit pie mentions parsnips, something I don't normally have.
I was a bit out of it for my meeting with Will (spent half of it trying to poke the right side of my neck to unblock it), and then took a 2 hour nap. Then had LOTS OF TEA. Feeling better now. (Tea. Marvelous stuff. Diuretic, among other things. Plus if adrenaline is a treatment for allergic reactions that close up your lungs, I can see caffiene helping ones that close up the blood vessels in your neck.)
Once again bit by vi's most annoying and useless feature: the tendency to accidentally change the case of random text when you typo something (in this case, a dozen screens worth of text), and not notice. That was a totally unnecessary half-hour of cleanup. ("Undo" doesn't help if you don't spot the lower-casing until you've done an hour or two of editing on other parts of the file.)
Got the first major refactoring in my threaded rewrite of bunzip2 to the point where it compiles again. (And segfaults, but that's to be expected.) I'd estimate that once I get this debugged and checked in, that'll be somewhere around 1/2 what I need to do to get a 2-thread version of bunzip2 implemented.
The altorithm makes two major passes over the data. The first reads from the source file, and does huffman, run-length, and move-to-front encoding. It has to do all three to figure out where the end of each block is, because the blocks are 900k before compression (and should thus decompress to an even 900k), but the only way to find out how much _compressed_ data is in the block is to decompress it. The second pass undoes the burrows-wheeler transform and calculates the CRCs. The obvious parallelism is to split those two passes and have one reading into a buffer while the other writes the previous buffer, and then swap. If the huffman decode turned out to be twice as fast as the burrows-wheeler step (and given the speed ratio between gzip and bzip that's possible), then two burrows wheeler steps could easily happen in parallel.
For right now, I'm just doing two (one read thread, one write thread) to simplify things. And I'm making it so the same code still works uniprocessor with config switch.
Watched "The Face of Evil" on youtube. Off to feed Mark's cats.
The first tenth doctor DVD finaly arrived through netflix, and we're watching the commentary track of "the christmas invasion", and it's "video commentary", meaning there's a darn pop-up window (showing three people with party hats huddled around amicrophone) blocking 1/4 of the picture. I keep wanting to grab it and drag it out of frame, but alas it doesn't work that way...
Mark informed me that multicast is still used by the "internet 2" people (both of them, apparently) to do videoconferencing.
Yay, Mark got a job!
Remembered that the Wii acts as a youtube video viewer, which was finally enough incentive to fire it up. Watching "The Curse of Peladon" (third doctor). Alas, when I fullscreen the video, A) the frame rate drops to about 5 updates/second, B) it doesn't go quite fullscreen, there's still a large wii icon bar across the bottom I haven't figured out how to dismiss, yielding black bars along the sides of the video to maintain the aspect ratio. So it's using just over half the screen, and can't even update that much at full speed. Sigh.
I remember when people used to care about "multicast traffic" and the m-bone and such; a way of sending one packet to multiple recipients. For the record, that stuff's dead. Youtube doesn't use any of that, and neither did Napster before it. Instead they send a unique copy of the data to each user, via a standard TCP/IP connection to a web server. This is called "unicast", although the name doesn't get used much for the same reason the names for Earth's moon and sun (Luna and Sol, respectively) don't get used much.
These days you download video the same way you download any other web content, there's a file at a URL and you download the whole thing from beginning to end. Even Youtube is doing this behind the scenes, if you want to dig through a bit of Javascript to find the URL. (Things like bittorrent are more efficient from a pure downloading perspective, but since they download data out of order you can't stream it and view while downloading.)
The failure of multicast was 100% predictable to those of us working in this area circa 2001. We got some details wrong; for example those of us who thought cable TV companies might be less stupid expected more regional cacheing. Although Youtube/Google have multiple data centers full of rackmounted PCs serving data into big fiber optic connections, they only have a dozen or so enormous data centers around the world instead of a small one in each city. But that's details.
The reason multicast traffic died was essentially due to improvements in data compression technology. The first modern multimedia distribution system was Napster, and Napster's trick was the mp3 compession format. Modern modern multimedia-specific data compression formats can crunch multimedia stuff down to about 1/10th of the size of the data compression formats that came before it. So a 650 megabyte CD became around 50 megabytes of MP3s, making individual songs (3-4 megabytes) feasible to download circa 1998 via standard TCP/IP connections, even over dialup. Instead of fancy downloading technology, better data compression made the multimedia files an order of magnitude smaller.
Of course data compression can't handle missing data, and broadcast techniques drop packets. The original purpose of things like RealAudio was to do without the data that didn't make it through, reconstruct the missing bits as best as possible, and gracefully degrade the output. They put huge amounts of effort into making this work. But with better data compression, sending 1/10th as much data produced better output, as long as it was sent reliably.
The DRM people loved multicast, of course. A normal file that lives at a URL can be downloaded via right clicking on the link your browser and going "save as", and there's really nothing the server can do to stop this. Viewing and downloading are the same action. With multicast the data is sent to special applications that do nothing but display it, and they probably don't get a complete copy.
The final nails in the coffin were that A) you need special servers to send multicast data, while any web server will do for unicast, B) podcasting: separating the download from the viewing means any old program can view the data, and you can take it with you on portable devices that aren't even connected to the net.
The only reason this is worth commenting on is the sheer amount of _effort_ that went into multicast. The are internet RFCs, the Linux kernel has a multicast stack, RealAudio built its original business model on it (and of course Microsoft blindly copied them). And within 10 years, it dried up and blew away because it's just not how the internet works.
Met Mark for lunch at a strip club. The $2 steak and fries were good (and I don't feel bad about taking advantag of them after buying a $4.50 diet coke). We filed the LLC paperwork for Impact, and then my driver's side car window broke again, so I took it to Lamb's. They have to replace the regulatory thingy they put in back in May. Part's still under warantee but labor was only guaranteed for 6 months. Oh well.
One more swing at patch (I keep getting distracted from it). Got add and remove implemented, although not debugged yet. I need to make it understand the git "move" syntax too, because renaming files is one of those things people actually do, and "delete whole file, insert whole file" is a _horrible_ way to represent that.
I need to work on the Impact website. I need to put together a current TODO list, considering the last five I have floating around are all highly incomplete and full of stale entries, and I'm not quite sure where they all are anyway. (Yes "collate todo lists" is a perennial todo item of mine.)
Heard back from the author of "Lipitor, thief of memory" who agrees Terry Pratchett might be suffering from the problem his book's about, and asked said author to email Terry directly. (Well Terry said, "I know it's a very human thing to say "Is there anything I can do", but in this case I would only entertain offers from very high-end experts in brain chemistry." This guy at least has an MD and wrote a book about this specific problem, which is more than I can say. :)
The Tin Can Tools guys are prototyping a USB-powered hammer board based on some suggestions of mine on #edef on freenode. They're calling it "nail", of course.
Still banging on patch. Less Dr. Who watching today, but the night is young, as they say. (Prenatal, even. The sun is still up.)
Expanded the architecture section of kernel.org/doc, and slightly tweaked and htmlized my gplv2 compliance howto (which isn't official until the SFLC lawyers sign off on it).
I seem to have inspired a thing. Not entirely sure how, but I look forward to seeing it in action.
Incorporating Impact's going to cost $300 to do an LLC. Sigh. Oh well, needs to be done...
Watched "Tomb of the cybermen" (an old Patrick Troughton Dr. Who episode) on youtube. Wow did it suck. I suppose it's a bit like complaining that "the great train robbery" didn't have much of a plot by modern standards. The three main characters (The Doctor, Jamie, and to an extent Victora although she had nothing to work with) could all act, but none of the guest stars could survive in a modern commercial, let alone series. (I don't mind the horrible special effects, but the _acting_ has to be at least half decent.) Still, "The gunfighters" remains unassailble as the worst Dr. Who episode of all time.
Banging on patch.c. Got it working... except for inserting new files, which breaks a couple of rules. (Patching a file that doesn't exist, zero context lines.) Deleting an entire file is another unimplemented corner case, again zero context lines and deleting the resulting file. Also, right now I'm using the +++ line as the filename, which works for everything except comparing a file with /dev/null to delete it. Sigh.
Fade pointed me at this marvelous review of Windows XP as an upgrade from Vista, covering Look & Feel, Performance, Device Support, Reliability, Gaming, and Multimedia:
Conclusion
To be honest there is only one conclusion to be made; Microsoft have really outdone themselves in delivering a brand new operating system that really excels in all the areas where Vista was sub-optimal. From my testing, discussions with friends and colleagues, and a review of the material out there on the web there seems to be no doubt whatsoever that that upgrade to XP is well worth the money. Microsoft can really pat themselves on the back for a job well done, delivering an operating system which is much faster and far more reliable than its predecessor. Anyone who thinks there are problems in the Microsoft Windows team need only point to this fantastic release and scoff loudly.
Well done Microsoft!
So I'm working on building toybox commands as individual files, like I did for busybox way back when. Unfortunately, this fights with the config system. Do I say "scripts/individual.sh hello" to build one command, or do I have the user switch on what they want with menuconfig and then autodetect that and build the enabled features? I'd lean towards the first (and come up with a way to say "make all_individual") except that commands have sub-features, and you need menuconfig to specify which sub-features you want.
However I do it, dealing with toys/toylist.h turns out to be a pain. Just trying to do this by hand (come up with an individual.c and gcc invocation to build hello.c) The second half of that file has been extensively designed to do something very specific: creating lists of information about all the commands in toybox. It's quite well-suited to its original purpose. What it's not well-suited for is cherry picking an individual entry from those lists without instantiating the whole list first.
For the individual command case, I want to have toy_list[] contain just one entry, and set toys=toy_list, and drop out the rest of the list. But the initialization of that list is controlled by USE_BLAH() macros, which come from the config system. I need to switch off all the USE_ macros except the ones for the current command, and within the C code I don't even know what they all are. Distinguishing that requires a sed preprocessor step.
Possibly I could do a big stack of ? : operators the way I'm doing a big stack of || operators, so that toy_list is just the one entry I need. I could even skip the first NEWTOY() declaration (for the toybox command, which has no USE_ guard) by doing 0 && in front of the whole lot (the precedence between that and ? : works out ok in this instance.) Except for two problems: 1) thing={blah,blah,blah} syntax is only accepted in an initializer during a declaration, so sticking (1 ? {blah} : {blah}) around it may not work B) what would I test against? Making a preprocessor symbol that's 1 for this command is easy enugh, but how do I get zero symbols for all the other ones?
I could break down and just run sed against the header file to chop out the declaration I need, but part of the reason I'm doing this is that "sed" doesn't work the same way on various different platforms. I still don't care about Cygwin but I do care about MacOS X. So adding my sed implementation to toybox and building it as a separate executable without having to go through the config system (which uses sed) at the start of the build makes sense. But if the scripts/individual build needs sed then I'm in chicken-and-egg land...
Got the first half of patch checked into toybox. The patch output is going to stdout, but it seems to be applying hunks properly now. With an offset even, albeit no fuzz factor yet.
I need to think more about the offset logic. Right now it doesn't care where the hunks say they are (in terms of starting line number), it just tries to apply them all in order based on context lines. This is probably good enough for most real world uses, although the downside is that a single failed hunk means all the other hunks in the patch fail too. In order to improve on it I'd probably want to read in all the hunks for a file and try to apply them in parallel. Worry about it later...
Reviewed Roberto Foglietta's readlink patch adding [-fem] (alas, incorrectly due to the gnu behavior being complicated and persnickety in ways that are not at all obvious upon casual inspection). In the process I noticed that MacOS X has a "readlink" command that essentially has no common options with the gnu version. (It shares -n, but that's it. Their -f implements totally different functionality, and there is no -e or -m.) So the question of whether or not this is worth doing at all comes up. I still lean towards adding -f.
Got the high-gain antennas source at least minimally reviewed. It's stock BusyBox 1.01 with three lines of tweaks, some bundled config files, and they didn't run "make distclean" since the last time they built it. Other than that, it's stock 1.01. Whether or not that corresponds to the binaries they've been distributing, I couldn't tell you. I have no idea how their firmware images are packaged, and I haven't got one of the devices to install it on and take a look. The file command says "data" when I point it at the firmware images. Wheee. I know a number of people who slap a jtag on these sort of devices and reverse engineer the hardware for fun, and they could almost certainly extract the appropriate binaries... if I had the hardware.
Yay, new laptop charger. Dell replaced it under warantee, and got it to me next day via DHL. I'm impressed. (I didn't even buy the extra service plan, this was straight warantee coverage. There are some advantages to buying new hardware instead of used...)
Went to CompUSA's going out of business sale during my day and half offline. Most of the stuff they have is still more expensive than Fry's even with the extra 10-20% off, but I bought Fade a copy of the Mac VMWare for christmas. (And gave it to her, since she already gave me my PSP and she might as well start using it now.)
I really need to learn to use branches in mercurial. Roberto Foglietta sent me a readlink patch I want to apply, but it hits some of the files I've already changed for patch. But patch isn't ready to go in yet...
Jon Stewart continues to be amusing, even on cspan.
Oh come on, Terry Pratchett gets diagnosed with Alzheimer's right after being put on cholesterol lowering Statin drugs which are known to screw up memory. And nobody makes the connection?
My laptop charger has stopped charging. I have no idea why. Great. I've plugged it into three outlets, some of which are powering lights, and it won't light up. Not good...
My laptop finally failed to suspend again last night. I let it get to 2% battery before hibernating, and after grinding away for five minutes trying to suspend the battery gave out. Software suspend gets really really slow when you've done it several dozen times without rebooting. (Mostly the "free as much memory by swapping it out" phase, the "write out what's left" phase is still relatively fast. Fragmented swap file, I'd guess.
So I lost six desktops crammed full of todo windows. Let's see, what are my current pending projects:
And that's just the immediately pending stuff. I've actually been swap-thrashing about as much as my laptop has, trying to get all these done at once. The bottom three have kind of gotten knocked down until I get releases out of each of the top three. And once I get the _bottom_ three done, then I need to seriously job hunt.
Implementing patch for toybox, I am learning _so_ much about the weird corner cases of diff and patch. (This is a fairly common occurrence, I learned tons about mount implementing it, learned sed by implementing it, and so on. For one thing, unified diffs can't have hunks out of order. (If they do, the gnu version of patch fails with a specific error message for this. If you reorder and then renumber them, the first one applies with an offset and the second fails.) So it's doing a simple one pass substitution.
Two cute blonde women stopped me and spent half an hour talking with me. Sadly, this does not fall in the "good things" column for the day.
Remember when I visited Austin back in May and got the inspection updated? May 2007, so the inspection would expire 05/08? It turns out that Lamb's on Far West put the "8" in the month field and the "5" in the year field, so it says it expired in August 2005. Yeah, I got pulled over. Plus the glove compartment was empty (there was a huge pile of papers in there, where did it all _go_?) so there's no insurance paperwork to give them. (We make the car extortion payment, really.) And the registration sticker on the car expired last month, although their computer did note that the registration was updated (just the new sticker they mailed to us isn't on the car yet).
The whole thing becomes about a $10 fine once I cough up the appropriate paperwork, but the sad part is I did all the right steps to make my car legal. (I wonder if Lamb's has inspection records. I _did_ get it inspected, when I got the driver's side window and the air conditioning fixed. The inspection sticker I had before that nominally expired _after_ the one currently on my car.)
Ok, this guy has more time on his hands than I do. (The lights are apparently running on one of those hammer boards. The music is Trans Siberian Orchestra.)
Why is the GPL like Star Trek? The even numbered ones are better. Why is the GPL like Star Wars? The new version 15 years later just doesn't measure up to the old stuff.
An entire swat team (eight or nine people in green army camoflage fatigues with grey t-shirts over them that say "Austin SWAT") are having lunch at Chick-fil-a. I wonder why?
Oh wow, the assembly produced by tinycc is horrible. There's an "fldz" instruction and yet it's doing "fldl 0x0"... Ok, after this release, there's some serious cleanup to do. But for now, I've ignored Firmware Linux for too long. The wrapper needs to be updated so the x86-64 toolchains are less brittle.
*blink* *blink* *blink*... And I broke patch. The gnu one. I have no idea how I did this, but there's a file that it's _insisting_ contains a reversed patch, yet it doesn't. (The 2.6.13 patch to the linux kernel source to make "init=/tools/bin/init.sh" not break.) It's removing a hunk of code from init/main.c, and said hunk of code is in init/main.c. If I let it go "assume -r" like it prompts, it adds a second copy of said hunk of code. Yes, the patch has minus signs. (Minus signs remove code. Plus signs add code.) There's no -r being passed on the patch command line. I checked the starting line number, it's only two lines off. What the...?
I think it was the upgrade to 7.10 that did it, actually. Neither my scripts nor my patches changed. (I upgraded the kernel tarball to 2.6.24-rc4 but that's just standard preemptive stuff, there's no obvious
Right, screw it. Time to implement patch in toybox. This is just _silly_.
Hmmm... Toybox only really has wrappers for file descriptor functions, not FILE * functions. I hate to have two parallel sets of infrastructure that do the same thing in a project designed around being small and simple. The file descriptor functions are inherently more simple: it's what the OS is doing under the covers anyway, there's no need to worry about an internal buffer not being flushed... Unfortunately, doing getline() without a pushback buffer boils down to single byte reads, which are painfully inefficient. (If you overshoot, you either have to maintain your own buffer of the extra data you read, or hope that your filehandle is seekable, which pipes like stdin aren't.)
Finally made expressions like 'long l = ("blah" && 0)' resolve properly at compile time as a contant in tinycc. Now I've gotten distracted into installing wine and spending five minutes poking at it to see if ./win32-tinycc is actually doing anything interesting. (Answer: no, it's building x86 Linux binaries. Apparently the only way to tell tcc to build Windows output was to compile it on a Windows host.)
I've ranted here before about HOST != TARGET, and in my build system rewrite I've confined all the weird "you can you run code for an x86 target on an x86-64 host but the library paths are /lib32 instead of /lib" stuff to ./configure and not in the C code. But from a purely pragmatic viewpont, I can't test the win32 backend unless I use Linux to do it. I haven't got a Windows machine. With two brief exceptions (the time it took me to complete "starsiege" and about three months keeping around a World of Warcraft partition), I've never had a windows machine. I went from DOS to OS/2 to Linux with the occasional ricochet off of Solaris, AIX, JavaOS, etc. The closest I've come to supporting Windows was writing code in Java or Python that could be deployed on it, or changing the output of my CGI scripts to work around bugs in Internet Explorer.
I've now inherited a project that has a win32 backend, and it seems a shame to rip it out, but I simply don't do Windows. So I'm playing with Wine...
tinycc 0.9.25-pre1 is out. It only builds on i386 at the moment, but that one's building itself from source and passing its test suite, so it's time for a snapshot. Real release still at the end of the month. I need to get at least arm working, possibly win32 as well. (I'll care about c67 when I find a user and an emulator.)
Added the test suite back to tinycc, using the new build infrastructure. This is just the first pass, the second pass is to automate building tinycc with itself and see what the output of _that_ is. But the first pass seems to have three separate instances of breakage, according to the diff. Wheee.
Right now, you need to "ln -s i386-tinycc tinycc" in order to run make/test.sh because it tries to use the "native compiler" symlink, and the build isn't making that yet. This is because it's not trying to automatically determine what host you're running on. This sort of belongs in ./configure, but is just complicated enough I have the urge to put it somewhere else. (Mostly configure is "if environment variable X isn't set, set it to a default value". Adding in actual logic that _does_ stuff goes against the grain, because I want people to feel free to edit configure to whatever they need, or set all its variables themselves as environment variables. Alas, I need some logic for certain things, and this is one of them. The abnormal library paths for x86 binaries on an x86-64 host are another...
Anyway, once I get the known bugs worked out and have it in a working state I can regression test, then it's time to tackle the 8 gazillion warnings it produces while compiling...
Fixed the octal parsing logic, which never quite worked after I consolidated it last month but I couldn't test until now. (And _this_ is why I've been holding off fixing up the warnings: even simple changes break things. How people ever wrote programs before they had their own machine to actually test it on is one of those great heroic endeavours of history like the people who recited the entire Illiad from memory. (Of course they could spend months learning one story, and make a career out of a dozen or so, and these days the rest of us have a bookshelf full of 300 of the suckers.)
Lots of things are broken on x86-64, but most of these are actually due to x86 not really being a native compiler. You need a native compiler for -run to work, you need a native compiler to run the test suite. (You can compare the output of a 64-bit build of the test binary with the output of a 32-bit build, but anything that uses sizeof() or align() is going to be different. I fed CFLAGS="-m32" into test/test.sh, but either I'm doing it wrong or gcc on Ubuntu doesn't want to produce a 32-bit binary. Possibly I need to install something. Luckily, I have a 32-bit xubuntu installed in an image file under qemu, which makes testing so much easier.)
Another fun thing that makes testing easier is the following script:
(cd toybox/toybox && hg serve -d -p 8000 -a 127.0.0.1) (cd firmware/firmware && hg serve -d -p 8001 -a 127.0.0.1) (cd tinycc/tinycc && hg serve -d -p 8002 -a 127.0.0.1)
Not only does this let me fire up a web view of my projects when I'm not online (via http://127.0.0.1:8002 and such) but it lets me check them out from qemu ala "hg clone http://10.0.2.2:8002 tinycc" and then "hg pull -u" to update them. It turns out that 10.0.2.2 in qemu is an alias for the host system's loopback. (This is also what makes the distcc trick work in the FWL build, I need to finish that and check it in...)
Hanging out at Subway today, for a change of pace. No outlets or net access here, but free soda refills and 3 hours of battery life in ye newe laptop. I'm loving it. (No, McDonalds, you can't own common phrases out of the language. And this applies to you too Nike, with the "just do it" thing, which was a line Ford Prefect to Arthur Dent in the BBC hitchhiker's guide to the galaxy in 1980, among other things. It was even vaguely shoe-related, telling Arthur to prepare for hyperspace by lying down and grabbing a towel between his ankles...)
While walking Mark through the toybox code yesterday I noticed that checkin 186 is broken. The default toys.exitval is 1, not 0, because this is what error_exit() returns. This means that half the commands (ironically enough including "true") are exiting with 1 instead of 0 on success. Which is broken.
The question is how to fix it. I'd rather not patch each command (and add back the 4 bytes to each one), but instead change the default value and have error_exit() and friends check for 0. This means that error_exit() can't print an error message but then exit(0). I think I'm ok with this, since I can't come up with an actual use case that would need to do this, but I've been pondering it a bit before making the change...
I'm watching Mark export his iPhone's cellular network connection to his macbook. That's... both sick and amazingly cool. Support for all this is apparently built in. [Correction: Mark informs me it's not built in, it's software he installed. Considering he's got a shell prompt on his iPhone, I should have guessed this.]
Hmmm... Granularity problem in the tinycc configuration. The logic in tinycc itself is adding "/lib" and "/include" to a base path ala "/lib/tinycc". But the ./configure script isn't passing in "/lib/tinycc", it's passing in separate "/lib/tinycc/lib" and "/lib/tinycc/include" paths.
On the one hand, having just one install location is simple, and simple is good. But explicitly specifying each path both provides more control and more visibility. It's an aesthetic decision as much as anything, both ways work fine but what should the design be? These are always the tough ones...
Ideally I'd just have ./configure prepend these paths to the library and header search paths it's already assembling, but unfortunately the Linux kernel build wants to know where the compiler headers are, but to supply its own headers for everything else. So they have to stay separate. Which means the fully orthogonal separate paths aren't fully orthogonal if we want to build an unmodified Linux kernel, so I might as well take advantage of "simple", above.
Ok, that was a fun couple hours of debugging.
Trying to make a new test script for tcc to build and run the test suite, I noticed it was trying to link against "tinycc-1" rather than "tinycc-i386". The i386 comes from TINYCC_TARGET which is initialized from "$1" in make.sh, and that seemed a natural place for a 1 to be coming from. But it wasn't, the macro was set properly, and the thing built just fine, but with the wrong value. Huh?
Fast forward two hours later, after a dozen debugging blind alleys (and re-learning that #warning doesn't evaluate macros). It finally hit me: TINYCC_TARGET was evaluating to i386, and i386 is itself a macro in gcc that evaluates to "1" on i386. It's one of the compiler builtins.
This didn't come up when it was quoted. Sigh. And this is likely to be a problem on _every_ platform. Even though there's just the one user of this, I can't pass it in as "tinycc-i386" either, because unless I put quotes around that the i386 is matched and turned into a 1. Sigh. Can't assemble it out of substrings or pass it in as part of a bigger string, I _must_ put quotes around it if I want "i386" to wind up in the result. And putting it in a subdirectory named "i386" has the same problem.
I note that I always test for __i386__ and friends, because #defining i386 or arm is just too dangerous. But the gcc guys do it anyway. I note that this built just fine on an x86-64 host. :)
Although the Schlotzky's at UT has free wireless, it's no substitute for Chick-fil-a. Among other things, its sodas taste very strongly of chlorine.
I've heard a half-dozen variants of speculating on what, exactly, the reindeer games Rudolph was excluded from consisted of. I am aware that a preoposition is considered a poor word to end a sentence with. These days we're allowed to boldly split infinitives that nobody using "them" as a gender neutral singular pronoun has split before, too. Isn't it great not to have a language institute? The lingua franca is English, in large part thanks to L'academie Francaise pruning French back until there's nothing left. Although laws mandating a language's continued use do help to drive home how deeply popular it must be among its native speakers, to need a squirrel cage like that in order to survive. (Welsh is having an actual popular revival, possibly pushing it past Klingon. Good for them. I wonder how many new words English will steal from it?)
Huh, "./i386-tinycc -run hello.c" is segfaulting on x86-64. Works fine on x86. I'm guessing this is a 32-bit vs 64-bit thing. I used to think it wrote a file in /tmp and execed it, but no, mallocs a big buffer to hold the executable and calls it as a function pointer. Cool, but not gonna work when host!=target no matter what fancy BINFMT_MISC configuration you've set up. Even running x86 code on x86-64 requires an exec to switch modes.
Spent several hours in the evening trying to get wordpress working for fade's new blog. Eventually asked Mark for help. I spent two hours making no progress (fighting with MySQL administration), he fixed it in about six minutes, including installing the rest of wordpress.
Apparently it's possible to switch VT support off in the bios, meaning a core 2 duo would claim not to have the extensions KVM needs, which could explain why kvm was degrading to normal (slow) qemu. What I don't know is whether suspending to disk, switching this on, and resuming from disk would make the kernel suddenly demand prozac.
The downside of having a laptop with 2 gigs of ram and 4 gigs allocated to swap, now that I've figured out now to close the lid and unplug it from the wall until the suspend has started actually writing to disk, is that I haven't actually rebooted in weeks. I have six desktops full of windows. I have windows open for kernel documentation, firmware linux, toybox, tinycc, the 64-bit paper with eric, playing with the hammer board, a copy of Amarok open with a half-finished podcast (an interview with Randy Millholland of "Something Positive" from 2005), probably about a hundred unrelated web page tabs open in (count count count) eight different Konqueror windows, kmail with seven half-written replies...
A reboot is actually a bit of a production. An _unexpected_ reboot is a pain the ass.
Ah, it appears that my problem is that not all Core 2 Duo processors have the VT extensions. The T5300 does not. Oh well, time to look at kqemu I guess...
Ok, dear whoever made the "kqemu-common" package: where's the rest of it? Kqemu is a kernel module, this package doesn't contain the kernel module. The only other package that comes up on a search for "kqemu" is the source code. Why bother with a kqemu-common if there's nothing _there_? There's no "recommends" packages, nothing...
So, tinycc path logic. It builds "hello world" again. Yay.
Still trying to figure out what the intention was in some of this code. I've almost certainly broken the Win32 support because I can't test it and I'm not quite sure how it's intended to work.
I also want to add some of the toybox "which" logic to tinycc, because I want to look for the compiler include and lib directory relative to the tinycc executable. Locating your current executable on Linux is either a question of doing a readlink() on /proc/self/exe, or looking at argv[0] and using it directly if it has a "/" in it or trying to find it in $PATH. In toybox, that's about 16 lines of code from which_in_path() and 33 lines of find_in_path(). Still, I can do that later, lemme just get it basically working first...
Done. Wheee. It builds "hello world" again, on both x86 and x86-64 hosts. Not to clean up the warnings, revisit the octal escape logic, update the test suite...
The correct invocation to use "netcat -f" with the serial port on the hammer board turns out to be:
stty 115200 -F /dev/ttyUSB0 stty raw -echo -ctlecho -F /dev/ttyUSB0 stty raw -echo ./toybox netcat -f /dev/ttyUSB0 stty cooked echo
Added much -v logic to tinycc.
Darn it. I can't get C to do something I want it to do. Right now I'm doing:
cc '-DBLAH="string"' tcc.c
And then in the source, going:
char *x = "prefix-" BLAH;
And letting string concatenation turn that into "prefix-string" for me. What I _want_ to do is take the quites out of the macro definition because the actual value of the string is a shell environment variable, so I need someting like
-DBLAH='"'"$BLAH"'"'
Which is just insane. Worse, if I try to echo that line out for debugging purposes the extra quotes get stripped. It's very hard to both print and execute that line from a shell script, because if it's evaluated twice quotes vanish.
Unfortunately, my first attempt at supplying macros without quoting them (cc -DBLAH=string) was to have C code that looks like:
#define TOSTR(x) #x char *x = "prefix-" TOSTR(BLAH)
But that gives me "prefix-BLAH", not "prefix-string". I can't figure out how to get the darn macro to be evaluated before it's converted into a string. Grrr. Frustrating.
Yay, the ##c channel on freenode knew the answer:
#define TOSTR(x) TOSTR2(x) #define TOSTR2(x) #x
And _that_ gives me a TOSTR() macro that does what I want.
Ah, that's where that article went: milk the new oil.
Oh brilliant. Just brilliant. They marked the bug invalid. This is the "kde thinks there's no network connection when there is" bug, which dozens of people have hit and was "status high" before being marked invalid. What is WRONG with these people? Yeah, Napoleon said never attribute to malice what can be adequately explained by stupidity, but at this point that's actually starting to be an argument _for_ interpreting it as malice if you ask me... AAAAAAAAHHHHHHH!!!!!!
Maybe I should try xubuntu...
Cleaning up tinycc. Nobody's done this in years. (For example, tcc_add_include_path() can't return nonzero: there's exactly one return statement in the function and it returns zero as a constant. Therefore the code checking the return value and printing an error if it's nonzero can never trigger. Of course the reason I noticed this is I'm removing all the seprate add_blah_path() functions and implementing a single add_dynarray_path() function that operates on a common structure type, but that's a side issue...)
At Starbucks', trying not to listen to a man use religion to convince a young couple to buy insurance from him. Really. It's not often you hear "God's plan for you" and "government regulation of Google" in the same conversation.
My wife has finally gotten fed up with Livejournal, and is backing up her journal to move it somewhere else. She was annoyed at the random deletion of communities without explanation or appeal, and all links becoming mouseover popups that block the page you're reading, and the mandatory content filtering which recently added a click-through so that when reading her journal anonymously, you get a click through asking you to confirm that you're 14 years old. (Really.) But the last straw appears to have been six apart selling livejournal to a Russian company with connections to both the censorship-happy Russian government and Russian organized crime. (Admittedly, lots of overlap there.)
Neither of the above two paragraphs strikes me as things that belong in a normal day.
Added netcat to toybox, and the serial port in qemu seems to be behaving itself. I'll try the board again when I get home from starbucks. Going through the "low hanging fruit" list and adding "chroot", I wasted several minutes trying to figure out why it was complaining about:
char *default[] = {"/bin/sh",0};
Before noticing that "default" is a C keyword.
I find myself in need of "netcat -f" to play with the new ARM "hammer" board. (It's either use netcat -f or try to get minicom working, and I've forgotten which way you swing the chicken.)
Are the arm "hammer" developers trying to make some sort of baking soda related pun? Huh. Anyway...
Unfortunately, using the netcat version in BusyBox 1.2.2 results in a seriously misbehaving serial port connection to the hammer board, and I'm not sure if it's a bug in the old busybox version, or in the software on the board, or in the Ubuntu kernel, or what? I don't poke at BusyBox anymore thanks to Bruce, and I've now written two netcat instances (one in 2001 and again when I rewrote the busybox version), so trying to debug busybox's netcat is uninteresting (even though I wrote it), so this is a good excuse for me to add this to toybox. I can get it working under qemu then try to attack the hammer board again.
I should also go through the "low hanging fruit" section of my toybox todo list and see how many of those I can get done in one sitting. And then, finish the path logic rewrite over on tinycc.
Right, off to do that. (Somewhere away from the cats.)
Walking into Starbucks reminded me of the Hitchhiker's Guide to the Galaxy. Specifically, the part: "And if we're unlucky? The Vogon captain might want to read us some of his poetry first." Sunday afternoons all the coffee shops seem to have distracting "entertainment" and no free tables. Even the coffee shop inside Barnes & Noble was full. Wound up in a booth at McDonald's. (Yay laptop with actual battery life.)
Hmmm... Couple of interesting corner cases. The OLDTOY() macro needs an argument description string just like the NEWTOY() macro does. In theory different commands that use the same basic function can have different arguments. In practice A) nothing does yet, B) their command_data structures would conflict if they diverged too much (although that's nothing a typecast to another structure type can't fix). This comes up because "netcat" has a largeish number of arguments, and doing an OLDTOY(nc, netcat) for it requires repeating this longish option string. Modern compilers should be smart enough to merge these strings (I wonder if tinycc knows to merge duplicate strings yet) but it's a bit of a maintenance burden keeping them in sync by hand if they change.
An alternative would be to pass NULL to oldtoy and have that mean use the old option string, although it's not immediately obvious to me the best way to actually implement that. Hmmm...
The other corner case is that the option parsing logic doesn't know to make "<2" conditional on "-f" not having been supplied. For now I can just do the test by hand, but if it comes up again I may need to complicate the option parsing logic some more. Sigh...
Spent today under the weather, got nothing done.
Both toybox and tinycc are getting contributors, and traffic on the mailing lists. On the one hand, this is a really positive development and I want to encourage it as much as possible. On the other hand, my ability to wander off into my little corner and focus exclusively on one problem for three days is now being interrupted by editorial duties. Replying to other people's questions and reviewing their patches are priority interrupts, not getting around to them for a week is bad form.
Meaning spending a day under the weather and getting nothing done leaves me with more work tomorrow. Wheee...
I may finally have to give up on kmail. No matter what I try, I can't get it to show me the email address of a message I'm reading. Not unless I either reply to the message, or mouse over it look at the info field at the bottom fo the screen. This is _sad_. It used to do this, but now it's decided to be more outlook-like and hide information, and I can't find a way to make it _stop_ doing this in the configuration anywhere.
Yes, this is enough to make me switch email clients. First of all, I've always had an easier time remembering email addresses than names, and this makes that go away, which is _deeply_ annoying on a personal level. It also makes certain mailing lists unusable (such as the Dell desktops mailing list: is this message from a dell employee replying to a question, or is this message from a user asking a question? Can't see unless I can see if the email is from somebody @dell.com..). Maybe thunderbird's developers aren't so amazingly stupid they think an email program shouldn't even have the _option_ to show email addresses?
Met with Mark today. Spent three hours talking about C with him.
The Hammer board from the Tin Can Tools guys arrived. Broke it open and spent some time frowning at the contents of the CD it came with. I'll probably actually power it up and try it out tomorrow.
Starbucks continues to have a good "hot chocolate with orange syrup". Still no blackberry syrup. Lemon cake's nice, but pricey and I got an end piece when I ordered a slice.
Finally got loopfiles() checked into toybox. Amusing how a function so small takes longer than big ones. (It was almost, but not quite, right for the longest time. Handling the empty list case with a seperate if() outside the loop turned out to be a lot cleaner than trying to handle it inside the loop and having to perform the same end-of-list test three times.
Banging on tinycc for a bit. The dynarray logic is all wrong, there are two related variables for each one (a pointer array, and a length), but they're not in a structure. So basically everything that adds to, frees, or iterates over the contents of one of these suckers needs a seprate function because it needs two unrelated variables and is thus harder to genericize. However, include_paths, sysinclude_paths, and library_paths are char *, but other things (cached_includes, loaded_dlls, sections...) are various struct pointers. Hmmm...
I need to fix the dynarray stuff to genericize the logic I added for handling colon separated paths. I did it for library_paths but not the two include_paths, and they're almost exactly the same code, I refuse to block copy this...
Mark says that the Darwin 8 install ISO (I.E. text mode MacOS, runnable under qemu) lives here, but it needs a login. Oh yeah, Apple's interested in actual open source development on this... Made an account, now it wants me to read a big legal thingy and agree to it (sigh)...
And it won't boot under qemu. It made it as far as some ACPI thing, then hung. Ten minutes, no CPU usage. Wheee...
It's difficult to be particularly enthusiastic about the windows output mode of tinycc. About half of it is hidden behind #ifdef WIN32 meaning if you try to cross compile from a Linux box, it probably won't work. It's also full of endianness assumptions, meaning if you try to cross compile from a big-endian machine it probably won't work. And then there's the x86-64 host I'm trying to make work: Windows uses LLP64 instead of LP64 (meaning a pointer won't fit in a long integer on 64-bit windows, which it's guaranteed to do on any Unix including MacOS X). I really can't bring myself to care about 64-bit windows, but I'm making the rest of it 64 bit.
On the one hand, it would make my job way, way easier to just drop all the win32 code. But there are users, and it sort of worked a few months ago...
Oh well, for now, keep it there but not much I can do about the bit-rot just now. Once I get ELF and PE properly untangled (and start adding Mach-o), perhaps I can try to test it under Wine...
Finally got the new toybox list created. Time to move the them.com lists over...
I have driven my new laptop (with 2 gigs of ram) into swap thrashing. I knew it was only a matter of time...
Is anybody else kind of frightened that this whole biofuels thing will drive the price of food through the roof? Yay renewable resources, but if we dug up half the farmland in the country because it had oil under it we'd all be terrified of the effects, yet half the corn and soybean crops going to fuel automobiles for the forseeable future doesn't make anyone even slightly nervous?
Of course making fuel from plants that were bred for _taste_ isn't the world's most efficient idea. You want ethanol? Figure out how to ferment kudzu already. Corn really isn't very good for this. Ethanol as a whole actually sucks pretty badly, it only has about 2/3 the energy per gallon as gasoline anyway. Vegetable oil biodiesel works ok (just about a 1:1 replacement for conventional diesel fuel, with very little processing), and we were already making plastic out of soybeans so there's a certain amount of poetic symmetry here. But making fuel from food is still a kludge. Fuel production needs its own plants specially bred to do _that_ well, and that's going to take a while. And even then, land's a finite resource: the fabled stockpiles of government cheese and butter ran out back when milk passed $3/gallon, and I believe paying farmers not to grow food (so they don't exhaust the aquifers under the midwest) ran out this past year. Atlanta's already running out water. It would be nice if somebody competent was in charge of worrying about any of this.
I suppose corn syrup no longer being cheaper than cane sugar or honey is something we can look forward to from a taste perspective, and McDonalds is valiantly holding the line on its 99 cent double cheeseburgers, but face it: inflation's running something like 10% annually. I know the shrub administration disagrees with this, but since when can anyone involved in that mess do math? Yeah, "volatile food and energy prices" can triple in the seven years since the last regime change and that's ignored because nobody ever has to actually pay for food or energy. Gas is now $3/gallon, milk's more expensive than that, the falling dollar is even raising the prices of cheap imported chinese things made out of lead, the mortgage industry collapse means housing prices may have STOPPED climbing sky high for the moment... What on earth are they measuring?
And think about the falling dollar for a moment: dollars buy less around the world, dollars buy less here too, but we're not having massive inflation on a scale last seen under the Carter administration (which was triggered, ironically enough, by the OPEC oil embargo of the 1970's). Riiiight.
I'm in an interesting ping-pong state at the moment. "Ooh, new toybox patches from Charlie, must process. Now what window do I have toybox source code open in... Ah, this window has RFC 1321 describing the MD5 algorithm, need to finish reading that... Of course I need to finish sha1sum up first, where did I leave off on that? Ah yes, I was editing catv.c to do generic "iterate through file arguments" code... Hey, phone call from Stu telling me that my tinycc project was on the Reddit front page (for about five minutes, fallen to page 3 by the time I looked), I was banging on that all last night and should finish it up. Left off in nasty tangled path logic. Frown at that for a bit... And _this_ window has a .bat file for compiling tinycc on Windows with mingw, except that won't work with the current version (although the fixes are mostly just simplifications) and I haven't got a test environment for that. Speaking of test environmnets, over in THIS window is a Firmware Linux build that failed because the darn gcc path logic went all wonky again, substituting the x86-64 linker during an armv4l cross compile. Why didn't it do that last time? Right, frown at that later, too many open windows. Why hasn't the git repository on kernel.org updated in 8 days? Linus must be on vacation. I still have a half-dozen documentation patches to push upstream and a large todolist to go through there before I can consider the kdocs project at a good stopping point... And you don't want to _know_ about my web browser tabs. (Still haven't finished the 64-bit paper for Eric, either.)
Progress is being made on all of these, but in 3-minute increments. (So what do I do about it? Update my blog, and rant about how I miss the McDonalds' $1.99 "American meal" back in y2k.)
Phone call about a possible job tomorrow. Off getting the car fixed. The usual.
I wonder if I can con Mark into adding a MacOS X backend to tinycc? He has a testing environment for it, and does understand how OS X executes applications.
Another good article on the long drawn-out death of the music industry. Most of the articles I've collected on this are a bit old now (such as this and this, which I linked to when I wrote this. I note that was written so long ago that this hadn't been written yet...
Of course these days, Weird Al has personally uploaded every video he's ever made to youtube. (And yes, he had to get youtube to put "don't download this song" back up after explaining irony to them.)
I despise apache2. They decided to totally rearchitect the thing to make threading work better on Windows NT, dragged the Linux world along for the ride, and made gratuitously incompatible changes to the configuration files while they were at it so an apache1 howto doesn't tell you how to make it work in apache 2. Plus they changed the name of the executable so you keep needing to remember to append a "2" to things.
The reason I'm despising it right now rather than ignoring it is setting up a self-signed https certificate is more trouble than it's worth, and when apache refuses to work it often won't tell you _why_. Tired, off to do something else now.
Broke down and posted on the old tcc mailing list. I try not to, but when there's actually a thread about me that's... er... "based on a lack of information", shall we say... Mentioned the new tinycc mailing list while I was there.
Checked in the sha1.c I did on the trip to the toybox archive. Might get to hook it up as an actual sha1sum command this evening, who knows?
Finally got the car's oil changed, about 4000 miles late. Lamb's says that the front brake pads are worn out (suspected that), the right front tire has a chunk taken out of it (ah, interstates), and the steering and radiator fluids need changing (wheee). Probably about $500 altogether, deal with it on Monday.
It turns out "another brick in the wall" has a music video. Ok, the whole of Pink Floyd's "the wall" is a giant music video for said album. I'm watching the scenes of students jumping into a meat grinder and other students taking fire axes to their classrooms (now the building's on fire) and remembering that my high school Chorus teacher showed us this movie in class one day. (It's musical!) All I remember is that it was better than having to deal with class. (Oh, now they're throwing their teacher in the fire... Aw, it was only a dream sequence.) Turned int a deeply screwed up and fairly depressing movie about halfway through. No wonder they showed it to us in high school.
House MD was much better...
Long drive. Got a few chances to poke at sha1.c but mostly I was either driving or sleeping. Got in around 11pm, reassured the cats a bit, collapsed into bed.
On the road back to Austin. At the moment Fade's driving and I'm trying to get some work done on my laptop.
The public domain sha1 code works much better if you A) tell it you're on a little endian machine, B) change the various instances of "long" to "int" on a 64 bit machine. Now it's producing the right answers, so I can start untangling it and shrinking it down.
The SFLC fundraising letter I mentioned yesterday is now online. (Due to me emailing them and asking them if they'd put it up, but that's only mildly cheating.)
I'm used to treating my communications with them as "more or less NDA" by default, since they're lawyers and attourney/client priviledge and all that. (And due to previous experience with other laywers that I _someday_ hope to be able to talk about.) But if your fundraising letter is meant to be kept secret? You're doing it wrong. So I asked. :)
I note that I still haven't given them money either. But I did turn down a share in a five figure settlement and told them to keep it, so I'm not _that_ guilty about this.
Somebody dug up Richard Stallman's policy statement on why gcc must be a huge tangled hairball. Not just monolithic, but without clean separation between any of its pieces. This turns out to be intentional: to prevent anyone else from using parts of gcc with backends or front ends under a different license (like the Stanford Checker did anyway, or GCCfs, or the first versions of LLVM...)
In reality, this tangled hairballness of gcc is probably the main reason it's turned into such a bloated unintelligible pig. There's a Linux Weekly News article on this topic today. (Yes, I mentioned my tinycc project in the comments. Cleaned up its web page, too.)
Darn it, I haven't gotten the revamp of the 64-bit paper done. Eric did his pass in a day or so, but mine's a boatload of research that spawns several tangents with each new paragraph and has to be edited back down. I wonder how much I can do on the car trip, without internet access? I was hoping to focus on tinycc and toybox on the trip back...
So the the SFLC has filed two more lawsuits. Good for them.
I know the enforcement actions are in my and Erik's name, but we delegated it all to them and it's mostly just "Can we proceed against _blah_? See attached document for details," and then giving them permission and rewriting the quotes in the press releases into something I actually might have said.
We've delegated it like this because Erik's hall of shame just didn't work, and trying to do it yourself can eat your life. These days gpl@busybox.org goes straight to them and we don't even see the complaints until the lawyers ask us for a green light to file against somebody (with attached "why we're doing this" references which I then try to clear half an hour to read). They even include a prepaid fedex envelope when it's time to sign and return actual documents.
I don't mean to sound disinterested: somebody has to keep the industry honest and I'm glad to do my part. I choose to release code under GPLv2 and I'd like the license to _mean_ something. But I'm coding, writing a research paper, and Eric and I are going to try to make time to get a doclifter release out before I have to head back to Austin. I'm very glad that somebody _else_ can do all the heavy lifting on this, which includes figuring out who needs suing and who doesn't.
I suppose I should put a link to their promo button on my site somewhere, although since it links to their main page rather than their donations page I'm not sure how useful it would really be. (They sent out a nice fundraiser email last month explaining that they get the majority of their funding from vendors, but if they don't get _some_ individual contributions they might lose their public charity status, and they don't want to be just a vendor mouthpiece anyway. I considered sending 'em something, but I haven't got a paypal account, and I like being able to say "and it hasn't cost me anything" when talking about my relationship with the SFLC. Still have the email marked as "todo" in kmail, for what that's worth. :)
Good thing about Ubuntu 7.10: the left and right scroll buttons in the Konqueror tab bar work again. (You couldn't scroll left without a mouse wheel in Konqueror 7.04, and my laptop's touchpad hasn't got one.)
I ordinarily wouldn't be interested in something that close to rap, but That Calls for a Wilhelm Scream is just inspired. (Hey, I'm a sucker for the movie making process. Not so much actually watching movies as what goes on behind the camera. It started as an interest in special effects technology and got seriously fanned by Bernie Brillstein's book Where Did I Go Right?)
So the obvious thing to do (for now) with the tinycc build thingy is probably to create a config.h file with the appropriate #defines and just have the source #include that. Don't force the shell to try to deal with strings with quotes in them, it's not good at that.
You know, I never knew that the Fifth Doctor was married to Trillian. (and this was amusing too. And this is hilarious.)
Yes, the flash plugin is working again!
Finally got a chance to look at the public domain sha1.c code and it doesn't work. Or at least it gives a different sha1sum than the sha1sum command. Hmmm... It's much easier to start with something that works. Hard to debug this if it's doing several hundred complex mathematical steps before I get a result I can compare against something known. Probably I should go look at the wikipedia description of the algorithm or something...
So the sound on Eric's motherboard still doesn't work yet due to a driver bug in Ubuntu 7.10. He had a backup sound card he's had in a box for something like three years, but the sound output of that skips randomly a couple times a minute. So today he went out and bought another cheap sound card, plugged it in, and it has the same skipping problem. Turns out both of the pci cards have a cmi8738 chip and the 64-bit driver for the cmi8738 is buggy. So that's three different instances of sound hardware, two different types of hardware (the onboard sound is an intel 945 chipset), and Ubuntu 7.10 can't competently drive _any_ of them.
Oh yeah, ready for the desktop. Right.
P.S. Dear kmail developers: for years now, kmail has shown the actual
email address ala "Some Name
Slept until about noon-thirty after ditching the Arkham game at 2am. (The owner of the game apparently took it home at 3, so I didn't miss much. It "plays much faster" with only 5 players instead of 8, they were maybe half finished by that point.) Decided not to buy a Sunday badge at this point, hanging out with my laptop in the lobby instead, trying to catch up on Mt. Todolist.
Building libtinycc.a in a canadian cross is fundamentally awkward. If you build a sparc cross-compiler and then build an arm-targeted tinycc with the sparc cross compiler, you do indeed get a sparc hosted tinycc that outputs binaries for arm. But to build libtinycc.a you need an arm cross compiler that runs on your host, and where does that come from? In theory we should always be able to build a native cross-compiler for the target; if we can't then we don't support that target anyway, so it doesn't matter. But knowing exactly when to do this so it's not unnecessary work is slightly tricker...
Speaking of tricky, getting a shell script to be properly verbose is a minefield. In theory you can just do:
function showcmd()
{
echo "$@"
eval "$@"
}In practice, that doesn't work if your command is "CC=cc" assigning a variable, and if you do anything like:
showcmd $CC -DTHING='"'"$THING"'"'
The carefully arranged quotes that should result in the C code doing the equivalent of:
#define THING "contents-of-$THING"
get stripped out by the "eval", which evaluates quotes _again_ and doesn't pass them on to the program you're running.
Hmmm... The closest to what I want seems to be "set -x". This has the unfortunate side effect that instead of printing "commandline" it prints "+ commandline", but maybe there's a way to turn that off. (And there is, "set PS4=".) And of course it prints "+ set +x" when I turn it off again. Wheee...
Orthogonal behavior: wouldn't it be nice to be able to beat it out of gnu bloatware? And you wonder why I want to use something else. While gnu make continues to be crap, I'm starting to see why people don't use gnu bash for this either: it's crap too.
I could write a C version of "showcmd" that would do what I want, but compiling something in order to build is a bit silly. I could trivially do this in python, but do I really want to make that a build requirement?
Why is "show all the commands you shell out to, but not the internal plumbing" such a hard concept for bash to cope with? The dos command.com used to be able to do this one (and had @command to suppress it).
The prettiest girl at this convention asked to borrow my laptop so she could view a rough cut of her newest music video. There are times it's good to be a geek. :)
Short term toybox todo items: xstrcpy(), attribute_printf, and look at this...
1:30 am at Philcon. Exhausted when I got here (and suffering from way too big of a dinner) and took a nap, now everything's pretty much closed up until morning (including of course registration). Oh well, pound on laptop. The internet access here is $10/day. I can wait until Sunday, I think.
Still happy I figured out how to get the flash plugin working yesterday. (See tail end of yesterday's post.) My 7.10 laptop slowly approaches the level of usability that 7.04 had...
Taking a whack at updating my writing page, at least a little. Updating my User Mode Linux HOWTO for 2.6.23. Found _another_ problem with the Defective Annoying Shell (/bin/dash) while doing this: my example User Mode Linux command line:
./linux rootfstype=hostfs rw init=/bin/sh
And when UML boots up, there's no command prompt. Why? Because of dash. It'll respond to commands you type, but there's no prompt so you have to figure this out for yourself. Why is there no prompt? Because /dev/stdin isn't a tty (because /dev/console has always historically been weird for reasons I've never tracked down any reasonable explanation for). It's sitting there waiting for a command (with no script to run and an initial nonblocking read from stdin returns nothing) but it can't figure out it's in interactive mode so it gives the user no hint it's waiting for a command. Presumably it thinks you went "/bin/dash < script.sh" or some such.
Sigh. Very much not a bash replacement. Horrible judgement call on Ubuntu's part pointing /bin/sh at that piece of trash, but that was a bug they introduced in 7.04, not a new one in 7.10.
[Many hours later] Eric Flint's talk is good. He says he has a series of articles called "against big brother" on Baen's Universe, that should be enough to Google for it later. (His GoH talk at Philcon is basically what's going to be in his next article there, apparently.)
The con suite has caffeine free Mountain Dew. I'm still boggling, hours later. Spent more money than I should in the dealer's room. The Voltaire concert was great. (I wonder what it would take to get him to Penguicon?)
Playing "Arkham Asylum" or some such in the gaming room, not particularly impressed (as can be guessed from the fact I've pulled out my laptop and am updating my blog instead. (The gaming room is full of people yelling at the top of their lungs, for no readily apparent reason. It is not fun.)
Poking at the tinycc make files a bit, but not getting much done. Occurred to me that "./make --clean" is stupid, there should be a "./clean.sh" and each command should be a separate script and they should all be as small and simple as possible, so once again I'm refactoring the new build system. Presumably I'll have all of this out of the way at some point.
Woken up at 8:30 this morning (7:30 am Texas time) by a phone call from a recruiter. One blog post and there's blood in the water, apparently. :)
Darn it, Kmail used to have a "file->new->main window" menu option that would let me open a second kmail window open looking at a different folder. Apparently this feature went away in the increasingly misnamed "upgrade" to Kubuntu 7.10. I wouldn't mind so much if switching windows didn't take upwards of 30 seconds (linux-kernel, 74,582 unread messages, and my own email box currently goes back to 2004), and didn't tend to lose my place if I fetch new messages while not looking at that folder.
Spent far too much of the day replying to email and working on the revision to the 64-bit paper. Much research. I have a huge pile of stuff I want to do to tidy up the documentation thing, and to get tinycc ready for a release, and work on toybox (got some of that done yesterday, at least), and firmware linux (Mark's building it under MacOS X), but right now I'm about to head out to Philcon, and may not have net access again until late sunday night.
When I get back to Austin, _then_ I hope to be able to tackle the todo list. I fear I've gotten to the point where working through my todo list spawns new todo items faster than I dispatch them, but still. Nothing for it but to press on!
Figured out how to get the darn flash plugin working. I had to uninstall the "gnash" package. No matter what I tried, gnash insisted on trying to handle every flash file (instead of the actually functional flash plugin) until I not only shot it in the head but removed all traces of it from the system. You _cannot_ use the two side by side, gnash neither offers nor respects any configuration options. (I can watch the daily show again. Yay! Ok, there's a writer's strike, but it's the principle of the thing and a bit of an archive at this point.)
Off to Philcon...
And so the Documentation job ends, although they're being really nice about it and paying me through the end of the month. I've written up three different "I think the documentation job is over" blog entries over the past month, but deleted them instead of posting each time because I wasn't quite sure enough to make it a reality by effectively putting a resignation letter in my blog.
Got a phone call this morning. It's nice to have closure.
This was one of the stranger jobs I've had. The Linux Foundation didn't quite seem to know what they wanted me to do. "There isn't enough documentation for the Linux kernel" is a problem description, but not a job description. I think they figured that out _after_ hiring me. The pervasive uncertainty has been my main source of stress about this job ever since.
Then again, it took me three months of working on documentation full time just to find the edges of the problem space, and figuring out what needed to be done was essentially my day job. The fact this led me in a different direction than the Linux Foundation guys expected isn't entirely surprising.
I suppose it didn't help that the guy who hired me, Tom Hanrahat, quit to join Microsoft before I could even sign the contract. Then I was reporting directly to the CEO, which didn't work because he was constantly busy and traveling. Then I was reporting to the technical advisory board, a committe full of Linux developers elected to join a mailing list, each of whom thought I was someone else's responsibility. (Reporting to a mailing list you're not allowed to subscribe to is a surreal experience.) Then I was reporting to the newly hired CTO, who turned out to be busy and constantly traveling (only managed to actually make 2 of the first 5 weekly phone calls he set up, although he was very apologetic about it). Just before the six month mark, I got a phone call explaining I needed closer supervision (something I'd actually asked for earlier), and was given a specific assignment with a deadline for the first time since the interview process. I heard back from them exactly twice between the time I handed it in and this morning's phone call. (Once one of the tech board members asking for a different patch format so he could rebuild the xml on his own machine, and once the CTO assigning me more work with only slightly more specificity than "do more of that", half of which I'd already done by the time he contacted me.)
On the plus side, I had a remarkably free hand for six months to improve kernel documentation. I did the best I could, and I still think that locating, collecting, collating, and indexing the huge piles of existing documentation out there is a necessary first step before trying to fill in the holes about what isn't properly documented (at least in any sort of systematic fashion). Plus if you can't _find_ the documentation on something, then it isn't documented no matter what somebody wrote in a blog post back in 2005, or once posted about on linux-kernel, or what the -git commit comment for the feature says.
The Linux Foundation had apparently expected I'd become the kernel's Documentation subsystem maintainer, and that people would naturally start to write and submit more of the stuff once there was one. (Or maybe I could pester them into doing so, or inspire them by example, or some such.) Unfortunately, that's not how maintainers work. A maintainer's function is editorial, they filter through a slush pile of submissions, reject the unacceptable ones, and polish what's left into shape to publish. Doesn't matter whether it's software or print media, the editorial function remains the same. As Alan Cox says, "A maintainer's job is to say no."
Besides, documentation has the worst case of bikeshedding in the entire kernel. Everybody who can read documentation thinks they know how to write it, and how it should be organized. If I tell somebody like Greg Kroah-Hartman "I really don't think foreign language translations should be added to the Documentation directory, they should go on the web", and he ignores me and adds them anyway because he disagrees, what am I supposed to do about it? (Answer: focus on a web page that I _can_ organize into a coherent whole.)
I still need to reach a good stopping point before moving on to other projects. I was trying to do that by the end of October, but they interrupted me to work on SCSI documentation instead and then the 7.10 install ate a week and change, and now I'm at Eric's. The kernel.org/doc page needs about another week or two of work before it's a paticularly useful resource for other people.
I'll start job hunting when I'm back in Austin after Thanksgiving.
Still up in Pennsylvania, visiting Cathy and Eric Raymond.
Finally got mailman set up. There is now a new mailing list for my tinycc project. It might even work.
Mail was bouncing all night because I forgot to run genaliases after adding mailman's virtual domain database to the postfix main.cf. The thyrsus stuff worked because that's the primary domain of sendmail, so that goes in alias_maps instead of virtual_alias_maps. Plus I forgot to change PUBLIC_ARCHIVE_URL in mm_cfg.py when I told apache the archive alias was "lists" instead of "pipermail". It's probably still broken in various corner cases. (I didn't do the recipient delimeter change because it says it's optional, I don't think I enabled that, I don't quite know what it does yet.) But I can has mailing lists now!
Fade is here. Yay!
Fade tells me that the espresso shots packaged in little coffee creamer containers are now on sale, so we don't have to get them from the Exxon anymore.
Got mailman working today. I _think_ it's working, I'll poke at it a bit more before moving over the lists I have on them.com. Eric has a list for the jargon file now, too.
Meanwhile, Eric's been doing his pass updating the 64-bit paper. I tackle this tomorrow.
Eric bought new hardware today, and we spent far too much of the day juggling machines and setting things up.
Got 15 minutes to spend with tinycc and fixed the \n problem (it was a thinko on my part). It now builds and runs "hello world" just fine. Now to get it to build itself, and the test suite. But first, hacking on the build infrastructure a bit more to get all the environment variables set in a separate configure script sourced from make. (And if people run ./configure it'll set the environment variables. But it tests each environment variable first, and won't re-set ones that are already set, so you can override these in your actual environment if you'd like.)
Except that it's not quite that simple. An x86-64 host is also an x86 (32-bit) target, except that the shared libraries are in /usr/lib32 instead of /usr/lib. So if I build an i386-tinycc on an x86-64 host, I need to set the linker's library search path to /usr/lib32 instead of /usr/lib. (Note that varies by _host_, not target. At runtime it all works the same, the library search path is supplied by ld-linux.so.2 when the file actually executes).
Speaking of the runtime linker, tccelf.c has this lovely #ifdef block:
/* name of ELF interpreter */ #ifdef __FreeBSD__ static char elf_interp[] = "/usr/libexec/ld-elf.so.1"; #else #ifdef TCC_ARM_EABI static char elf_interp[] = "/lib/ld-linux.so.3"; #else static char elf_interp[] = "/lib/ld-linux.so.2"; #endif #endif
So FreeBSD sets the runtime linker to /usr/libexec/ld-elf.so.1, ARM EABI sets it to ld-linux.so.3, and everybody else uses ld-linux.so.2. Except that uClibc needs ld-uClibc.so.0. But anyway, that shouldn't be in the C source, that should go in ./configure or something. But that varies by target, which means it would have to re-source ./configure each time through the loop. But _that_ conflicts with the logic in ./configure that won't re-set a variable that's already been set. Hmmm...
All this is fixable. I just need to think about it. I'm trying to come up with a cleaner compiler design than the horrible mess gcc imposes upon people, and I'm still in the process of identifying and discarding all the broken assumptions both Fabrice and myself got from too much exposure to the mass of fundamental conceptual wrongness that is GCC's path logic. (Don't mince words Bones, tell me what you really think.)
Fade arrives tomorrow. Looking forward to this.
That's the second time software suspend (or "hibernate" as Ubuntu insists on calling it) decided it didn't want to suspend. Instead it wanted to blank the screen, do something that occasionally accessed the disk in short bursts, but never flush everything to disk and never power down. It's actually _less_ reliable now than it was in 7.04 (which failed due to lack of allocated disk space, but seldom for other reasons).
Once again, 7.10 is a regression. And you can't blame this one on KDE.
Out of morbid curiosity, I tried to boot the "Indiana" preview (I.E. GNU/Solaris) under qemu. First I tried "qemu -cdrom indiana.iso" but the default memory qemu emulates (128 megabytes) is too little for the solaris kernel to initialize (!). Added "-m 256" to that command line and it went further along, until it went into a slow endless loop doing:
WARNING: /pci@0,0/pci-ide@1,1/ide@1 (ata1): timeout: abort device, target=0 lun=0
Went and played with ReactOS instead. Its livecd boots up to a desktop. Whee. Doesn't do anything, but hey. It's much easier than trying to get Wine to do anything. (Normally, installing this sort of thing is the hard part. But with Wine, getting it to _run_ anything is rocket science.)
Ha! Finally got tcc running under x86-64 to compile a runnable "hello world". It seems to have trouble parsing the \n, but otherwise it worked.
I had to set the paths by hand, though:
TINYCC_CRTPATH=/usr/lib32 HOST=i386 TINYCC_LIBS=/lib32:/usr/lib32 ./make --fast ./i386-tinycc -I /home/landley/tinycc/tinycc/include hello.c ./a.out
On an x86-64 host, the 32 bit libraries are in /usr/lib32 (rather than /usr/lib where they are on an actual x86 host). This means that "x86 on x86-64 system" is actually a slightly different build target. Wheee. Not _quite_ sure how to gracefully configure this yet. Need sleep first.
This means the library search path, the C runtime path (where crt1.o/crti.o/crtn.o live, which isn't a search path but a specific location) all had to be adjusted (above). The header path (where #includes live) are the usual /usr/include because the asm directory of that is full of #ifdefs which include the 32 or 64 bit versions (although since I'm not installing tcc yet, I need to feed it the -I to add the compiler includes from the build directory). And the runtime linker path is the standard "/lib/ld-linux.so.2" because the 64-bit version is apparently "/lib/ld-linux-x86-64.so.2". (They couldn't change that and remain compatible with existing binaries.) I need to make that configurable too, so it can build against uClibc.
Right now there's a lot of #ifdef logic in the source to provide values for various targets. I need to think through how I _want_ all this to configure.
And fix whatever's wrong with \n parsing...
I'd like to thank Jim Tan for the patch he sent, but alas:
host b.mx.jtan.com[207.106.84.147] said: 554 5.7.1 Seems like a
dynamic IP address. See http://www.jtan.com/blocklist/ (in reply to MAIL
FROM command)
Eric Raymond's FIOS line has had the same static IP for a year now, yet some of the black hole lists out there are still broken. This is not my problem, but it's sad I can't think a contributor.
Jim: if you manage to read this, thanks for the patch. I've applied it. And may I introduce you to this marvelous thing called postgrey?
No, I don't know what those A things are, it's a bug in Kmail that inserts high asci crap in the whitespace of anything I cut and paste out of my email. (Makes it really hard to compile C code since gcc dies when it encounters this, yet it doesn't show up in vi for some reason.) Yet another thing to be thankful to Kubuntu about, although this was broken in 7.04 too...
Spent rather a lot of the day setting up a new grelber (the server several incoming ports of the above FIOS line forward to). It went live to mail bouncing (uninitialized alias and greylisting databases) and web pages not resolving (feed the behind-the-firewall IP address to the apache virtual hosting setup, not the outside-the-firewall IP address), but I think those are sorted now.
Did some cleanup work on my new tinycc thing. I need to get it building (and working) on x86-64 before I do another release. (It doesn't have to output x86-64 binaries, it just has to run on an x86-64 host to produce x86 and arm binaries.) Working on it. In addition to the zillions of sign mismatch warnings, there are now "pointer assignment from integer of different size" warnings which are probably actually important. Waffling between cleaning all that up immediately, vs making the minimal changes to get it linking against the right library paths so I can build and test "hello world" before making extensive changes that might potentially break it, without being able to test between each change.
Either way, it's probably not likely to work out of the box. Oh well, I can fix that. (And then learn enough x86-64 assembly to make a new target. Not looking forward to it, but it has to be done.)
Spoke to Fabrice Bellard (ok, typed, he was on freenode as "hint") and made sure he didn't have a major objectin to my tcc->tinycc fork. He said I could do what I like with tcc. Yay. (While I technically didn't do anything that would require extra permission, this makes me feel much much much much better. The Bruce thing left deep scars. Luckily, Fabrice ain't a Bruce.)
I now have two active contributors to toybox, one of whom has queued up a lot of patches for me in his own mercurial repository. I'd like to encourage this, but I couldn't quite manage to clone his repository through static-http. (For the record, I'm a big fan of the mercurial export and import commands. Annotated patches that maintain revision control history. Perfect!)
Went back and repaired the html markup in the November 3 and November 7 entries. The downside of editing my blog in vi is I tend not to view it much in an actual web browser. (It's mostly notes-to-self anyway. Mostly I keep it as is for the historical record, but if I'm going to use the package list in the June 17 entry as a guide to upgrading the next time I install, I'm going to add four or five packages to it when I find out that yes, I had to add those too. A balance between historical accuracy of what I wrote at the time, vs updating my notes to match reality...)
Rsyncing 40 gigabytes of backups from the old grelber to my laptop, so I can sort them into some kind of order. It's copying (via rsync) at just under 1 megabyte per second, and I am so _totally_spoiled_ because that's too slow. I have a laptop that has over 40 gigabytes of currently unused space to copy them to, I have a wireless connection that can go that fast, I am currently doing the download on battery power without particularly worrying about it (the indicator says I'll have to worry in 1 hour, 29 minutes)...
(Smacks gums, shakes walking stick.) I started with a 300 baud modem. (Both ways!) Hooked up to a commodore 64. (In the snow! Or at least with video output that sparkled because of some kind of interference with the character set rom that could cause spurious sprite collisions. Also, the picture was kind of fuzzy if you tried to display red or do an 80 column character display which meant each character was 4 pixels wide, on a monitor that was essentially a high end televison without a tuner. And the display got _really_ upset when mom ran the vacuum cleaner.) It had 38911 basic bytes free and we were _thankful_ to have it! (Just under 39k, for those of you counting along at home. Plus a 4k block at 0xC000, and you could swap out the rom banks to read the memory underneath by zeroing all the bits at location 1, although this meant you lost both the OS and BIOS. Yes, basic can be an OS. Or at least a program loader with built in command language and very special purpose editor and monitor features. Imagine the love child of grub and bash, only much smaller than either of those. And with the screen's character buffer substituting for a command line history. (You want to re-run a previous command, you cursor up, and the cursor moves _up_ and screen scrapes what you typed earlier.)
Those were the days. Boy did they suck, in retrospect. Which is why I'm spoiled by this measly 1 megabyte per second transfer rate. (1000000/38911 is 25.6996736141 according to Python.)
Right, bedtime...
Today looks like it's going to be _insanely_ busy. Oh well, not entirely unexpected.
Something I did installed exim4 on my laptop (possibly an autodependency of mailman?), and it was preventing my ssh email forwarder from binding to port 25 on loopback. That took some head scratching to track down. Yay "lsof -i".
Trying to replace docproc.c with a shell script. This involves understanding how "make xmldocs" works, which is slightly non-obvious. (Writing new code is easy. Reading existing code is always the time consuming bit.)
Argh. When applying a directory full of nicely broken out patches to one's temporary directory, be sure to specify the temporary directory and not the source directory which a mercurial repository that has pending changes in there that haven't been broken out yet. Sigh. There went an unnecessary half-hour of cleanup work, since I did it _twice_. (First I thought patch wasn't breaking hard links. I'm updating make/functions.sh so the temporary copy of the Linux directory uses hard links, which are much faster to create and eat less space.)
Eric is still preoccupied with Wesnoth development. (They apparently just had a release, which he's happy about the bug count being lower than usual in for a C++ application.) Showed him the C++ Frequently Questioned Answers page (much chortling while he read bits of it) and talked about what would be involved in porting more of Wesnoth to Python. This led me to investigate Python OpenGL bindings, which led me to PyOpenGL, which Ubuntu has prepackaged as "python-opengl", which is associated with the truly useless python-opengl-doc package. (How do I write "hello world" code that uses this? Download a demo source tarball from their website, note that it has a 0 byte readme, compile it, note that it won't run if you don't install it and the installation must be run as root and requires a third party installation script that doesn't come with it. Not likely.) But the website's documentation page linked to an OpenGL Tutorial which has a python translation of the example code, and that theoretically can run standalone. Downloaded it, ran it, and it segfaulted in glutInitDisplayMode(). (Not a python traceback, a segfault. Which really shouldn't happen in python code.) This is a python opengl app to open an empty window. It doesn't even try to draw a polygon (that's lesson 2). And the package segfaults.
Eric's of the opinion that if I download it and build it from source, the segfault will probably go away. Let me just say that I am so deeply impressed by the Ubuntu 7.10 release right now, words fail me. So do random Ubuntu packages, in unexpected ways. Right, on to other things...
AAAAAAARRGH! Speaking of compiling things from source, the x86_64 cross compilers I built for gcc are broken. I didn't notice because all the mini-native stages built, because the breakages is in the path logic again. (If you move the toolchain to another directory, it goes "boing" again.)
It's using the host assembler for all targets, because once again the path logic sprug a leak. Because none of the 15 places (!) it's looking for the assembler actually contain it if you relocate the cross-compiler-$ARCH directory, and thus it falls back to the one in $PATH. I can make the wrapper adjust the $PATH for its child process so the right one is first, but then collect2 can't find ld (which we just adjusted the path to _add_!) I can delete collect2 and symlink it to the correct ld during toolchain prep, but I wanted to add C++ support to this thing someday...
And you wonder why I'm rooting for pcc and banging on tcc? This thing is _broken_...
But anyway, I _did_ confirm that Ubuntu x86-64 does indeed run 32-bit executables, no problem. Good to know. And I can tell tcc to build against /usr/include (which has an asm directory full of #ifdefs for i386 and i686) and link against /lib32, and output 32-bit exectuables which I can run right here to test.
Still getting used to the fact that the video on my laptop is now the correct 1280x800, not 1024x768 stretched to a cinema display. This is a _good_ thing about the upgrade to 7.10, I guess I was just spoiled by the preinstalled 7.04 where just about everything worked out of the box.
Arrived at Eric and Cathy's. Too late to get any work done, I'll catch up Friday and over the weekend.
A big advantage of the new make/make.sh is that it won't delete the old htmldocs directory when it's not rebuilding it, so I should be able to rsync updated versions with impugnity (or something spelled similarly to that). (Technically I separated them so htmldocs is no longer a symlink to Documentation/DocBook, so rebuilding Documentation doesn't delete the old htmldocs. Wrote an actual installer to move files from one to the other.)
Well it's a big advantage to _me_. I've accidentally blanked the http://kernel.org/docs/htmldocs directory and left it that way for days enough times to not want to do that again.
The merge of the i386 and x86_64 architectures introduced a new architecutre, x86, which is "special". Every other architecture has an arch/$ARCH/ directory containing Kconfig and Makefile entries. If you try to "make ARCH=walrus allnoconfig" it complains "No rule to make target `.../arch/walrus/Makefile'." meaning if you _added_ that it would use it without modification to the rest of the kernel infrastructure.
But the new x86 directory doesn't have a "Kconfig". Instead it has Kconfig.i386 and Kconfig.x86_64, and it added special case code to the top level makefile to check for these and redirect them. Yet x86 itself isn't usable as an architecture, if you try to allnoconfig it you get "can't find file arch/x86/Kconfig". Because other architectures like powerpc can combine 32-bit and 64-bit versions in a single directory and still work like all the other architectures, but x86, isn't that special? It's no longer detected by the script that generates http://kernel.org/doc/menuconfig because it's just that special.
Meanwhile, while trying to document menuconfig and I just realized I can't _run_ menuconfig until I get network access again. Kubuntu installed curses, but not curses-dev. (Wouldn't it be nice if there was some way to tell it "install the -dev for every package that is otherwise installed? I have the _libraries_ but not the header files. Considering that Firefox alone is 32 megabytes, and KDE is enormous, why are they quibbling over header files?)
Speaking of firefox, here's what "aptitude show firefox" has to say:
Description: lightweight web browser based on Mozilla Firefox is a redesign of the Mozilla browser component, similar to Galeon, K-Meleon and Camino, but written using the XUL user interface language and designed to be lightweight and cross-platform. This browser was previously known as Firebird and Phoenix. This is a build of a random development version (aka trunk). It is ment for preview and not for production use.
No, I didn't build it myself. That's the version that comes with 7.10. And no, my cut-and-paste didn't respell "meant" either.
Tenessee seems to have a disproportionate number of cute girls. I wonder why?
Poking at the kernel build, trying to make "ARCH=x86" build. It seems to want to default to i386, so I'm making a list of all the places I need to force it to x86-64. It's getting -march=i386 from _somewhere_, and the only place that's set is arch/x86/Makefile.cpu, which is only included from Makefile_32, which is included from Makefile (all in the same directory), but tweaking Makefile to include Makefile_64 instead doesn't seem to have any effect. (Possibly commenting out an include directive doesn't work. I'd hate makefiles much less if I could stick a printf in them to see what variables are set to at a given point, but they're explicitly designed not to allow that, which is sad.)
It has something to do with bouncing off of scripts/Makefile.build... Ah, that needs to blank KBUILD_CFLAGS. Except that breaks the build later on. (Breaks the x86_64 build, too.) Right, worry about that later.
So "sed -n 's/^!.//p' Documentation/DocBook/*.tmpl" produces a list of 300 file imports of which 184 are sourced for Exported symbols, and 111 for Internal symbols. (There are also 4 F references and 1 D reference.) Sorting for unique files gives 271, so several are imported more than once. let's take a random look at one, kernel/resource.c which is imported in kernel-api.tmpl line 307:
<sect1><title>Resources Management</title>
!Ikernel/resource.c
!Ekernel/resource.c
</sect1>
So, A) this file is being imported twice, once for each kind of symbol, B) the symbols aren't marked as to whether or not they're exported, and what type of export it is.
Ok, so half of the function of docproc.c is to find all the exported symbols in the files. As far as I can tell, this is several hundred lines of C to duplicate approximately the following shell script snippet:
for i in $(sed -n 's/^![EI]//p' Documentation/DocBook/scsi_midlayer.tmpl) do echo $i egrep -o "EXPORT_SYMBOL(|_GPL)[ \t]*\(.*\)" $i done
Watching somebody write C to avoid a five line shell script is... Um... Painful. Oh well, at least they didn't do this part of it in perl.
So docproc.c shells out with 'kernel-doc -docbook -nofunction function_name filename", using -nofunction for internal functions and -function for external functions. Can kernel-doc take a mix of those? (No of course not, that would be too easy.) Reading the perl... What a mess. In addition to the xml output type being used, this thing can also output html, gnome, man, and text. This script is carrying along huge amounts of code that are never used. Of this 2007 line perl script, maybe only half that is actually needed.
Ow. Headache. (I can't read much perl in a sitting. I can untangle some pretty bad code, but my brain rebels against even the cleanest perl. It's evil.) Ok, back to driving...
Starting the long drive up to Eric and Cathy's. I planned the trip when I thought the documentation job was over, but I have my laptop with me and can work on stuff along the way.
Still fighting with Kubuntu 7.10. They've broken vim in new ways every release for a while (obviously a distro put out by people who don't like vi), but the bit I can't figure out how to undo is that it stopped scanning embedded vi comments, ala:
/* vi: ts=4 :*/
That sets the tabstop to 4, which is what I use when I write code. (Yeah, the kernel says you should indent 8 spaces. This is one of the reasons I don't write much kernel code. Until I spent time on busybox I used to use two spaces per indent.)
It's hard to look up something like that when you're not sure exactly what it's called. Possibly I can set something in the config file to bring it back, but if so I don't know what. Several things they removed (like syntax highlighting) were yanked at compile time. I tried looking for a bigger vim package, but my first attempt didn't help matters... Ah, there's a "vim-full". Maybe that's worth a try... (Sigh, they built it against the gnome libraries. It's a command line text editor, and they infected it with Gnome. That's sad. If I wanted to use kate, I'd use kate.)
Nope, it _still_ isn't reading the tabstop reset out of the file, which the previous version was doing.
Not a happy Kubuntu user. The networkmanager bug shows no progress, and none of the ones I've filed have been responded to, that I've noticed.
[Many hours later]
I love mercurial. I'm on the road, running from battery, no net access to be seen (this truck stop claims to have some, but it doesn't work), and checking the updated make scripts into the kernel documentation repository, locally on my laptop. This is _so_cool_.
I also like my new laptop. As much as I'm annoyed at Kubuntu for once again breaking stuff that USED TO WORK, moving from 32-bit to 64-bit is like getting a whole new laptop, only 6 months after actually getting a whole new laptop. I can has battery life!
I'd give Kubuntu a lot more leeway if the breakage I've experienced recently had anything to do with moving from 32-bits to 64-bits, but it mostly didn't. The hard drive thing was arogance. Knetworkmanager is just stupidly designed. I'm repeatedly told that using flash 9 via 32-bit nspluginwrapper works for other people, so why doesn't it work for me? Why does vim's feature set contract so drastically each version? And so on...
In each of these cases, half the problem is that whatever it is has been automated to the point where there's no easy way to diagnose the problem. It either works, or a failure occurs deep in the guts of something and I have to pull out my UberGeek hat to figure out what's actually gone wrong, let alone how I might possibly fix it. The hard drive thing just lurks until the drive fails. The browser plugin recognition thing is so automated I can't point it at a plugin sitting in a known location, or get it to say WHY it doesn't want to load it (or if it even noticed it was there). The whole knetworkmanger experience is layers of opaque breakage (silently ignoring my hardware, silently crippling konqueror, and I can't get either to tell me anything, they just _fail_ with no explanation at all)...
Oh well. Still, yay Mercurial, yay battery life, yay 64-bits. (And some things are better in the new kubuntu. For one thing, the "dynamic" frequency scaling no longer seems to get stuck at 800 mhz.)
It's monday, back to the salt mines.
So, when the scsi documentation thing came in as an interrupt, the kdocs make/make.sh was on the floor in pieces being converted from a big shell script into a series of smaller shell scripts. That's why kdocs site hasn't updated in a few days (including getting the new scsi_midlayer.tmpl into the htmldocs page).
The advantage of this breakup is that the big script did lots of things and sometimes I want it to do just _one_ of those things (such as rebuild index.html from master.idx). Doing all those things together is very very slow. (For example, make htmldocs is a 5 minute job even on my laptop. The "many sections" version is _way_ slower than the -nochunks version, I think that it's re-parsing the entire xml file for every separate html file it outputs. And -j 2 makes no difference because this part of the kernel build is wrapping a shell script. Anyway...) The search for RFC entries in source comments is almost that slow as well, and I didn't want to make that an unconditional stage, but adding more "--short" style flags with different granularity was getting silly.
So I broke the script up, but the downside of having lots of small shell scripts turns out to be lots of duplication. The checks that $LNXDIR and $WEBDIR are set get a bit repetitive after the fifteenth one of those; common code seems like a good thing.
So when I was almost done breaking them up, I glued all the little scripts back together into one big include file and I'm making it define lots of bash functions(). So if I need to call one or two of the things, I can source the file and call the functions by hand. (Or make a wrapper script with arbitrary granularity, in about 30 seconds.)
Of course I broke stuff doing two extensive conversions back to back without without ever reaching a good testing point, and now I'm debugging it all again. I'd have finished this last week if I hadn't lost several days upgrading my laptop to the pile of bugs that is Kubuntu 7.10.
It's a monday...
Tried easybuntu to see if that could install the flash plugin. Nope, it hasn't got a config for Gutsy because that one "just works" for a definition of works that does not actually involve working.
Wow, 64-bit mode builds very very slowly compared to 32-bit mode on the same hardware. Probably the 64 bit integer sizes are blowing through cache more quickly (almost certainly doing bad things to the stack). Or it could just be gcc, which is what's been trying to build for what, half an hour now? I haven't timed it...
Nope, it's gcc. All the other packages seem to build about as fast on x86-64 as they did on x86, but gcc builds waaaaaay slower.
[12 hours later] Ok, finally killed the "./forkbomb.sh --fork" session. For some reason, m68k died with an internal compiler error (maybe that one doesn't work on an x86-64 host, have to see if it's reproducible), and none of the others had made it out of the cross compiler build after being left to run overnight. (The most advanced one was trying to build uClibc.) Back on x86 on the same hardware, an entire forkbomb parallel build of all targets took about 3 hours total.
At a while guess, the CFLAGS="--param ggc-min-expand=0 --param ggc-min-heapsize=8192" I'm doing is starving 64-bit gcc, at least when building itself. That optimization works fine on 32-bit platforms, and makes the insn-attrtab build of gcc _not_ trigger the OOM killer on a virtual machine with only 128 megabytes of memory. I'll try making that conditional on [ "$(uname -m)" != x86_64 ] and see what happens. (It would be nicer to test sizeof(long) without having to compile anything, but I haven't figured out how yet. Nothing obvious under /proc or /sys presents itself, sysinfo is useless, none of the standard environment variables shown by "set" indicates this...)
Ooh, that's _much_ happier. Built x86-64 in 24 minutes, 6 seconds (with only 27 minutes "user", so not taking as much adantage of SMP as it could, but that's what --forkbomb is for.)
So, that optimization is only correct for 32 bit systems. Maybe gcc-min-heapsize=16384 is a better on 64 bit? I can play with this...
Darn it, I had a mechanism to tell vi not to call fsync() on its .swp file every hundred keystrokes (In a heavily I/O bound system causing the process to pause for 20 seconds or so, a _pain_ when you're typing). But I apparently didn't note it in my blog. Yeah, the new install's doing that and I forget how to switch it off. It was some "set" command in the vimrc file...
And I can't search the help because they broke that into a separate package I haven't installed yet (vim-doc) and I'm at a starbucks which has T-mobile branded lack of internet access. (And no more blackberry flavor syrup until spring.) Great...
Ok, whereis vim, "strings /usr/bin/vim | grep sync", and that contains "swapsync". Try ":set swapsync=n"... well, it took it. But no, I just got the darn pause again. That didn't work. Hang on, but was it a bad pause? I _am_ doing a forkbomb build in the background, with the memory constraints I'd added switched off. The occasional three second pause while vi gets knocked to swap is kind of expected. It's a twenty second pause twice in the same minute that gets on my nerves... (Yes, I'm typing to see what the pause behavior is like. So far, seems reasonable.)
Kubuntu 7.10 still sucks: the saga continues.
I shut down the laptop last night rather than trying to suspend, on the theory that Kubuntu might think it's windows and that behaving differently after a reboot was somehow a good thing. But no, knetworkmanager is still broken this morning.
Went into the system settings menu, fired up network settings, switched off "zeroconf network browsing". Nope. All the other buttons are greyed out. Ah, eventually figured out I had to scroll down to see the "administrator mode" button (despite my display being bigger than 1024x768 now and the window maximized: it's designed for a display _bigger_ than 1024x768. Brilliant.) Went in and told both network cards (which _this_ thing can also find; EVERYTHING but knetworkmanager seems to be able to) to configure themselves via dhcp, and hit "apply". It sat there with a "please wait" dialog for literally thirty seconds (timed it, hello dual 1.7ghz 64-bit processors). Then I exited, and it told me I had unsaved settings. What the...? There was no save button or ok button, just the "apply" button I _already_hit_. Told it to save again anyway. Another 30 second wait. Went into knetworkmanager, and this had NO EFFECT, it still has no cards.
Right click on knetworkmanager (bringing up approximately the same dialog via a different method, first time I tried it was while the other one was open and it told me it was already open), disable eth1, enable eth1... Hello 30 second wait... And it accomplished nothing. Disable and re-enable eth0...
I can set this interface up manually, no problem. I just can't figure out how to get the darn overcomplicated "dbus uses this configuration file" thing to work.
Tried to switch Zeroconf network browsing back on, and got a pop-up that this opens port 5353 and there could be security implications of that. Um, hello, THIS WAS THE DEFAULT. Your default is insecure? What's wrong with you?
Firefox is happily browsing the web. Konqueror still refuses to connect to any site. This is just _sad_. Perhaps this will finally force me to give up on Konqueror.
Between this and the sprained wrist, I really haven't accomplished much on my computer in the past couple days...
Installed all the packages I need on top of the base install, and nailed the approprate bits of the old kmail config into the new one. Got email reading and writing, but when I click on a link in an email it says it can't connect. Of course, it's trying to use Konqueror, which is WILLFULLY BLIND.
Oh, and the "submit bug report" widget for KNetworkmanager doesn't work either. Judging by the error pop-up, my "send custom host name to server" in kmail (which I have to do for my outgoing email to work) isn't being used by this thing. Wheee...
And when I right clicked on the kmail toolbar to add a "reply to all" to the thing, Konqueror crashed (Kontact actually). Luckily, it had saved the change before crashing, so it was there when I restarted the thing. I had to re-hide the Kontact application selection bar and the new "favorite folders" thing, but I closed and re-opened this time to make sure that's saved.
LAYERS of broken. Layers.
Ah, found it. There _is_ a gratuitous incestuous relationship between Knetworkmanager and other kde applications, through dcop. They _do_ check if it's ok to access the network and don't even _try_ if dcop says it isn't. That's just sad. And totally unnecessary complexity that exists just to break, apparently. Yay, Konqueror works again, and now I trust it about as far as I can throw it because I know it's got stupid built in. Hmmm...
Other things I'd hoped would get fixed by the upgrade to 7.10, but weren't:
Sometimes when I click on something without moving the mouse cursor (not even one pixel, easy to do on a touchpad), the click doesn't register until the cursor moves. So I can click on the close window button, wait five seconds, then move the mouse and the window closes instantly. (Clicking again also flushes the previous click.)
Probably as a side effect of this, the mouse can get stuck "down". This is most noticeable when I click on a scroll bar one click at a time to advance the screen, and it gets stuck down and scrolls 20 pages before I can stop it.
Also, the "Korganizer reminder daemon" that pops up whenever I start kmail is korgac. "rm /usr/bin/korgac" and it stops popping up. I didn't write that down last time. (I don't _use_ Korganizer, I use kmail.)
Darn it, the flash plugin no longer seems to install automatically. Instead it linked me to the adobe website so I could download a tarball. Of the 32 bit version, on a 64-bit machine. Ok, this can be _made_ to work, maybe, but it's not exactly end-user friendly.
And for some reason, in 7.10 the dell special function keys no longer work. (In 7.10, fn-cursor up would increase screen brightness, and fn-cursor down would decrease it. And it remembered, separately, the profiles for AC mode and battery mode. I just unplugged it and it dimmed the screen for battery, but the function keys aren't working anymore to brighten it back up. I guess that was one of the special things Dell did that I lost going to stock Kubuntu...) Ah, but if I click on the battery icon the pop-up now has drag bars for brightness, in both modes. Ok, I can live with that.
When batter remaining time drops below blah, action: suspend or hibernate, which one is to ram and which one is to disk? It's possible that suspend to ram works now, I should give it a try. But which is which?
After several attempts the free plugin to play youtube videos seems to have installed. Why repeating the _exact_same_action_ more than once was required, I have no idea. Oh, and it only works in Firefox, not Konqueror. Dunno why.
I still can't watch dailyshow.com though. That requires the actual flash plugin, and all my attempts to get it to work on a 64-bit system have met with failure. (Even though Ubuntu's website says I can just install "flashplugin-nonfree" out of the repository, neither Konqueror nor Firefox seem to notice that exists. Opened a bug on the ubuntu website about it.)
Firmware Linux 0.3.0 is out.
Watching Yakitate Japan at Mark's place. Every time I think this thing can't go further over the top, it manages.
Wow Kubuntu 7.10 sucks.
I've been cleaning things up to reinstall my laptop for a few days now. (Can't do an upgrade to switch from x86 to x86-64 installs, and I don't like to accumulate unknown cruft endlessly anyway.) Finally got everything backed up and all the various windows closed so I could shut my laptop down and do it. Burned a cd, wiped the old install, repartitioned the entire drive (reclaiming the two FAT partitions Dell put on the thing and making swap 4 gigs so it should suspend a bit more reliably)...
But within the first 30 minutes, I found several "showstopper" bugs in Kubuntu 7.10. The biggest is that knetworkmanager is _useless_, it doesn't think eth0 or eth1 exist (despite ifconfig having no trouble finding them). I'm told this is because it reads /etc/network/interfaces instead of actually looking at the hardware like ifconfig does, and the install only put loopback in there (despite warning me it was installing "unsupported" ipw3945 drivers for eth1). Why it didn't find eth0 is an open question.
When I go to the command line and iwlist/iwconfig/ifconfig/dhclient the wireless card up, that works fine from the command line. But Konqueror still refuses to access any websites. Other programs can, but not the web browser; it believes knetworkmanager, not reality.
I managed to log onto the #kubuntu irc channel and they said to do a dbus restart. This made it think my battery had been unplugged, took down the network connection (bye-bye #kubuntu irc channel), and generally made the system very unhappy. I'll reboot it as soon as my backups have finished restoring.
During all this, I was untarring a couple dozen (compressed) gigabytes of backups from my external USB drive, meaning the disk was constantly busy writing. This slows the system to a CRAWL. As a result of this, I right-clicked on a Konsole tab, told it to quit, and when nothing happened for 10 seconds or so, I did it again. Then both of them went through and konsole crashed. Yes, this took out the session that was extracting the big tarball, so I had to start over again. Wheee...
And of course the default behavior of vi is gratuitously unusable (now in new ways!), although this has the same fix as 7.04 did: "cd /etc/vim; rm vimrc.tiny; ln -s vimrc vimrc.tiny".
Went into /sys/class/net and did an "echo add > uevent" for all the interfaces there. No difference. Hmmm. Shut it down and deal with it in the morning.
So bzip2 does about 2 gigabytes an hour when working on one processor on my laptop. (Admittedly the sucker's stuck at 800 mhz again instead of 1.7 ghz, but that's Ubuntu 7.04 and software suspend doing stupid things. Again.) I know this because I'm doing a full backup of my laptop to the portable external drive, due to yesterday's hard drive scare. (Incrementals are nice, but I need to install 7.10 soon anyway.
I've meant to do a multi-threaded version of bzip2 and bunzip2 for a while now. The algorithm works in 900k chunks which are processed totally independently, so it's not really that hard. Conditionally adding thread support to the toybox build system, that's a bit harder... But having an SMP laptop is quite motivational for this sort of thing. :)
Ooh, got a new assignment from the Linux Foundation. Maybe this means I'm still employed by them?
The SCSI standard is a long and boring read. Still unclear on this whole LUN thing, but I'll get there.
Still haven't heard back from the Linux Foundation yet. Oh well.
Oh I really didn't need this. And according to
smartctl -d ata -a /dev/sda | grep "Load_Cycle_Count"the current value on my drive is 195422, and the article says they're rated for 600,000 cycles. I've only had this thing for four months. And yes, Dell chose one of the hitachi drives that has the biggest problem with this. Wheee...
The head of Ubuntu's laptop team blames the BIOS. Apparently working around a bios problem to avoid eating hardware would be a bad thing.
So I can make it stop doing it with "hdparm -B 255 /dev/sda" but have to redo that every boot. And adding it to the boot scripts isn't enough because the bios value gets reset by software suspend (add a script to /etc/acpi/resume.d), and then there's going into "laptop mode" by unplugging the AC adapter and running off of battery (maybe /etc/laptop-mode/laptop-mode.conf)...
Pinged Jonathan Corbet of lwn.net. Let's see if he's heard about it. Also subscribed to the linux-desktops list at Dell to give them a hard time.
Need to find a State Farm insurance office to get one of those Personal Article Policies Mark told me about. Laptop insurance...
There was a blog entry here but I don't remember what it was. Once again my laptop was using too much memory to suspend last night, and by fiddling with it (opening the lid while it was suspending once too often) it went "panic" and I shut it down. When I fired it up again, the entry apparently hadn't been saved. :(
I remember reading lots of scsi documentation, banging on tinycc...
Ok, I'm impressed. A day or two back I re-watched the Dr. Who episode "Father's Day", and now I'm watching the commentary. And despite how many times I've seen it, I'm getting distracted from the commentary by the episode and have to keep rewinding. (Good episode. And yet the two _after_ it are the ones that won the Hugo...)
Took advantage of clause 3 in LGPL to convert my tinycc fork to GPLv2.
Finally slogging through Randy Dunlap's SCSI documentation bug list. 90% of this is stuff that was already wrong in the kerneldoc comments in this files, but now's as good a time as any to fix them.
Finally set a time to go visit Eric. I drive up early next week, fade flies up a week later, and we drive back sometime before Thanksgiving (so we can feed Mark's cats while he's out of town).
Grinding away at a Firmware Linux 0.3.0 release. Taught the shared tarball extraction infrastructure thing to check the sha1sums of patch files applied to the tarball, and not just the tarball itself. (Way too many instances of "why didn't that patch work" headscratching can be blamed on that.)
Poking at the kernel htmldocs infrastructure to see what's involved in getting proper export type annotations into the htmldocs. I probably have to tweak the perl. Sigh.
Reading the tccboot source code. It's quite clever.
Walked to Epoch just after midnight (well I had to do something to burn off all the mutton). Banging on tcc a bit. (Er, "tinycc". Trying to remember to use the new name; tcc isn't my project.) First, I'm trying to make the build less tangled by creating a ./make shell script, and a Makefile that's just a wrapper calling that so people get the UI they expect.
I'm removing the ./configure stage ("./make --fast" if you don't want to build all the cross compilers), and also trying to remove most knowledge of what host it's building on from the make script. The one remaining bit that cares about this host is just a convenience, so A) --fast can build just one target, B) so that one of the cross compilers can be copied (after it's built) to the name "tinycc" (from "i386-tinycc" or some such).
I also need to come up with a libtcc for each target. I can test the arm target as soon as I get FWL building an EABI target.
I wonder if setting up wine would help me test the output of tcc for win32? Technically it's building a win32 cross compiler right now. (And I'm not _too_ worried about whether or not it compiles natively on win32 because I can always build the win32 cross-compiler and then build that with itself, then just put the binary up on the web page. Yes, build the windows version from Linux.)
I also have to finish fixing the test case:
printf("%d\n", ((void *)0) || "1" || 0);
Which outputs 0 built with tcc because I didn't quite get the constant the propogation logic right on my second attempt either. (But now it's broken in a _different_ way then it was before, which is... movement, if not progress.)
I don't think I ever linked to Mark's Flicker photostream of Fade's and my wedding. Proving beyond a doubt that NONE of the people in that room were photographing well that day. (Embarassing photos of me, Fade, Steve Jackson, Eric Raymond...)
He also took the cutest kitten picture ever, but then I'm biased.
Oh, I reversed the order of the RSS feed (Cathy's Google home page was showing the oldest three entries, rather than the newest three). If this screwed it up for anybody else, now you know why. (I thought everything sorted by date, but apparently not...)
Much laundry done today. Got about half of Randy Dunlap's response to my scsi patch processed while at it, then it was time for dinner. Sheep! Mark refers to this as "the mutton of the gods", and I really can't argue with this assesment. But then, I'm a sucker for well-done sheep. (I didn't know they only served it as ribs, but it's been several years since I was there last. Well I found it once by accident before I even met Mark, and then couldn't find it again. It's a fairly nondescript building with a name that tells you _nothing_ about what's in it. It's one of several places in Elgin that's famous for sausage, and that's nice enough, but the mutton's what you really want. Now if it just had Rudy's creamed corn...
Oh, the buttermilk pie was really good, too. Much groaning and laying on couch ensued at home, until PeeJee knocked my soda onto my backpack and I had to take my laptop out to dry off.
Got my last assignment finished and submitted.
I should look into configuring a mail client that can do the "send patch patch series from the command line and make each one a reply to the previous one" trick, because sending 15 patches by hand is a _pain_. (First attempt at #2 went to linux-kernel instead of linux-scsi, out of sheer force of habit. 2 of them said x/16 instead of 15. I note that kmail does not seem to be not cut out for this.)
Someday I need to teach the kernel docproc infrastructure to process both exported and non-exported symbols in the same pass, and tag each symbol with the export type. (The way it works now is just sad: you get either the exported _or_ internal functions, even if there are kerneldoc comments for both. And if you do both types, it complains and inserts garbage into the output if one of the categories becomes empty, say because all internal symbols you've commented get exported or the last commented export is removed from the file. Plus it doesn't list exported symbols that _aren't_ commented, and doesn't say which kind of export it was: EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL() or one of the weird per-cpu ones.) Plus the html output could be made to look a lot better, display-wise, despite xsltproc being evil incarnate.
Now that I have some free time, I'm trying to figure out how to set up mailman on landley.net, so I can do toybox, firmware, and tinycc lists on my domain. I was requested to leave the old tinycc mailing list but this is my darn fork and I'll do what I want with it. Renamed the mercurial repository to make that clear. I should cut a release of the work I already did before going off and doing massive code refactoring. For one thing, I'm going to make it install as "tinycc" and use that name (instead of tcc) to distinguish the two. Yeah, Fabrice still has tinycc.org but he mostly used the name "tcc" for the thing, especially the release tarballs. Easy enough to distinguish that way. I should still ping Fabrice about it just to be polite...
Planning a trip to Eric and Cathy's place in a week or two. (Fade has to get a plane ticket, but that's for a week after I drive up.) Trying not to overlap with thanksgiving because it's harder to get cat-sitting when other people have reason to be out of town, but there are definitely worse places to be over the holidays than Eric and Cathy's...
Penguicon has a gaming GoH. Still no tech GoH. (I have no idea who they're asking. Mitchell Baker? Jonathan Corbett? Oh well...)
The cashiers at the Chick-fil-a on Ben White are now addressing me by name. Which is odd, because I've paid cash all along and don't wear a nametag, but I guess I've just come here _that_often_. (Well it has wireless! I have to log in to use the wireless and that's probably where they got the name from.)
The C++ Frequently Questioned Answers page is marvelous, especially the section that summarizes the major defects in the C++ programming langauge.
I got a list of files to make sure I use the kerneldoc comments out of, but some of them haven't got kerneldoc comments. Does this mean I should skip the file, or read the code try to add them (by friday)? Hmmm...
I hate DocBook validation errors. Tag mismatch I can find and fix, but validation gives me a backus-noir dump... Ah! Run xmlto on the _xml_ file, not on the tmpl file. Duh. (Ok, that was a waste of half an hour.)
Argh. I am incorrigible. I rush to have something to show at my 6am conference call (first one in 3 weeks!) and what do I do? Get distracted writing a totally unrelated script to grep through the kernel source for all mentions of an RFC and list which files mention that RFC. The result.
Ok, phone call happened. For the first time since I started this job, I have a clear, concrete, and achievable assignment, with a specific deadline. Wow. Exciting. Sleep now for a bit, then I can get to work.
Now I'm re-reading the SCSI-HOWTO, the SCSI Generic HOWTO, and the scsi source code, and generating a long list of questions for James Bottomley (the SCSI maintainer).
I'd say I was feeling under the weather, but the weather's turned cold and windy and generally fall. Maybe I'm feeling at the weather.
Last night I was in a hurry and when my laptop wouldn't suspend (again) I hard powered it off rather than playing whack-a-mole trying to get memory consumption down until it would suspend again. I just noticed that when I booted it back up, my "reply to all" button is gone in kmail. It's reverted to factory defaults for some reason. My attempts to edit the toolbars haven't managed to bring up an editing menu for the _right_ toolbar yet (right clicking on it brings up an editing menu for some other toolbar). I've had to look up the keyboard shortcuts (hit "a" to reply to all).
Sigh. I like kmail, but I despise Kontact. Dear KDE developers: loss of orthogonality and modularity is not a plus.
Updated man-pages to 2.67, seems to have worked this time. The code page removal step was what was failing, because there are a half-dozen symlinks that point to nonexistent files (because doclifter couldn't translate them).
My upstairs neighbor has decided to learn the Saxophone. Went out to McDonalds to use the laptop on battery for a while.
Beaking my head against kernel-traffic some more. I figured out that the main problem seems to be the old "/bin/sh points to dash" issue (and that Ubuntu has a /bin/echo that doesn't understand -e), but the nested makefiles nature of this thing has so far prevented me from actually fixing it. Environment variables I set don't get passed on to child instances, nor do make variables defined on the command line.
Wound up renaming /bin/echo to /bin/echo.broken and temporarily pointing /bin/sh at bash. (The ubuntu developers _do_ develop it under itself, right? You'd think they _notice_ that dash is crap and they ship an echo that doesn't understand "-e", but no...)
Whee, now it's made it this far:
xmlpretty --PrettyWhiteNewline \
--PrettyWhiteInd \
--CatchEmptyElement \
rss.tmp > rss.rdf
/bin/bash: xmlpretty: command not found
What the heck is xmlpretty? Oh well, it created an empty version of the file so if I run it again it may skip it... Yup. And now it broke because it wants stats/latest.html and the file it's creating is called stats/latest_stats.html. I can see where this file is being created, but not where it's getting the dependency. I really really really hate make. Right: MAKE A SYMLINK BY HAND...
Ok, the build made it to the end, and it has archives and everything. The archives say they were created using tools I haven't got, operating on a file format I may not have going forwrad, so this is deeply reassuring, but for the moment, I've recreated what's already there. That's something.
So I _think_ the *.xml files are the source, not the .raw files. Ah, I see, the raw files get recreated if you make "spotless", which is apparently this thing's way to say distclean. But the xml files contain a lot of stuff that looks generated. So: title, author, issue, intro. Check. Then the stats section is the bit that looks generated. And then "section" tags... The stylesheet is in the directory above, making it hard to grab this directory in isolation. Where's that set. From xsl. Oh sick. Yeah, I remember mark mentioning that. Barf.
Right, postprocess the result with "sed". That's the easy way to do it: let the nasty overcomplicated machine spit out what it wants and then cut the ugly bits off with an arc-welder. Maybe later write a python program to read the xml source and spit out the html directly...
Fixing up the codepages is:
find . -name "*.html" -print0 | xargs -0 -- sed -ri 's@href="\.\./(.*)css/kt.css@href="\1kt.css@'
Hands up everybody who saw this coming.
Need to add some new material to master.idx and not quite sure where to put it. It's testing and development material, which is sort of the heart of the document yet doesn't currently fit anywhere. Things like coding style, fault injection, how to read a panic message, etc. There's a lot of build up at the start (compiling, installing, reading source code) then it goes into a huge section on the guts of the kernel that's always goingt o be at least half the total material here. (And then a section on hardware goes logically right after that.) Then at the end there's a little bullet point on submitting patches in the "following linux development", section but that section's mostly about keeping up with outside development, not about modifying the sucker yourself. Hmmm... That section's half about "understanding the development community", which involves both the read and write aspects of kernel development.
Logically, modifying it yourself comes last. Read everything else first before doing this. In reality, people never will, and the only way to really learn this is by doing. All the documentation in the world just makes it _easier_, it doesn't actually teach you the way getting your hands dirty can.
Poked at getting kernel-traffic building so I could mirror it at http:/kernel.org/doc/kt but said build is deeply unhappy. (It has a makefile. Bad sign.) Mark was able to provide a little help at first (showing me how to anonymously rsync the whole of kernel-traffic, source files and all, to my laptop), but he's feeling distinctly under the weather right now. I've made very little progress on my own...
The mkxmlman.sh script was skipping the install step if the build returned an error, and doclifter is still misconverting a half-dozen files. Oops. So much for "improving" the error handling...
So 85% of embedded software is written in C. Good to have numbers.
Learned about the ubuntu-minimal and build-essential packages today while waiting for a tow truck for Mark's car. Mark or I might try get those to build for the various qemu target platforms, under FWL, to do an Embuntu.
Finished Naomi Novik's new book Empire of Ivory today (another potential Penguicon guest I suggested several times, and was apparently ignored). Ends on a cliffhanger, dunno when the next book comes out.
Michael Kerrisk notified me that something broke xmlman. Sigh. Doclifter (actually manlifter) should NOT be as brittle as it is...
Oh wow, what _is_ my todo list on toysh?
I need to write a getline() with cursor control, which means I need to query the tty size. The magic for that is:
struct winsize tty = { 0, 0, 0, 0 };
int ret = ioctl(1, TIOCGWINSZ, &tty);
plus $COLUMNS and $LINES, plus echo -e "\e[s\e[999C\e[999B\e[6n\e[u"
That's probably enough for the short term.
Ha! Finally got qemu-system-ppc to boot a PowerPC Linux kernel which I built from source. (Thank you, Milton Miller.) It only took what, two years to get this this point? :)
Checked the relevant bits into the Firmware Linux mercurial repository. The interesting part now is that qemu-0.9.0 (last stable release, from February) boots all the way to a shell prompt, but the development snapshot I've been using hangs probing the virtual IDE controller. Updating the snapshot to see if that fixes it, if not I poke the qemu mailing list.
I wonder who the Penguicon guys are thinking of inviting as their tech guests this year? I could suggest a dozen people (Inna Kuznetsova would be great. Robert Young and Tim O'Reilley would make a heck of a pair or work great individually. Jamie Zawinsky has musical interests. It's hard not to just BLURT OUT suggestions to these guys. It's _october_ and they still only have two GoHs, one of which I invited!
Oh well.
Finally finishing up my git quickstart document. As an example, I used the git bisect to figure out what broke the qemu's powerpc IDE support, and it turns out to be this.
My sister Kris had her fourth (and presumably final) kid this morning. Second daughter. Not yet named, but apparently healthy.
Whataburger is a nice place. Open 24 hours (not just the drive through but you can go in and sit down at 4am), they have pumpkin pies (of the McDonalds apple pie "extruded tube of semi-flattened pastry" style of pie, not the wedge shaped ones, but still), and their condiments include honey (for the chicken strips) and lemon juice (for the diet coke). Plus they have chocolate malts. If they had wireless internet and electrical outlets, it'd be perfect, but my laptop has 3 hours of battery and I've got all the OLS papers and a 1-deep mirror of the LWN kernel article index on my laptop, so I've got plenty of stuff to bang on. (Note to self: get a 1-deep mirror of the Linux Journal archive page so I can shuffle _that_ into the index when I'm not online. Not much I can do about wikipedia, but I'm not quite sure what to do about it anyway. Wikipedia articles are anecdotal evidence at best, subject to change without notice, and their moderators haven't impressed me as... er... consistently sane. If wikipedia's motto was "For amusement purposes only" I'd feel better about linking to it.)
Hmmm... Unpleasant side effect of moving kdocs/htmldocs into kdocs/Documentation/DocBook is that "make/make.sh --short" rebuilds the Documentation/DocBook directory, deleting the htmldocs from the last build. Rebuilding htmldocs takes _forever_ (13 minutes 53 seconds on my snazzy dual-processor laptop, although that's running on battery so it's at 800 mhz instead of 1733), mostly due to xsltproc being kind of evil, which is why --short doesn't do that step. (It skips that and the hg pull, because I'm not always connected to the internet. Like now.) But if I forget and rsync up to the website after a --short build, it'll delete the htmldocs, which is bad. If I exclude the rebuild of Documentation from --short, I have to do a long build to test applying the 00-INDEX patches to a copy of the Documentation directory. Maybe it would help to add a third mode?
On a totally unrelated note, Telluric Currents are cool.
I have no idea what's up with software suspend. If my laptop is throttled back to 800 mhz when I suspend, that sometimes becomes the new maximum speed on resume. If I have the volume turned down when I suspend, that lower volume sometimes becomes the new 100% on resume. Half the time I have to "modprobe -r ipw3945; sleep 1; modprobe ipw3945" to get the wireless card to start working again (usually because the stupid userspace regulatory daemon dies). The scheduling of the flash plugin goes all wonky after a resume (which is very noticeable when it tries to play video, I have to kill it and restart to get something other than a slideshow), which is a cousin of the "thought it was keybounce" problem from last month. And of course it's decided to reset my clock to eastern time zone again, despite me telling it the sucker is central and it _knowing_ it's central right up until it does a resume.
The frightening part is that all these weird behaviors are still an improvement over what it was doing six months ago. I've been using software suspend since 2003, and someday it may work as well as APM did. (Oh well, a reboot fixes it all, and Kubuntu's been making noticeable progress on this front. I'm really looking forward to 7.10. Maybe suspend to disk won't be my only viable option anymore...)
Took some time out last night to cool down and bang on toybox for a bit. Refamiliarized myself with the mke2fs code. (Wow, has it really been six months since I seriously worked on it?)
The Documentation patches I pushed upstream are finally starting to filter from Andrew's tree into Linus's, meaning make/make.sh doesn't build due to patch conflicts and I have to find each one that's been merged and yank it from the directory of ones I apply locally. Progress! I need to automate this somehow. :)
Going down through the empty master.idx sections in sequence, which means I'm currently Documenting the user half of the build process (make menuconfigure, make, make install). Lots of this is already in the README, but the granularity of that file doesn't work in this context. Still, linking to the README. Linking to the "make help" output. Linking to the menuconfig help I converted to html... There's a definite tension to "concisely describing what you need to do" and "linking everywhere you can learn more about this topic". But what else is new?
I'm trying to figure out how disturbed I should be by this message.
It was in response to my contention that laptops with a known and fixed number of built-in hard drives are an interesting device enumeration case (and will continue to be in future), which shouldn't be artificially saddled with all the problems of room-sized NUMA clusters (in hopes that the Linux desktop users will get fed up with the pain and fix the NUMA space's enumeration problems for them, and no that wasn't a supposition on my part but something I was explicitly told earlier in the thread).
I don't particularly care about the thread (and stopped replying already) because the people who write that part of the code already have their minds made up, and arguing with them makes them angry. (Besides, David Lang's doing a better job of defending my viewpoint than I was anyway.)
The bit that disturbs me reads:
> News flash! The kernel wasn't built just for you, and over time, more > and more people will have multiple disk drives of the same type, so we > will need to solve the device naming problem sooner or later. Why not > solve it sooner, especially given that a number of companies (not just > IBM) are funding the organization that is paying *your* salary are > interested in solving the general case?
The author of that bit is one of the members of the Linux Foundation technical advisory board that approved "paying *my* salary" (as he reminded me in a separate private email message today.)
I was unaware that the Linux Foundation had a "party line" I was expected to tow. (Especially on issues where I'm not claiming to represent them.) But that's just a side issue.
What I'm more concerned about is the implication that my evaluation of the validity of a given technical position is expected to change based on the source of funding. Yeah, I'm familiar with this in plenty of other contexts (the Tobacco industry comes to mind), I just didn't expect it from a member of the Linux Foundation's technical advisory board. Maybe he's not claiming to represent them in this any more than I'm claiming to represent LF when I express a technical opinion... but then why is he bringing up the "remember who's paying your salary" issue in the first place? And emailing me off-list to _remind_ me he has a say in it?
My previous coping strategy with this sort of thing was to avoid jobs that put me in this sort of position (at least beyond the point where I could ignore it by reprioritizing my todo list). At this point, I don't really expect the third quarter check to be approved anyway, so it's probably moot. But they've already paid me through the end of the month and I'm working to get http://kernel.org/doc in a shape I can point the world at (maybe get it reviewed on lwn.net) before then.
When a parent goes "as long as you live under my roof, you'll do X" a large part of what they're doing is encouraging their offspring to move out. Everybody knows this, right? I'm not reading too much into this?
Oh well, ignore it for now and see what they do. Meanwhile, I have work to do to get kernel.org/doc ready for public consumption by the end of the month...
Got up late, headed out to breakfast with Fade (it's still breakfast after noon if it's the first meal of the day) at the Jimmy John's on Anderson, got home, made tea, tidied up a bit, grabbed my laptop, and headed to Metro to get some work done... And freenode won't connect. And mark asked for a ride to Ikea (his car's in the shop), which Fade was pondering visiting earlier today.
Not looking like a productive day.
Back from Ikea, and I'm involved in a SCSI layer flamewar. I don't _want_ to be involved in a SCSI flamewar. (Religion, politics, the Linux scsi layer, and sysfs. Topics to avoid discussing in polite company if you want it to remain polite.)
Note to self: plugging in a usb ethernet dongle apparently makes knetworkmanager disconnect from the wireless access point in favor of the new interface. (Even when there's no cat5 cable plugged INTO the ethernet dongle. You'd think the MII transciever status would be "disconnected", but that doesn't matter: it got a HOTPLUG event! Woo!) On the bright side, unplugging the dongle generates a new hotplug event for knetworkmanager to respond to, and it reassociated with the access point automatically. Smooth, in an "I still want to smack it" sort of way.
Heh. The other day I changed how htmldocs builds. I'm building out-of-tree when I do "make htmldocs", which complicates applying patches to the original (I don't want to screw up my mercurial repository). So I symlinked the out-of-tree Documentation directory to the temporary copy I install into the kernel documentation web page (which is already patched). The upshot of this is that http://kernel.org/doc/htmldocs is now a symlink to Documentation/DocBook (just keep it where it builds rather than copying it into another directory). This means my 404 checker, which I run against Documentation to figure out what 00-INDEX files I still need to make and/or fix, is now trying to parse all the index.html files generated under Documentation/DocBook, and giving me pages of complaints (mostly to do with links to directories that don't end in /, links to destinations with #anchors in them, total failure to understand mailto: links...) Wheee...
So, back to the organizational question of what to do when I encounter something like Paul McKenney's excellent RCU documentation. It's a nice, well-organized lump, the only thing I'd do differently is reverse the order (oldest first, starting with the most basic introductory material). It's only _polite_ to link to it, and somebody else will keep it updated.
That said, it's not 100% complete (doesn't include Documentation/RCU/* from the kernel source code), and it duplicates some resources I already plan to index (lwn.net kernel articles) and in some cases have mirrored (OLS papers). If I don't index it myself, I have to teach my 404 checker about indirect coverage: I link to something that links to that, so don't pop it up as "not linked to".
The real question is whether having the data inline is a better user interface. This is a "clutter vs convenience and transparency" thing. Judgement call. I should probably ping Paul McKenney at some point and ask his opinion...
New Terry Pratchett book (Making Money). Didn't even turn on laptop today.
Sat down early to virtuously go through some documentation todo items I've been putting off, and followed a Project Wonderful ad to this. Four hours later, I'm through the archive and have yet to start on any of the actual work part.
Greg KH is _way_ too fond of wrapper libraries that give him an excuse not to document the kernel API. "The implementation is the spec" is always a bad sign. (This time I'm not just talking about /proc, now I'm talking about usbfs and libusb.) More source code to read in order to write up documentation...
More shuffling of master.idx, moved the cross compiling bit up to "building the kernel" and checked in last night's stuff too.
Poked at the m68k config in firmware linux but found out that qemu-system-m68k is still just coldfire. The infrastructure can apparently emulate an m68k processor now, but the only boards it knows about are coldfire (and thus under the Linux m68knommu arch rather than ARCH=m68k). The Linux kernel m68k defconfig is apparently for an amiga. Wheee...
Found two things wrong with the m68k Linux build in 2.6.23 while I was at it. Sent in patches. This doesn't seem to be an architecture that gets a lot of love. :)
Apparently, qemu cvs grew cris support over the weekend, I should give that a try. Ok, the gcc build for cris wants "limits.h", presumably from the Linux kernel headers, which is something no other architecture I've tried up until now has wanted...
Am I the only person trying to get consistent behavior across multiple cross compilation platforms?
Hmmm... Where does discussion of ttys belong? If I write up a half-dozen paragraphs comparing pipes, files, and ttys, does that go under VFS? Char devices? The TTY layer? Userspace API?
This is what's hard about documenting. Getting the info down on paper isn't so hard, _organizing_ it so you don't have to read all of it to understand any of it, that's the hard bit.
Each section of http://kernel.org/doc now links back to the index entry that linked to it. (The links are bidirectional.) I _really_ need to come up with a synonym for "index" because the whole thing is an index of kernel documentation, but there's also an index to that index, which is sort of a table of contents... Each section now links back to the table of contents entry that linked to it? I guess that's clear enough...
Installed mailman on my laptop to play around with it. I should really host my own mailing lists (although technically that would put them on grelber, the machine in Eric's basement, which is where my domain is parked)... The install tells me I should remember "newlist mailman". Right...
Collated some history of the Linux process scheduler. OLS papers are indexed partway through 2007. Still need to shuffle the slides into 2006.
I should write up a how to program in C tutorial, I've got two people who could use it and I've been asked several times in the past...
Fade and I drove through the night, and got into Austin around 11 am, much earlier than we expected. I didn't get any work done on the trip at all, I tried to sleep when it was my turn so I could take over driving.
Memo to self: the passenger seat of a Pontiac Grand Am, when you can't recline it all the way, is a fairly effective torture device. Don't ask me how, but I strained a muscle in my right foot trying to sleep in that thing, and Fade's back is all screwed up. Oh well.
Napped muchly, then walked to Mark's apartment to feed and pet his cats. (Only fair, he fed our cats while we were away all week, and then he had to leave for Arkansas thursday...) About a 3 hour walk, I think. Not bad. I'd planned to do an hour or two of work from here, but his internet has a WEP key which I don't remember, and of course knetworkmanager didn't retain. (Key to the front door, check. Key to the router, no.)
Driving back to Texas today. Frantic last minute packing of things my mother left us kids which my grandparents are still holding on to, plus email, plus downloading things to work on during the car trip. (I have now mirrored the whole of the lwn.net kernel index on my laptop, for example.)
Volume 1 of the OLS 2006 papers now has title and author names (which means I've at least glanced at each paper). Broke down and wrote some summaries of some of the papers as I went past since I've already read about half of these anyway. (I bought the printed proceedings that year, it was in the bathroom for several months in Pittsburgh.) Need to do a second pass to attach the lecture slides.
My grandparents, Fade, and I visited the remains of the house my parents built today. Considering the last any of us were there was 30 years ago, my grandparents are now in their 80's and I didn't drive, it took a bit of effort to find, but we eventually found it (after a phone call to my father at the end; we were off by about 150 feet) off of McCraken Road in Lake Helen, Florida. It's at the end of a little, slightly overgrown dirt road on the left, right before Reeves Run.
According to the neighbor who lives at Reeves Run, the house burned down years ago (and the woman who bought the house from my parents was arrested for money laundering for columbians or some such), but the foundation is still there, a big rectangular hole lined with cinder blocks, full of rubble and with a couple 10 foot high pine trees growing out of it near the back. Most of the utility room is still standing, the brick front steps and porch are still there and concrete slabs where the garage and back porch were. The entire site is somewhat overgrown with knee-high weeds. What was the back yard has been reclaimed by forest, but bits of the front yard and the side near the driveway are still clearings.
The foundation seems much smaller than I remember the house being, but I was 5 when we left. The land is for sale, but they want $190k for it.
I was working on getting gen_opic() to handle boolean and comparison operations on string constants ala ((void *)0 || "fred"). (The problem is that the string constant's value isn't in the SValue "constant" field, since it's a symbol that's not resoled at compile time, so the current code yields zero/false values when it shouldn't. All these tests only care about "is this NULL or not", and that we _can_ resolve at compile time even if the actual value isn't a constant. This is what was screwing up the toybox "hello" command; I thought I'd gotten this functionality working earlier, but apparently I only got the first half working.)
But then I got asked to leave the project, so I have. Ever since the OpenBSD guys decided to resurrect pcc to have a BSD licensed compiler to build BSD with, tcc has been inherently less interesting, and I was only really still working with it because I had momentum built up and it was so close to doing waht I wanted it to anyway. Once pcc builds OpenBSD, the Linux kernel guys will probably accept patches to remove some of the gcc-isms in the kernel source, and that's probably a faster way to a GNU-free Linux system than banging on tcc was anyway.
I really should put my spare time into toybox, that project's been neglected far too long. And I should catch up on the 64-bit paper follow up articles too. Or maybe I should finally get started on my computer history book. And it's not like I have a shortage of kernel documentation day job work, either...
I'm of course reminded of other projects I've been asked to leave, which makes me wonder if Penguicon is going to stick with two guests of honor this year? Every month or so I check up on them, because I still plan to attend even if I'm not involved in organizing this year. (It is Fade and my anniversary, after all.) It's hard to resist shouting out the occasional suggestion, but I'm not a meddling parent when the bird wants to leave the nest. I'm not running Penguicon, Matt and Jer are. I've told Fade a few of my suggestions for potential Guests of Honor (who I might be able to get even at this late date), but if I want to push my vision for an event I should start a new convention to do it in. (Not that I have _time_ to do that, these days. :)
I suppose this is the downside of having 8 gazillion interests; I'm not particularly motivated to "defend my turf" on any of them, since there are so many others I could be spending time on...
I did keep a nail from the house visit today, though.
Ok, an array of function pointers is int (*wubba[42])(char *a);, and int *wubba[42](char *a); is treated as an array of functions _returning_ a pointer (which is illegal). Also, int wubba(char *a)(char *b); is a function returning a function (illegal), and int wubba(char *a)[42]; is a function returning an array (also illegal).
Yes, I'm banging on tcc's type parsing.
And I have fixed (at least this chunk of) tcc's type parsing. :)
Finally heard back from Andrew Morton. The first email was about the old version of the arm/00-INDEX patch (from late august), but that's updated and there are now 7 queued patches in his tree that I've authored (and one I acked). Yay. I should suck the -mm tree into FWL and get that testable, but that comes after I get building under itself automated.
Added some content to the master index with more to come.
Tried to build binutils-2.17 with tcc and it hit the gcc dynamic array extension. In theory now we have alloca() (which was part of the Grischa patches that David Wheeler broke up and submitted, hg 450), this becomes relatively straightforward to implment. Unfortunately, there's a corresponding change to the behavior of sizeof(). With this program:
#includeint main(int argc, char *argv[]) { int walrus[atoi(argv[1])]; printf("%d\n", sizeof(walrus)); return 0; }
Running "./a.out 42" produced 168 as the output. I.E. the sizeof() on a dynamic array is itself dynamic, not computed at runtime. This makes a certain amount of sense, but is a large implementation change for sizeof(). And _that_ is the hard part to implement.
Ok, doing the base functionality is also funky too, since I have to check if it's a global variable (and barf then, but with a different error). Also, checking the type is nontrivial. Did you know you can feed it a long long, but if you give it a pointer it complains about non-integer type. The closest in tcc seems to be gen_assign_cast(int_type)...
Oh wow. I'd just like to say that:
int thing[42], func(char *a);
Is the _weirdest_ construction I've encountered in a while (declare global variable _and_ a function prototype on the same line. No, it's not returning an array, note the comma... But both tcc and gcc seem to accept it, so...
Still pushing 00-INDEX patches. Haven't heard back from Andrew yet on any of them...
I set up a little vpn for Fade so she could send email through her work server from my grandparents' house. This should let her hang out at coffee shops in future, too. Getting it to autostart on MacOS X turns out to be non-obvious. (Well, ok, the _right_ way to do so is nonobvious. Hook into /etc/rc, sure. Get either SystemStarter or launchd to launch a program apparently involves editing the apple version of the registry, and the only program I've found that does it for you is $5 shareware. Finding out that what you wanted to do is controlled by SystemStarter (until OS 10.4 and then launchd afterwards) was nontrivial. I thought this would be a simple gui thing "click on program, say "run it as this user on startup", but no. Clicking on the program won't even run it. "It's a python script! Don't open it in a text editor, RUN it. The executable bit is set! It starts with #!/... Try shift clicking? No, that didn't help..." Presumably I have to set some extended attribute in the "resource fork". No idea how, last time I played with that sort of thing was on OS/2...
The much hyped usability of MacOS X is actually pretty shallow. It's great as long as you never try to do anything unexpected.
Sitting down at the mac and trying to switch programs was a bit tricky too. Are there multiple desktops? How do I minimize windows? How do I get back a minimized window? Is there a list of running programs? It turns out there's this nifty functionality attached to the F9, F10, and F11 keys, but unless you KNOW that the desktop is darn opaque.
Head-scratching at tcc. The constant propogation stuff is tied to integers, and now I want to get "long long" values to resolve constants the same way. I don't want to duplicate the logic in gen_opic() in an entire second function (gen_oplc()?), but I also don't want to typecast every int to a long long and then typecast it back. Hmmm...
Ok, I taught gen_opic() to do everything with longs. The types should truncate properly (CType is a big union which should adjust for endianness) down to whatever it really is, and doing all math with the largest possible type simplifies things down to one code path. This is all compile time so shouldn't impact the binary at all, and constant optimization shouldn't be enough of a hot path to noticeably slow down the compile by using long long here. (I should benchmark that though, since speed _is_ a primary goal. But simplicity is also a primary goal. Simple, fast, correct.)
To those of you who emailed: yes I'm aware of Jonathan Corbet's video4linux documentation, and it's great stuff. I didn't say there wasn't good video4linux documentation, just that attempting complete coverage of 00-INDEX files in Documentation turns out to be nontrivial. :)
I really hate the kernel Documentation directory. It demonstrates all the failings of "documentation written by people who already know this stuff". For example, I read Documentation/SM501.txt and it never said what this chip does, how to use it, or where I might encounter one. The introduction is this:
The core driver in drivers/mfd provides common services for the drivers which manage the specific hardware blocks. These services include locking for common registers, clock control and resource management.
So I glanced through drivers/mfd/SM501.c (I don't know what "mfd" means either) and the source code seems to imply this chip has something to do with a clock (what chip doesn't?), can generate a framebuffer, and might can be used through USB. Huh?
Luckily, Kconfig is actually read by end users, not just developers, and then tend to complain if you assume you already know everything you're about to tell them and thus don't need to actually say anything. Kconfig says that mfd means "Multi-Function Device", and Kconfig also says:
The Silicon Motion SM501 multimedia companion chip is a multifunction device which may provide numerous interfaces including USB host controller USB gadget, Asyncronous Serial ports, Audio functions and a dual display video interface. The device may be connected by PCI or local bus with varying functions enabled.
That's much better. I just patched that into the Documentation version because I wasted 15 minutes trying to track this down. (It might be easier with google, but I'm on a long car trip with my laptop right now. I still don't know if this chip is regularly encountered on i386 or arm or what, but it's still an improvement.)
Now I'm looking at Documentation/video4linux, and I've made it to the bttv directory. What's a bttv? It's a kernel module, but what do the initials stand for? (Better Than Television?) This directory doesn't need to say, because the people who wrote it already knew.
I hate the Documentation directory.
We were going to hit the road for my Grandparents' house at 9 am today. This didn't happen. I'm still rushing to get my documentation stuff in today...
Got the darn mirror script written and checked in, and the OLS papers are now being split up from this A) automatically, B) using scripts that are in the make directory and in source control, C) without truncating the last paper in each volume. Hopefully, this whole mess is now reproducible by somebody other than me. (When I get it down to "check out tip from hg and run this script to reproduce the entire site", I'll let you know. Working on it...)
You know, actually integrating all the OLS papers into master.idx turns out to be darn non-obvious. (I expected this, I just didn't expect it to be quite this time consuming.)
For example, OLS 2001 had a paper on devfs. I just added a section on devfs in master.idx and wrote a couple paragraphs about how devfs is obsolete, was removed from the kernel, and replaced by sysfs and udev. So, do I link the devfs paper in there? Or do I link it into the "history of hotplug" paper that's half-finished but which I linked to as the "for more information, see..." which explains _why_ devfs was replaced... (Leanning towards the latter...)
The friday deadline is starting to loom noticeably, especially since Fade and I head out early tomorrow morning so I really need to get it done tonight.
Which of course means I'm stuck on a TCC problem. Always fun when fixing one bug reveals another, although this particular bug seems to be manifesting because my test case is accidentally using "<" when I meant to use "<<". Oops, but in a good way. I'd add this to tcctest.c except how do you put stuff into the test suite that's not supposed to compile in the first place? (This is _supposed_ to generate an error, it's just not supposed to segfault the compiler...) Heisenbug still in full effect, but an sprintf() followed by a write() isn't screwing things up as badly as dprintf() and printf() were. Avoiding stdio seems to help...
Ok, I've got the corruption nailed down to a call to gen_op(TOK_LT)...
Ok, the problem is that gen_op() is calling vswap() which is assigning to vtop[-1] when vtop hasn't got two arguments in it. What the... Follow the bouncing ball: vtop is initialized to vstack-1 at tcc.c line 8446. (That's... creepy. Ok...) unary() does a vpush_tokc(), unary() does a vpushi(32), lexpand() does a vpushi(0), lexpand() does a vpop(), lexpand() does a vdup() which does a vpushv(), lexpand() does a vpushi(240), lexpand() does a vpop(), lexpand() does another vdup() to push, and then gen_opic() pops one, gen_opl() calls gtst() which pops another, gen_opl() calls gtst() again which pops a third, gen_cast() pushes another, and then calls gen_opic() to pop it, another gen_cast() push and gen_opic() pop... ha, and then a _second_ gen_opic() pop. Stack is now empty. Followed by a gen_opl() pop, stack is now more than empty. More instructions are getting produced at the end than arguments are being fed onto the stack. But...
Wow that made my brain hurt. The fix is here..
Where the heck did the ols 2003 papers go? Oh well, archive.org still has them. I mention this because I'm finally implementing the mirror.sh script I've been meaning to do, to download the bits of the kdoc page that come from somewhere other than the mercurial archive or the kernel tarball. This is mostly to document where all this information came from, but it should also have the side effect of making it reproducible. Unfortunately, OLS continues its tradition of pointing at third party websites that go down over time. (Yes, kdocs is going to have the same problem, which is why we need a 404 checker. I'm mirroring what I can, evaluating other sites based on established longevity (lwn.net has been pretty good about staying up), and trying to make sure archive.org has most of the rest, although their database isn't 100% reliable. "Unable to connect to database...")
Taught tcc to use cur_text_segment==NULL to mean nocode_wanted, removed the old int variable and replaced it with the appropriate tests, and added a few new ones. This made the segfault go away when generating code before the first function, but now I've got a heisenbug trying to free the #define stack at the end of the program run. (When I stick in dprintf() calls in certain places, it _goes_away_ and the program exits normally. AAAAAAHHH!!!)
The bag for the Happy Meal I just bought at The Donald's (I'm 35, I'm allowed) is covered with Fun Facts, including "You can jump 6 times higher in space!" The interesting part about that is that at no point during the bag's design or production did that phrase come to the attention of anybody with both the authority to change it and the ability to understand what it actually said.
The Happy Meal also came with a toy, some kind of orange plastic weasel thing (chipmunk? Squirrel?) holding two peanuts it whacks against the side of its head when you shake it. The toy comes with a colored pencil (orange) you're supposed to stick up its' butt (really!) to assist in said shaking. The pencil says "Vietnam" on it in small black letters near the blunt end, which could mean that's where it was made, could be the name of that shade of orange (angent orange?), or perhaps it's just reminding us of the vietnam war (where some people probably did jump six times higher, if you still call it a jump when land mines are involved). We probably don't need the last reminder, since Iraq's doing it for us just fine...
Heard from my boss at the Linux Foundation today, and worked out some deliverables for Friday. I'm going to try to get the master index up to replace index.html, with all the OLS and LWN content in it, and also submit the patches to update all the 00-INDEX entries for Documentation. (The exports can wait a bit.)
I've been good and not looked at tcc today. (Working a weekend day to make up for being distracted so often over the week.)
Possibly I should do an rss feed for the 64-bit transition follow-up articles I'm writing?
Trying to get a major update of the master index up by monday. We'll see. Not much to say about it yet. (Ulrich Drepper wrote a great LWN article about memory, which reminded me to add the ars technica ram guide to the section that already had Mel Gorman's vm documentation. Exciting stuff, and yet again "more new random links added when I _should_ be going through the backlog of old links". New stuff comes in so fast I can't keep up, let alone sort through the old stuff. But I've got somewhere to PUT it all now, darn it.)
Funky tcc bug. Initializers for global variables are supposed to be constants, and if you try to initialize a global variable with something like "int a=blah();" tcc will notice and generate an error. But it generates the code to calculate the value (in this case a call to function blah) before checking whether the result was a constant, and when it generates code it outputs it to the current text section. If you haven't defined your first function yet, the current text section is NULL, so if the quoted bit above is the first line of your program, tcc segfaults trying to compile it.
The segfault happens in platform specific target code generator, in i386-gen.c the function g() which is definitely a hot path for performance. I don't want to add a test there. The problem is first spottable at tcc.c line 8431 or so, where it's confirmed it's got an initializer and calls decl_initializer_alloc(). At that point, if the current text section is NULL, there's a potential problem, but if the initializer resolves to a constant there can be all sorts of multi-stage math done here with bit shifting and parentheses and who knows what else...
Ah. There's a global variable nocode_wanted that does exactly what I want. Save the current value, set it to what I need, then restore it... Works!
That's half the problem solved; making the segfault go away. The other half is that "long long x = 1LL<<32;" isn't considered a constant initializer because long long bit shifts are loosing "const" status somewhere, hence the attempt to generate code to calculate the value. But I can worry about that later... Er, darn it. Except that case is still segfaulting, so apparently "nocode_wanted" isn't guarding that particular bit of code generation. SIGH. (This nocode_wanted stuff might also be the key to making if(0) {...} do dead code elimination. But only if it _works_...
Checked in a fix for the #include
Also, building allyesconfig of toybox (minus mdev, which still doesn't build due to hitting "tcc: undefined symbol 'strndupa'") mostly works, but if I run "./toybox help" with no other arguments it segfaults. But "./toybox help df" and such work fine. Something in the command line parsing code seems to be getting miscompiled. Hmmm...
Sigh. I didn't _mean_ to make the news today. But this thing kind of boiled over. (Hint: The proper response to receiving a cease and desist letter from a lawyer does not involve going silent for a week and not responding at all. Also, when slashdot people start asking about the license on your boards, claming that they're violating the EULA against reverse engineering by trying to figure out what GPL software it contains that you haven't given them the source code to? That's just stupid.)
(And groklaw. And Information Week. And linux.com. And the netherlands?)
I'm so glad the SFLC is handling all of this so that it's only a minor "gosh" thing at this end rather than a major intrusion. (Only two phone calls so far today. Admittedly more email...)
Hmmm... Turns out there's only a little over 10,000 exported symbols in 2.6.23 because some of them are per-arch duplicates. Makes the javadoc approach a bit funkier, but I'll work something out.
If I want stunnel support (https as a filter) it seems my best bet it combining xyssl with xrelayd. I'll have to try this.
For a while now I've been pondering changing the Firmware build so it doesn't have to untar and then rm -rf each tarball each time it builds it. (If nothing else this is redundant between the cross-compiler and mini-native stages, and when building lots of different targets it's a big overhead.) This involves teaching packages to build out of tree, which is something not all of them do.
However, an old message I got from Dennis Vlasenko pointed out that if you do "cp -sR" on a source tree, you create a bunch of symlinks to the files and can then build in-tree among the symlinks, all without ever actually copying the files. This is a nice hybrid approach, and I'd like to try it.
This raises a lot of questions: what's the best point in the scripts to create the extracted source tree? Where should the extracted source be kept under "sources" or under "build"? (Is the extracted source a temp file or not?) How do I keep it all in sync when upgrading to a new version of a source tarball?
I'm leaning towards putting the extracted source under "build", both because it can be recreated from the source tarballs and because it's several hundred megabytes. That says to me that download.sh is the wrong place to extract this, but doing it in setupfor opens up race conditions with parallel builds (ala ./forkbomb.sh --fork). Right now download.sh has such a race already (creating/removing sources/build-links). I could add a new "extract" stage after download.sh, but I'd rather not...
Sigh. This is way more complicated than there's any excuse for...
Fixed arm for 2.6.23. The new guard symbols that showed up were CONFIG_BLK_DEV and CONFIG_SCSI_LOWLEVEL, neither of which contain _any_ useful information, they just act as visibility guards for the menus they're in. Logically, an attempt to set a symbol inside such a menu should set the menu symbol as well, but the menuconfig infrastructure isn't doing that. I keep wanting to write my own, but the logical language to do it in is Python and I don't want to introduce that as a build dependency for anything. Sigh...
The fsync() vim is doing can hang the program for upwards of 45 seconds when the system is heavily loaded, and since it's based on number of keypresses, it does it when I'm _cursoring around_ and not actually changing anything. That's just... Sigh.
Amazingly enough, uClibc development appears to have gotten itself unstuck, at least in terms of commits to the main development repository. This is good, because Peter Mazinger's been so paranoid about his fork that not only won't he let me publicly put the tree I got online for anyone else to use, but he won't even tell me _when_ I might be able to do so. Strangely, his "or else" case is that if I give the tree out, he won't put his copy up for me to pull updates from anymore... Which he hasn't done in over a week anyway. Oh well, it wouldn't be polite to distribute his changes without his consent, but it's not useful for me to mess with his tree under these conditions either. I might as well follow the mainline tree for the time being and see what I can do to help out there...
The tvtropes website is highly addictive.
Went to Epoch with Fade today, then Jimmy John's. Re-read more Tamora Pierce books (up to "Emperor Mage"). Accomplished very little of interest.
Another day at the chick-fil-a with wireless internet in extreme south Austin. It's been pointed out to me that the strip club across the street (Palazio, never been there) also has wireless. No idea what kind of work environment that would make, and it's unlikely to be an economical option...
Went through the historical kernel repository and got export counts back to 2.6.0, plus lists of symbols that were added and symbols that were removed from each kernel. Not yet doing anything _useful_ with this information, but it's interesting to note that 2.6.0 only exported 6000 symbols and 2.6.12 exports twice that. Can we say "API bloat"?
My desktop has gotten itself confused again. Something is deeply wrong with the scheduler, to the pointttttttttttttttttttttttttttttttttt where ((there it goes again) half-second dropouts happen where X11 freezes, and when it comes back it's either dropped keys I typed during the freeze or decided the key was down and and repeated it. Most of them I'm correcting bbby hand, but I'm getting kind of tired ooof it. (I thought it was keybounce when it started doing it last night, but A) my keyboard isn't that dirty, B) it's not specific keys, C) this morning the pauses were noticeable. (At first I blamed vi's gratuitous sync feature...)
I tried renicing X11 to -10, and that seems to have resulted in the long stretches like the above. I vaguely suspect software suspend got it into this state, but that's just a guess. Top suggests the IPW3945 userspace daemon might have something to do with it (a few times after the half-second hang it showed up at the top of the CPU usage list for a tick).
Huh, it seems to have calmed down again, for the moment...
Heh. Linus doesn't seem to like C++. Especially this bit. Gee, where have I heard this before...
Yup, it's the userspace helper for the ipw3945 driver. It's running when the system freezes. Sigh. I'm not sure a reboot would help, it could just be the wireless access point I'm connected to is triggering bad behavior in the stupid closed-source driver...
My laptop's touch pad has buttons right under it, yet it also has tap to click enabled, which is evil and stupid and causes no end of grief for me. The way to turn off tap to click is to go to the kde control center and turn it off. Except Ubuntu replaced the KDE control center with its own windows-style "system settings" thing which has no device list. I can access mouse settings, but that widget doesn't know what kind of mouse I have (touchpad? trackball? It's all the same...)
Why does every distro out there keep thinking it must chip off bits of KDE and replace it either with inferior reimplementations or bits of gnome (ok, that's redundant)?
Got the rest of the seletal master.idx converted to the new format, and fixed the conversion script to actually get all the data output in the second pass (the contents attached to _other_ tags are important too :).
Wrote a quick script to iterate through the tags in the kernel mercurial repository and dump the exports from the release kernels. How many are there? Here's what wc says, which probably isn't quite accurate yet but is fairly close:
Yup, there are over 12,000 symbols exported in the current kernel, up from 9,500 back around 2.6.12. Right now I'm mashing GPL and non-GPL exports together into one big alphabetically sorted list so in the morning I can work out change figures and what symbols showed up when, but later on to document them I'll need to record what symbols occur in which files, in which order, which are GPLONLY, etc. Then I need to actually look up each symbol and document what it's for, one at a time. (This all by itself could take a year to churn through, but hey...)
Remember the large culinary explosion I mentioned a couple weeks back? Well mark finally got a few seconds of the video up. We were trying to do a "will it blend: liquid nitrogen ice cream". Apparently, the answer is "no: it will not blend".
Grrr. Software suspend failed again, and as always I lost track of the 15 in-progress things I had going. Let's see, what do I remember:
Day job stuff: Writing an email to Markus Rex (CTO of the Linux Foundation and apparently my new boss; good to know). Indexing the OLS papers (breaking down and just putting a _name_ to all of them, in the short term; descripts can come later). Converting the skeleton master.idx file to the new format. Collating my todoc.txt files and bookmarks and such to start shoehorning into said master.idx, and that includes the current index.html content so I can . Overcoming the code page bug in xmlto via bonking the output on the head with sed (done, and checked in I think). Working on a reproduction script that recreates the entire documentation directory with A) the mercurial repository, B) wget from the primary sources, C) running the build scripts in the right order. Indexing the Linux syscalls and ioctls. Doing a /proc safari for things like the "Using /proc to control the OOM killer" (oom_score and oom_adj and /proc/sys/vm/panic_on_oom)...
There are a dozen more things here but I guess I'll remember what they are as I bump into them.
Non-day-job stuff. Testing Peter Mazinger's uClibc (including figuring
out what's up with _syscall6(). Evaluating tcc's ability to build various
things (toybox allyesconfig, the linux kernel's asm-offsets.c, anything
that does #include
There was more, but what was it? Sigh.
Oh well, off to finish writing that email. (Which is about my new boss giving me new todo items like documenting all the EXPORT_SYMBOL symbols. A largish todo item is summarizing my exisiting todo lists for him, and telling him what I've already done. It would help if I knew. :)
On an unrelated note, I wonder how many cancers originate in stem cells failing to specialize properly? Stem cells express telomerase already, and the infrastructure to be mobile (metasticize) and form any kind of cell. Stem cells have to _lose_ capabilities to become normal cells, whereas an already specialized cell has to regain capabilities it lost to become a cancer cell. Some cells (like red blood cells, which lose their nucleus entirely) obviously can't do this. I wonder how many of the other types of cells drop infrastructure they can't easily get back?
On the other hand, you can usually torture a tissue sample (chemically, with radiation, etc.) to get cancer cells out of it. (But do those tissue samples contain stem cells?) Probably multiple mechanisms... (Don't mind me, I used to be a biology major. I have a number of strange interests I don't have time to pursue...)
My old xml validator from 2003 was an absolute piece of garbage (what was I _thinking_? Probably that it was a quick hack never meant to be reused, and at the time I had been away from Python for a while and was rusty. You don't iterate by character in python, you read the whole file into a single string, call .split("<") to find the start of tags, iterate through that list and split(">",1) on each of those chunks to find the end of the tag (and the data immediately attached to that tag), and then process that. Needs lots of memory to work on big files but most files are small and it's simple and fast. The 404 checker I wrote for the Documentation directory works like this, for example.)
Anyway, threw out the code I wasted an embarassingly large chunk of last week on and wrote a working replacement in a couple hours. And it is now generating an index and inserting section headings into the data. (In two passes, which is fine. See "simple", above.) It's not numbering the sections yet but I think I'll wait until I stop reorganizing them before having it spit out numbers that are just going to change in the short term. See make/indexsections.py in the kdocs repository for the script. Right now the generated index is at master.html on my kernel documentation page, but I'll probably replace the main index.html with this once I've got the existing content switched over. Maybe tomorrow.
A project I've been meaning to do for a while is a multithreaded bunzip2 implementation. The algorithm is trivially parallelizable if you take it a block (900k) at a time, and on a multi-core CPU it could go twice as fast decompressing. Starting from the implementation I did for busybox, it's probably only a weekend's work (a largeish chunk of which would be figuring out how to do multithreaded stuff under Linux: I did a lot on OS/2 and Java, but the posix threading API is just demented).
As with so many other things, I just haven't had time...
And gcc broke again. I did something, and suddenly it's not finding /lib/libc.so.0 during the dynamic sanity check build. Looking at it, I don't remember how it was supposed to do this in the first place. (I despise gcc's diseased path logic, and really don't want to deal with this right now.) Despite the man page, it pays no attention to LD_LIBRARY_PATH...
Sigh. Now that I'm antibiotics for this darn sinus infection, it seems to have decided to put up a fight. I'm totally out of it, sweating, and the chicken sandwich I had for dinner tasted like hospital gloves. The _perfect_ frame of mind in which to debug fiddly contradictory code.
Ah, the uClibc-install patch wasn't getting applied, thus the libc.so linker script had absolute paths in it (it forwarded to /lib/libc.so.0 rather than libc.so.0; this is a uClibc bug they won't take upstream), and that's what was overriding it. Otherwise, -nostdlib should take all that out. I think. (Someday I should make sure that if I try -lz or some such and zlib is in the host but not the target libraries, it doesn't fall back and try to link against the host version. I _think_ I already did this test, but just to be sure...)
Earlier today Fade and I went to Fry's. We were there for cables to hook up her speakers (since the old ones appear to be packed in a box we can't find), but I wound up getting a 4 gigabyte USB key while I was there. Backing up my email and website onto it as we speak...
Today on Obscure Problem Theatre, what to do if your current directory has spaces in it. Ordinarily, when I resolve paths in FWL I use "${blah}" in quotes to keep spaces together, but wildcard expansion doesn't work so well with this. I was using "${blah}/subdir/prefix-"* in include.sh to sort of get around this, but now I need to pipe that to sort because somebody's trying to build this on MacOS X where the files aren't automatically alphebetized, and it does matter what order patches apply in.
If I go "$(echo "${blah}/subdir/prefix-"* | sort)" and it produces two output files, they get reported as one string with a space in the middle. (Yes, "$(echo "hello")" is the correct syntax for using quotes inside $() if you want). Since "echo" doesn't insert a newline between its arguments there's actually no way to distinguish with that, I need to use ls or find instead to get one entry per line. THEN the question is how to come up with a shell grammar to make "for" distinguish newlines but not spaces as breaks between arguments. I could trivially do this with xargs, but "xargs for i in" doesn't work that way.
Last time I ran into a problem like this, I fiddled around with bash arrays for a bit found out you can't call xargs on a shell function, and in general came up with a solution I don't remember. Sigh...
I could run xargs on cat and pipe that into patch, but I want it to print out the name of each patch before attempting to apply it, so you can see which patch fails when something goes wrong.
Ah, I remembered: ls "${SOURCES}/patches/$1"* | sort | while read i; do blah blah blah...; done
Where do I find documentation on ioctls? The man pages have syscall documentation (for some of them, man page section 2 is primarily syscalls), and I can get a list of the ones uClibc implements from the uClibc source if I want to worry about order and coverage (which I do, but it's not at the top of the stack right now, that falls under "indexing" and thus behind the master index). But where do I find ioctls?
This came up in a conversation I'm having with the guy who wrote this. Since I'm writing my own mke2fs in toybox, I can improve the behavior relative to RAID that he had problems with. It turns out his main problem is that because the size of the block groups is a power of 2 (which is unavoidable due to the design of ext2), then the inode tables of every group wind up on the same disk of the RAID, which becomes a performance bottleneck. I can vary the starting locations of the inode tables (and the block and inode allocation bitmaps) since those are all in the group descriptor tables which only change at filesystem creation time and get cached during use so life is good there.
Unfortunately, in order to figure out WHERE to put them for best performance, I need to know A) the array stride, B) how many disks are in the array. There's a stride argument for mke2fs already, but none to say how many disks are in the array. He says that I can query the raid information with the GET_ARRAY_INFO ioctl. Which sounds cool, but I have no idea how to use it, and google isn't being helpful here. The man pages mention a few ioctls when they're talking about other things, but they don't seem to mention this one at all...
I guess what I need to do is go through /usr/include/linux with find, grep for "_IO.(" and then figure out what the heck all 564 hits mean. That's going to be a fun week, when I get around to it. I can start with Documentation/ioctl-number.txt but that file claims to be from 1999 and says "This table lists ioctls visible from user land for Linux/i386. It contains most drivers up to 2.3.14, but I know I am missing some." Which just fills one with confidence, no?
The todo list never seems to get _shorter_...
Note to self: Another C compiler to poke at is tendra.
Ruminated about the documenation master index on linux-doc. Then converted a couple sections of my index skeleton to <span> tags, pulled out my old xml/docbook validator (which I wrote while working on Halloween 9 in 2003) to start writing a parser for <span> tags...
Just got a bunch of tcc patches from Filip Navara. I'll have to look through that tonight...
This remains amusing: an explanation of the star wars plot that actually makes sense. And The Silmarillion in 1000 words.
I can now compile toybox with tcc, everything but df and mdev. The only changes I made to toybox were in the makefile, breaking out some gcc-isms so I can now say "make CC=tcc OPTIMIZE=" and have it build against tcc. All the rest of the fixes were to tcc, they're in my mercurial tree. Yeah, I should do another release...
Mirrored the 2004 OLS audio files (only the high definition ones). Not quite sure about the license on that yet, so I'll put 'em up after I've sorted that out. I have questions in flight to Andrew Hutton, but I haven't heard from him in a while...
Wow is the qemu build brittle. The fact it won't build with any compiler on the planet except gcc 3.x (and won't even build with gcc 4.x) is a hint, but the 386 qemu application emulation (qemu-i386) it builds segfaults immediately when linked against glibc-2.5 (which is what's in Ubuntu 7.04). It does this before main() is called, even. All the other application emulation (and system emulation) variants seem to work, just not that one.
Paul Brook on #qemu suggested building that one statically. My first guess after looking at the makefile was "make BASE_CFLAGS=-static qemu-i386". And make says:
make: *** No rule to make target `qemu-i386'. Stop.
That's right, qemu-i386 is an executable built and installed by make, but it's not a target make knows. Did I mention I hate make?
Paul insisted that the right way to build bits of qemu statically is ./configure --static, so I did that and tried to rebuild everything, which died in the linker phase. (Right, separate bug.) So I tried to rebuild just qemu-i386 and not the rest of it (since the rest builds find dynamically), but make still didn't recognize qemu-i386 as a target. Paul Brook said I should use --target-list, but when I tried it that didn't recognize qemu-i386 as a target either. At this point Paul said I was "being deliberately obtuse". It turns out that to build qemu-i386, the target is "i386-linux-user". Obviously, I should have guessed.
So: make distclean, ./configure --static --target-list=i386-linux-user and then type "make", and the build dies immediately on the very first line.
landley@dell:~/qemu/git$ make gcc-3.4 -DQEMU_TOOL -Wall -O2 -g -fno-strict-aliasing -I. -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -g -static -o qemu-img qemu-img.c cutils.c block.c block-raw.c block-cow.c block-qcow.c aes.c block-vmdk.c block-cloop.c block-dmg.c block-bochs.c block-vpc.c block-vvfat.c block-qcow2.c block-parallels.c -lz -lrt /usr/lib/gcc/i486-linux-gnu/3.4.6/../../../../lib/librt.a(aio_cancel.o): In function `aio_cancel64': (.text+0x2f): undefined reference to `pthread_mutex_lock'
And dozens of lines of further errors...
It takes _effort_ to come up with a build process this brittle.
Ok, the solution to that is "touch qemu-img" before running make. Right. I'd say it serves me right for using a development snapshot instead of the release, version, but A) the last release is incredibly out of date and there seem to be no plans for a new release, B) I reported this problem with qemu-i386 and participated in a longish email thread about it on the qemu list back in June.
Fixed some bugs in tcc, got some firmware linux targets to work with linux 2.6.23-rc5, got more OLS papers indexed, read the LWN report about the kernel summit documentation panel in Europe (which seems to have produced as little results as I expected), banged my head against the Linux build trying to understand the early boot process, poked the powerpc developers about prep support again, formented a minor insurrection on the uClibc mailing list. The usual.
Also made beef soup (we're out of those little mini-carrots but Fade found where the garlic powder was hiding). Petted cats. Read more books (rereading "Wild Magic" by Tamora Pierce). Installed little triangle pieces on the bathtub in hopes it stops the leaking. Broke the bathroom sink...
Peter Mazinger's uClibc tree is in mercurial, and has over 1000 changes relative to 0.9.29, but unfortunately it doesn't have a permanent location and the dynamic dns he pointed me at seems to have changed since he sent me email pointing at it. Sigh.
I'd meant to get the 2002 OLS papers all indexed by today. Admittedly this is before I noticed it was a three day weekend, but still. Must go work on that...
Got Fade's desktop machine set up, finally. It's switched off at the moment; she's still using her mac laptop. I'm not surprised.
As much as I like Mercurial, there are times it's just too dumb to live. When I say "hg revert download.sh" which is a file in the current directory, it has NO EXCUSE to spend forty five seconds doing a find in the "build" subdirectory which A) isn't checked into the repository, B) I have in .hgignore.
Got armv4l-soft running without any float support in the kernel, thanks to Peter Mazinger.
Why did I find out about the Linux Weather Forecast site from Computer World rather than lwn, considering Jonathan Corbet was apparently one of the people implementing it?
Note to self: I'm reading this but keeping it open puts me over the memory budget for suspend-to-disk to work.
So I'm back banging on my mke2fs for a bit, taking a break from everything else for a few hours. The user interface is _horrible_.
The problem is, my mke2fs combines gene2fs functionality, and works like an archiver. I should be able to do anything with it that I can do with "tar", and in fact once I get that implemented I'd like the two to be able to share code. (And zip, and cpio.) But stealing the tar command line interface isn't good either, because I have to be able to specify block size and inode counts and stuff like mke2fs does.
Possibly mkisofs is more what I'm looking for? That would mean the options would be more like: "mkext2fs [options] [-o filename] [path...]", which seems workable. Dealing with boot sectors isn't my problem. For partitions, you supply a starting offset (in bytes) and a length (in blocks, you can specify block size) for the filesystem.
Heh, once I get cpio implemented I plan to support the kernels initramfs generation text file format, and making mke2fs handle that then becomes trivial code reuse...
Ok, I need to do my report _today_. So what happens? Jakob Eriksson gets me started on arm big endian. Peter Manziger reminds me that I never figured out why my arm soft float userspace doesn't work if I disable floating point support in the kernel (almost certainly a uClibc configuration issue). Plus tcc and uClibc stuff: trying hard not to go down any of those ratholes right now.
Grinding away at the OLS 2002 papers so I can report that year as "done" in my monthly accomplishments. Lots of this stuff isn't hard, just time consuming...
I'm also reminded that it's really really hard to document a technology I don't like. (Well, writing a non-sarcastic description, anyway. Or listing all the reasons why doing the thing I just mentioned is a horrible idea.) Explaining how the SCSI subsystem A) doesn't use the block layer, B) seems to be the only thing that can't persistently enumerate devices... Or summarizing a paper on Linux Security Modules with anything but "In brief: avoid."
Right, I can do this. But giving bad ideas fair coverage, THAT is the hard part. (You can't be sufficiently familiar with all this technology to document it, yet not have any opinions about it. I can usually figure out where it's possible for reasonable people to have other opinions, but sometimes it's based on anecdotal evidence rather than any real understanding on my part...)
Well, I sent in my report. As always, I hit the button silently wailing "I didn't get enough DONE!", but I'm used to this and it's a half hour after midnight already (but not in Oregon yet :)...
Drove Fade to work again, and took my laptop to the Chick-fil-a on Ben White that has wireless internet. Got a decent amount of work done.
Pushed a couple patches towards linux-kernel (adding more 00-INDEX files to Documentation), and got into a discussion about cannonical patch submission format. Wrote a script to do that for me. (I suck at bureaucracy. If the reason for filling out a form is that it's going to be parsed by a script, then it should be generated by a script.)
Various tcc issues cropped up. I should cut another release. Still no sign of a 0.9.29.1 uClibc release, I really hope I don't wind up having to do that myself.
Added the kernel source README files, and a start to an index of google videos, to kdocs. Might possibly have triggered somebody breaking a chunk out of procfs (/proc/sys is a separate directory, might as well be a separate filesystem).
Two new netflix showed up, but got home too late to watch them.
So a big PowerPC conference is happening in my home town, and I can't go. Why? It's $500, with no discounts for being a student or anything. I don't get paid to work on PowerPC. This is a _hobby_. I can't justify taking $500 out of the entertainment budget for this when Fade and I could go to six flags or something for less than half that. Any Linux conference that makes it prohibitively expensive for hobbyists to attend is MISSING THE POINT ENTIRELY. Grrr.
Built FWL with 2.6.23-rc4 and not a single platform boots. Possibly the config has changed enough that miniconfig is dropping symbols, I need to dig up the better patch that fails when trying to set an unknown symbol. (Not that this necessarily helps when you set a symbol in a closed menu and menuconfig doesn't set the guard symbol for you.)
Wandered back around to menuconfig2html script and I'm trying to figure out a user interface issue again. (These are always the hard ones. Technical issues I can tell what a good answer looks like.) The problem is, if I want to break it up into individual files (to reduce the huge duplication in the 1.3 megabyte per architecture text files), where do I break them and how do I link them together? Right now I'm creating a separate output .html from each kconfig source file, which is easy to do but the result isn't necessarily grouped by menu the way menuconfig would do it.
The doctor warned me that the antihistamine pills she gave me are strong stimulants (the non-drowsy formula is, in fact, anti-drowsy), and even though it says take two a day I might want to only take the morning one or else I'd have trouble sleeping. I explained to her about the amount of caffiene I go through in a day, and she decided I should have no trouble taking both then.
I just had the second one, and it's putting me to sleep. Sigh...
Ok, figured out what to do about the menuconfig page.
I've started writing follow-up articles to the 64-bit paper. The first is about Apple's market share. I've already got a follow-up focusing on the difference between laptops and desktops.
Looking around for other compilers to test-build tcc with, I downloaded and tried to build lcc. Its makefile is crap. (Memo: something like "$Bassert" is not a synonym for "$(B)assert" if somebody has defined an environment variable named "$Bass"...) Anyway, to build it you do "make all BUILDDIR=build HOSTFILE=etc/linux.c", which doesn't seem to be documented anywhere. (Now that I've worked that out the hard way, I suspect I'll find the documentation soon. It always turns up when you no longer need it...)
Hey, there's no stddef.h in include/x86/linux. It's there in the other four platforms under include, but apparently they never bothered to test linux in this release. Wheee...
Yup, doc/install.html and it's useless. Sigh. Tried copying stddef.h from one of the other architectures into include/x86/linux (and into the build directory), tried running lcc with -I pointing it at those directories... Nothing, cpp can't find stddef.h when stdio.h includes it. Sigh. (I note, in passing, that both of the main authors of this puppy now work at Microsoft. No wonder the project doesn't seem to have amounted to much...)
In today's link trawl, I think it's the word "more" in the title of http://nielsenhayden.com/makinglight/archives/009329.html>More Republican Gay Bathroom Sex that offically ushers in fark-level "Hilarity ensues".
It's interesting to see a mainstream semiconductor company expanding into the energy business by selling solar panels. Insert your own "peak production" reference here, although the main reason for the increas in price since 2000 seems to be China going from importing essential no oil to becoming the world's second largest importer.
Could Moore's Law apply to lowering the costs of solar panel fabrication? I don't know. You'd need to stick a big lens in front of each cell to focus light on a smaller area, but that's cheap and if the efficiency of the cell is the same at a smaller size... Or perhaps it's something to do with old fabs that aren't up to belting out wafers at a modern resolution? Interesting to watch.
Milton Miller is making progress getting a ppc kernel booting under qemu, this time replacing Open Hackware with some code he built from the kernel source. Hopefully around 2.6.24 I'll be able to add powerpc to the list of supported FWL platforms and cut a new release.
The address of the place that might have records of the contact info for my condo association is Travis County Courthouse, 1000 Guadalupe St. #220, Austin, TX 78701. This is of no possible interest to anybody else, but I need to remember it...
This is highly addictive. No idea why it won't run under Konqueror, but it runs fine under Firefox...
Still out of it. Meant to collate the help stuff into a report, but wound up banging on toybox instead. (Why does that take less brainpower than doing documentation work? Huh, I'd have to think about that for a bit. I think it's mostly that documentation involves constantly learning new stuff (reading what other people have done and trying to edit it together into a coherent english explanation), and doing toybox involves taking concepts I already know and simply applying them to new situations. There's a certain amount of learning involved, but it's generated internally. Not constructing new mental models out of whole cloth based on trying to figure out how other people think. Which is tiring, and I start tired.)
The fact the humidity in Austin has been something like 110% all weekend doesn't help. I thought I'd left this behind in Pittsburgh. Poor George, her fur is matting again...
So one downside of using the Kconfig files as the master documentation from which to generate man page equivalents is that not all commands have config entries. The top level "toybox" multiplexer isn't selectable, and if you have toysh enabled things like cd and exit aren't optional. Ideally, I'd like to put invisible entries in Kconfig with help text appended to them, and not generate superfluous config symbols (USE_TOYBOX() and such) from that. I can do the first part, doing the second would require making the sed invocation that generates gen_config.h trickier than I'm comfortable with. I suppose for the moment I can just live with the extra symbols, and maybe clean it up later when I'm feeling less headachey...
I are sick. Spent most of the day sleeping. If I'm not better by monday, I should go see a doctor or something...
Went over to Mark's house. Participated in a large culinary explosion (and about two hours of cleanup afterwards) involving an industrial blender, liquid nitrogen, and the ingredients for mango ice cream, and everything else within a five foot radius.
But at least we caught it on video. And how often do you get to say that? Need to edit it down before posting, the file's a gigabyte and a half...
(Apparently, this, which Mark did less than an hour before we arrived, gave Mark a false sense of security. When you add the ice cream ingredients to the liquid nitrogen and turn on the blender, the human hand is not strong enough to keep the lid on the thing. I wound up taking what was left onto the back porch and making ice cream the traditional way, pouring LN2 into a bowl and stirring with a wooden spoon. It was good.)
Heading towards the end of the month, I should start closing down documentation things and trying to ship 'em. (The down side of working on dozens of things at once is very little visible progress gets made on any of them in short periods of time...) Still distracted by powerpc stuff, of course, but it turns out there's stuff to document over there too...
Sigh. The mips kernel build treats warnings as errors, so even attempting to compile an -rc kernel for that platform is... Kind of silly, actually. (An unused variable has been detected! Break the build!)
Still fighting with PowerPC. I stopped working on it for a bit because I got the impression it required an external utility (dtc) in order to build a kernel, which would put it in the "too broken for words" camp.
The point of the Firmware Linux project is to build the same set of packages for as many target environments as possible. This set of packages (the linux kernel, busybox, uClibc, gcc, binutils, make, and bash) constitutes the minimal self-bootstrapping development environment, I.E. the result is capable of recompiling itself under itself. This works well enough on mips, arm, x86, and x86_64 to build "hello world", and at least finishes building on sparc, and doesn't show signs of needing external utilities for m68k or blackfin.
If PowerPC (and only PowerPC) would require the addition of an eighth package (dtc), the PowerPC is "special". Special Olympics, Very Special Episode, Dana Carvey doing Church Lady "well isn't that special" style Special. This is unacceptable. I didn't even use "make headers_install" until the Linux kernel included its own copy of the "unifdef" utility.
Luckily, Milton Miller (mdm on irc) showed me how to build without that, and now I've drilled my way through arch/powerpc/boot/wrapper, fixed one more thing in a Makefile, and am currently trying to figure out why mkprep is segfaulting. But first: Dinner.
It was segfaulting because two things reading the elf header weren't translating endianness, and I'm cross compiling from a little endian machine to a big endian machine. Fixed it, and it built. Checked in, and now building from the checkout.
The build is going slow because my laptop is stuck at 800 mhz. Apparently, if you software suspend with the CPU clocked down for power saving reasons (ubuntu does this automatically) on resume the slower speed is the maximum speed the system can clock up to. (Oh yeah, we're a desktop OS. All the way.)
Walked to Epoch for the exercise. Had a chai, used the wireless there for a couple hours, walked back. To say my feet hurt would be an understatment, I think I gave myself blisters again. (Last time I did this on a long walk around Austin was 2000. I have been out of town for a while, haven't I?)
Added the old 2.4 Linux Kernel Internals document to the list of things I need to mirror, and started a mirror script to reproduce grabbing them all from their original URLs to populate a "mirror" directory under kdocs. I need to add the OLS originals into there, meaning I need symlinks from where they are now (under OLS in the individual year directories) into the mirror directory (which is currently flat). Then I need to make the mirror script reproduce splitting them up into individual files.
I've also started investigating what's involved in building an "allnoconfig" kernel from the command line, without using the makefile. In theory it's a few symlinks, a big gcc invocation, and some postprocessing. In practice, allnoconfig creates a 1.2 megabyte vmlinux image on x86 after compiling dozens of files, and stripping it down to something I can boot under qemu that just says "hello world" is probably the first step. (Then I can both document it as the start of a 2.6 LKI document and get tcc to compile that and go from there recreating tccboot. :) Two words that make all this hard: "linker scripts"...
Heard about a patch to make ARCH=powerpc build a prep kernel, tracked it down, and now I'm debugging it. (Well of _course_ it applies to some random git snapshot that's neither 2.6.22 nor 2.6.23-rc3 and requires some patches to be tracked down out of git applied before it will apply, and then I waste an hour or so figuring out "sure it builds a CHRP kernel just fine with the patch applied, but what I'm _trying_ to do is build a PREP kernel") and then some fixup patches worked out afterwards to make it compile. And this is _after_ talking to the author on IRC to get a more up to date version of the patch. Oh, and I had to fix usermode linux because that didn't build in 2.6.23-rc3 either but that was a simple missing #include. Oh well, it's progress...)
The workaround for the "scroll tabs left" button not working in Konqueror in Ubuntu 7.04 was to use the mouse scroll wheel when the mouse was over the tab bar. Alas, my laptop's built-in mouse doesn't have a scroll wheel...
Got up at 6:30 to take Dragon to the vetrinarian to be "edited for content". Got very little else done today (other than lurching around with arms outstretched chanting "Brains! Ikea! Brains!" My sleep schedule is all wonky now. Napping...
[much later] I've been doing the help system in toybox all wrong. Mostly I manage to avoid letting myself be influenced by "how busybox did it", but in this case I'd been distracted from toybox long enough that I just fell back into old habits and went down a blind alley for a bit (adding help entries to the array of command structures, I.E. polluting toylist.h with help information). That's wrong, it bloats the toy_list structure and requires updates to more files to add a new command. The only place that needs access to help text is help.c.
What I should do is generate block of null-terminated help entry text, with the entries in the same order as the command structure array. I can use python to generate a bunch of #define commandname_help "blah blah blah" entries in their own .h file (and ship that file with toybox releases so the build environment doesn't require python), and then define a NEWTOY macro to spit them out in the right order in help.c. Then get the index of a command via something like (toy_list-toy.which)/sizeof(struct toy_list).
So my google alert on my last name is now pulling up some "how to avoid foreclosure" column by a "Lisa Landley" which is either my father's new wife, or somebody who's taken it as a pen name. Sigh.
It also pulled up a porn site on gitsakichan.blogspot.com that has a droid post to the actual blog part, which apparently grabbed a chunk of the text of The Art of Unix Programming teacher's guide for one of the posts. So naturally I tried to report this to blogspot abuse. Blogspot links me to blogger.com, and that is now half Google. They don't seem to have any kind of abuse mechanism, and to post to the help group you need a blogger account. So Google continues to host a porn site. Oh well, it's their bandwidth...
It's also still noticing the CVS web interface from the busybox website, reporting to me pages that last changed 12 months ago. This Google alerts feature is _really_useful_, eh?
Grelber's back up. (You wouldn't be seeing this if it wasn't.)
After reading this lwn article which mentions documentation stuff going on at the kernel developer summit, I was pondering seeing if this travel fund (which I heard about back in may) would be willing to send me to Cambridge. Unfortunately, the summit's website says "Participation in the summit is by invitation only".
Nuts to your white mice.
Yay, I have internet access at home again. (We broke down and bought a low-end linksys at Office Despot when we went there to ship the old cable modem back to pittsburgh via the prepaid UPS label they sent us.)
Fade told me how to do hover text with the "title=" option to <span> or <a>. (It's nice to have a wife that webmasters. Or possibly web-mistresses.) George Boudreau also emailed me a way to do it that doesn't have the three second pause, but uses 65 lines of stylesheet, which seems a bit excessive.
Unfortunately, using either method A) looks awful due to arbitrary wordwrapping, B) can't easily be cut and pasted, which kind of defeats the purpose of code snippets. Hmmm.
And of course now that I have net access without going out for it, grelber is down (and with it landley.net and my email server). It was up a little while ago for me to check my mail, but now it's down again. I wonder if this has anything to do with the video card that Eric's been trying to replace for the past couple days? (The video card fan in his main workstation stopped working, and the cards he replaced it with apparently have a sucky driver such that Mozilla crashes X when doing heavy compositing, which is essentially "all the time". He's tried two different radeon cards, but they use the same driver. I hope he's not looting grelber for parts...)
Got up at 8 am, for the second day in a row. I am sooooooo not a morning person. It's 10:30 am already and I'm going "Hmmm, can I fall asleep here? How about over there?"
At a wireless coffee shop and used bookstore in Buda called "the coffee nut", which Steve Jackson recommended. It's nice. Now that Austin's turning into an evil republican controlled totally paved monstrosity surrounded by toll roads, we're pondering getting a place outside of town to the south or east, and availability of wireless coffee shops is a definite criteria. So far I've learned that hot chocolate with "blood orange" syrup curdles, but mint syrup still works fine. (And strawberry apparently tastes stale everywhere.)
Somebody's popped out of the woodwork to make toybox work with klibc. I haven't touched the project in two months, I feel guilty. I should finish mke2fs and check the low-hanging-fruit list. (It needs help infrastructure badly, I should probably do that first.)
Taught my little menuconfig converty script to output each input file to a separate output file. (It's neither linking them together nor indexing them yet.) Running this on all $ARCH top level Kconfig files (for i in `find "arch" -maxdepth 2 -name "Kconfig"`; do WHAM=$(echo $i | sed -r 's@.*/(.*)/.*@\1@'); mkdir -p "out/$WHAM"; ~/www/kdocs/make/menuconfig2html.py $i "out/$WHAM"; done) created 6629 output files, but only 429 unique md5sums (find out -type f | xargs md5sum | sort -k1,1 | wc), all of which are unique filenames (replace wc in the previous with: sed 's@.*/@@' | sort -u | wc), which is about what I expected. This shows the bloat inherent in the "one big file per arch" approach. What it doesn't show is how to index and display the unified data sanely, especially if kernel.org isn't big on server-side includes.
I wonder if you can do html "hover" tooltips on text rather than an image? The above paragraph would benefit from such.
Took a stab at updating my svn2hg script to be able to update an existing repository when you run it again, and found out that it doesn't work on the uClibc repository. The BusyBox and uClibc repositories are all screwed up, because there are something like 5 projects in the same repository (the other three are buildroot, tinylogin, uClibc++). The way svn handles tags is _insane_, so in order to get a list of tags corresponding to svn versions, you seem to have to do something like:
for i in `svn list svn://uclibc.org/tags` do echo -n $i svn info svn://uclibc.org/tags/$i | sed -n 's/Last Changed Rev: //p' done
Which A) is just nuts, B) doesn't tell you which tags belong with which projects in the repository. (The projects are organized by subdirectory.) Apparently, the way you see which tags affect the project you're interested in is to do an "svn list svn://uclibc.org/tags/$TAGNAME" and see if the subdirectory it shows you is the name of the project you're interested in.
This is all very uClibc/busybox/buildroot specific, and I'm wondering if there's a way to write this that's generic to any svn project. On the other hand, my only interest in svn projects is converting them to something sane like mercurial, so...
You can't handle "source" lines in Kconfig via a preprocessor approach, you have to parse the full syntax. Why? net/ipv6/Kconfig line 206, of course, which is a line of help text beginning with "source prefix". I suppose I could add a check that the line only has two words, but that feels brittle...
[time passes] Ok, try here, made by this.
Fun quote: Every single organization I've ever been part of that got involved with SAP spent years throwing money out the window in return for absolutely no benefit over when they started. I don't know what the S and the A stand for, but the P is obviously Ponzi.
Reese cropped back up yesterday, stopped by, and tried to introduce me to some other 24 hour coffee shops but we went to four and the latest any of them was open was 2am.
Upgraded toys/miniconfig.sh in firmware to be more forgiving of inputs and look better while running. It's still a gross hack.
Yes, Austin really does have wireless laundromats.
It turns out that gcc 4.1.2 has blackfin support, although the tuple is weird. The prefix is "bfin", but you can't append "-unknown-linux" onto that like you can with every other platform I've tried so far because blackfin is "special". Google suggested "bfin-elf", which at least compiled. The kernel's ARCH value is "blackfin". I'll probably have to grab an svn snapshot of uClibc to test that out. I should probably go back to poking at m68k first.
My Documentation 404 checker isn't filtering out the "index.html" files I generates. Oops. Easy enough to fix...
Took Fade to see the arboretum (it's a sort of open air mall near us with a Barnes and Noble in it). Much bookage. Reese cropped back up, stopped by, and tried to introduce me to some other 24 hour coffee shops but we went to four and the latest any of them was open was 2am.
Ok, got the Documentation 404 checker written, and sent a patch upstream for a glitch in Documentation/powerpc/00-INDEX that was confusing the translator. What's left in the first three months todo list that's still vaguely doable... Hmmm, need to add LXR (Linux Cross Reference) to the sources list, although I'm not sure how much of an improvement it is over linking to the mercurial web interface for the kernel.
Converting the menuconfig help to html, that's something I've meant to do for a while, might as well sit down and write a python script to do that now. Except it's not quite trivial to get one kconfig menu that incorporates all the architectures, because scripts/kconfig/Makefile feeds arch/$(ARCH)/Kconfig in as the starting point. Hmmm. "find arch -maxdepth 2 -name Kconfig | wc" says all 25 of the suckers have a Kconfig. Wheee...
And now you know why _this_ one is a time sink, and we haven't even gotten to "help" and "---help---" being two keywords that do the same thing, and whether or not indentation is _required_ for all lines of config entries after the first (it is for help text; and people complain about python doing this).
(Hours later...) The Gelato place on Guadalupe (across the street from the Dobie Mall, just south of it) has free wireless internet. And a very, very, very unhealthy seeming semi-frozen chocolate substance. Not quite "Chocolate, Chocolate, Chocolate, and Oreo", but making a valiant attempt.
I also looked into Sprint's new Wireless Plan Thingy. I can get a PCMCIA card (or USB, or some thin PCMCIA replacement thing I hadn't seen before), and then get an unlimited data plan (with national roaming). It's tempting. If it wasn't $60/month on top of what I'm paying now, I'd have jumped at it. Alas, it would be another $60/month if I wanted to get a second one of these cards for Fade's laptop; phones don't work that way but apparently internet does. Sigh. Still, it's a linux box so I can plug hers into mine with cat5 (sounds kinky) and it automatically reverses the polarity of the neutron flow because it's gigabit ethernet. Linux _invented_ connection sharing (well, techncally I think it was Pauline Middelink, not Linux itself), back when it was called IP masquerading.
I got paid! Woot! Now I feel even guiltier about not getting them my full monthly report in on time. Gotta go do that.
But first, finishing the menuconfig help -> html conversion script.
McDonalds, like Starbucks, has a distinct lack of wireless internet. Starbucks' brand of lack-of-internet is T-mobile, and McDonalds actually has two different brands of lack-of-internet: AT&T and Wayport. (I thought Wayport had gone out of business, but I guess that was one of the others.)
Wayport is every bit as classy as a pay toilet. I'll happily pay for a beverage or food to sit down and use the internet. I wouldn't even mind having to enter a code on the back of your receipt to get a couple free hours of internet access (which expires and everything, I can go buy a refill). But if they asked me to put quarters into the light above the booth in order to be able to read a book, I'd laugh in their face, and I will NOT pay for internet access on the same terms. A "small business" cable modem _with_a_static_IP_ is less than $100/month, and low end data-only plans are less than that. A router is a one time expense (again under $100); they probably pay more than $100/month to keep the place this size air conditioned (with this many windows and that much heat coming off the grills in the kitchen). They don't have a vending machine interface to get customers to individually pay for the air conditioning.
Fade is off at SJ Games, showing up to the office in person since we've moved close enough she can do that from time to time (and it's playtest night). Meaning when I get home I can plug into the cable modem (which will only talk to one machine at a time, you have to power cycle it to get it to believe a new mac address is at the other end of the cat5 cable).
I started my documentation fellowship a dozen days into the month of May, and it looks like I'll have my first quarterly report done about a dozen days into August. I had caught up at one point, but then OLS, the trip to Fade's California releatives, and the move all happened. Oh well. I have no idea if they'll renew me for the second quarter or not. We'll see.
Working on a link checker for kernel's Documentation/ dir, so I can see what the index.html files derived from 00-INDEX don't link to, and add them. And I'm still struggling with the markup I want to use to generate master.html (the big master index of the world's kernel documentation). I REALLY want to have something to demonstrate for the Linux Foundation before I ask them for more money, but it's hard to rush design issues. (Implementation is mostly just grinding away until it's done, but design you have to THINK.)
Fallout from the move continues. We haven't unearthed the wireless router yet and feel silly buying a new one when we've _got_ one here somewhere (we just haven't found which box it's in yet). Today I checked the net from Ikea, which has free wireless. (It has an html intercept login screen, but doesn't require a user account and works fine once you press the "use the internet" button.)
Yay, Andrew Morton accepted my "make headers_install" documentation patch.
Today's accomplishment was a little script to turn Documentation/00-INDEX and friends into index.html files, which is less useful than you'd think because 00-INDEX often doesn't list everything in its directory (see Documentation/arm for an example), and if you have an index.html file in the directory you can't list the directory's contents.
Went down and got the condo on the market. Mark finally got the last of his stuff out yesterday (I helped him lug stuff several times over the weekend), and I went back down the street to the realtor who'd sold me the thing back in 2003 and had her put it back on the market. (This makes the paperwork so much easier, she still has the original file.) She wants to put it on the market for around 50% more than I bought it for, and who am I to argue? That would just about double the equity I have in the thing, although it would get dumped right back into the "real house with yard" Fade wants to buy.
I've been posting work related things to linux-doc as I get them into semi-presentable shape. I've needed to be more open about this documentation stuff for a while, and sending my report _just_ to the Linux Foundation doesn't accomplish that goal.
I ran out of easy things to summarize a week ago, and now I'm to the half-finished stuff. It's hard to summarize something that isn't finished because when I sit down to think through what's left to do, I tend to get distracted into working on it. But the _reason_ these things aren't finished yet is they're generally huge time sinks, some of them multi-month jobs.
Sigh.
Nothing particularly computer related today. Unpacked a box labeled "good books" and read several of them (Discworld, Tamora Pierce, etc). Reading a mention of custard pie in Men at Arms (when the watch was visting the Fools' guild) I was inspired to _make_ custard pie. We have eggs, milk, sugar, a dozen little pre-done tiny pie crusts we brought with us from pittsburgh (which we usually buy to put chocolate pudding in), and the Big Green Cookbook Of Doom which I inherited from my mother. This cookbook has a title something like "the ladies home journal new home cookbook" and copyright dates ranging from 1920something to 1960something. It discusses bread making, various things to do with egg whites (and how to obtain them from eggs), and things like double boilers and carmelizing. It has a "game" section with recipes for possum and squirrel. This sucker is huge, and thorough, and the custard pie recipe is on page 500 something.
I finally managed to unearth the nutmeg a bit late (and it looked kind of granular rather than a powder you'd sprinkle; no idea how old it is), but the custard pielets turned out pretty well without it. Between that and the cookies Fade made yesterday, we're easing back into "life with a working oven" again.
Oh, and I finally dug up a pot to make tea with. Life is good.
I hate git, I hate git, I hate git, I hate git, I hate git...
I tell it to move directories. So it copies all the files out of them, but leaves the empty directories. I then go "git rmdir directory", which it hasn't got, it refuses to do "git rm directory' without -r, and when I feed it -r it lists the files I just told it to move out of the directory as deleted.
This is related to pushing the "collect together all architecture directories in Documentation/ into Documentation/arch" patch, by the way.
Metro is closed until sometime in september. Boo hiss. (It'll probably go out of business soon, which is sad.) So I've spent the day hanging out at the Schlotzky's on Guadalupe, which has free wireless (and has gotten rid of the annoying login screen even), and even free soda refills although I've been here long enough to feel guilty about that (and buy six cookies, a bowl of clam chowder, and half a sandwitch, so I don't feel _too_ guilty).
I gave up on trying to summarize this last month's documentation work for the Linux Foundation in one go and started posting individual topic summaries to the linux-doc mailing list instead. (It's good to be more open about this anyway, and I've meant to involve that list in what I'm doing...) I'm currently trying to summarize the man-pages html conversion but I'm suffering from the usual problem that when I try to document something half-finished I get to the part about what's wrong with it, and then it seems easier to just spend five minutes FIXING whatever it is than explain what needs to be done, and it's never five minutes...
Sword Camp has now noticed I'm not coming. I still feel horrible about that, but buying plane tickets was going to wait until I got the email with the exact dates, and I didn't get it before packing to move, and now that I've unpacked enough to go over my todo list it's too late to buy tickets at anything like a reasonable price. Plus after the stack of TODO items from OLS, detoxing on caffiene for the first half of last month so I worked at half speed, spending a week in california visiting Fade's relatives who couldn't make it to the wedding, two weekends at Eric's place trying to get more todo items done, packing for the move to Austin and them driving down here with a deeply unwieldy truck and four cats and unpacking everything again... Well, I'm far enough behind on work that I can't afford the time off, and I'm not quite up to driving halfway across the country again just now.
Luckily, next year we shouldn't be moving a few days before sword camp.
The cable modem installer is here. The cable modem's plugged in, and when I plug my laptop into it and run dhcp I get a 192.168.* address (sigh), but no nameservers. The installer doesn't know what dhcp is. He doesn't know what a nameserver is. He keeps saying "I'm just an installer"...
Phoned Eric (who's on a plane to sword camp in michigan), and he didn't remember the IP address of the machine in his basement either (it changed in december when he got FIOS, I have the old one memorized but not the new one), but he gave me Jay Maynard's phone number because he'd know (and he runs another nameserver anyway), except I must have typoed it because the number is not in service, and I can't call Eric back because the plane took off. Right.
Ah, I have the IP address in my backup copy of the zone file, of course. And it's not routing packets. And come to think of it, 192.168.* sounds like a factory default. And the cable modem's "ready" light isn't on, although the installer didn't notice this until just now. (No, rebooting my Ubuntu laptop is _not_ going to fix this problem. Honest and truly it isn't. The macintosh Fade plugged in couldn't talk to it either, and it's giving me a dhcp lease time of 13 seconds which is clearly not a sane value, that's a factory default, it's not getting upstream configuration from Time Warner.)
The fix (which I did) turns out to be to power cycle the cable modem. The installer kept having the central offic "hit" the modem because it wasn't "catching" it, whatever "it" was. (Presumably this was re-flashing it with configuration info from upstream, possibly telling it to tftp a new kernel? Makes a very loud beep, whatever it is.) Apparently, this didn't take effect until the cable modem was rebooted, which required unplugging it. The "ready" light came on, it started handing out a 10.x.x.x address (oh yes, much better, thank you), and started routing packets.
How do normal people ever use this company's services? Luck, I suppose.
It's now after noon, but hopefully I can finally get going on the darn Linux Foundation documentation report. (Although it would be really nice to find my linksys because this ethernet cable is 3 feet long and the cable modem is passing through broadcast traffic to my laptop.
Found the new 24 hour coffee shop somebody told me about in my livejournal a few months back. Spent a couple very nice hours online for the first time in several days, started the Linux Foundation report, lost track of time, and had to race off to deal with moving stuff again.
Took the truck over to my condo, spent an hour filling it with Mark's stuff, drove to his new apartment, unloaded it, drove it back downtown in rush hour, found the place to return it on the third attempt, filled up my backpack with 20 pounds of stuff from the cab of the truck that I'd forgotten to unload, walked home (the first two hours of which the sun was still up, and it turns out trying to go from metric to braker and burnett is non-obvious and involves lots of industrial cul-de-sacs with fences, and a small river).
Very, very tired. I got about an hour in on the report, but it's less than half done.
That wasn't fun.
It was great to see Adrienne in Missouri, not so great to get the truck hung up on a pole at a gas station and have to call a tow truck to get it unstuck, or the whole southern half of oklahoma which had no cell phone service (we took 75 instead of the interstate to get to Dallas) and I didn't know if Fade knew where the hotel was (we were going to call)...
Driving a 16 foot truck? Not easy. Doing so for four days? Very much not easy. And that was _after_ we got out late because packing took longer than we expected...
The cats are very unhappy with us, having been put back into cat carriers four times. Even Aubrey's hiding, although she's not very good at it.
Mark helped unpack the truck. We have too many boxes, many of which are very heavy.
I should have had my report to the Linux Foundation in today. I expected to get in two days ago. I haven't _started_ it yet. Internet access would be nice...
Moving day. Driving to Austin...
Working on my end of month report. Trying to get 8 gazillion half-finished things into a state worth mentioning, just like the last 2 months. Last month I was about to head to OLS. This month I'm about to head to Austin. Having major travel scheduled for the end of the month _and_ having my report for the Linux Foundation at the end of the month... Slightly stressful. I couldn't skip visiting Fade's relatives, and I had a huge todo list for visiting Eric earlier this month (some of which was work related, and most of which I didn't have time to finish anyway). As for upcoming travel, sword camp I skipped last year and my grandparents in florida were too sick to make it to the wedding in April but I _still_ haven't made it down to see them... Busy. Right. Hope it clears up soon.
New topic: If you switch to another desktop in KDE, and a window resizes itself while on another desktop, the resize silently fails in a way the window manager doesn't notice. This happens when qemu goes from simulated text mode to a higher resolution simulated graphics mode (as it boots Linux). So you get a window where the border just _ends_ at the bottom left and top right edge, and where qemu thinks the desktop is bigger than you can see, and there's no way to scroll it.
I finally figured out how to fix this without killing and restarting qemu: click the "X" at the very left edge of the title bar, to bring up the window manager menu that lets you move a window to another desktop and such. Select "advanced", and then select "special window settings". Go to "geometry", click the "size" checkbox, select "apply now" leaving the actual number intact. (Don't ask me why it's a fraction, I have no idea.) Then click "ok". Poof, your window expands to its actual size and is usable again.
The reason I have a qemu instance running is to host a fresh Ubuntu 7.04 instance in which I'm installing every package including the entire "universe" repository, so I can get a tarball of every single man page, to send to Eric Raymond so he can covert them with doclifter and put them on the web. (And also debug the tool some more against a larger dataset, now that he's finally moved on from Fedora.)
The way to install "the next 200 uninstalled packages" is to do:
aptitude install $(aptitude search . | grep '^p' | head -n 200 | awk '{print $2}')
And then to answer "y" or "yes" when it asks various stupid questions. It also brings up various curses-based configuration things, but you can usually exit those with minimal fuss.
I'd use a graphical tool like Adept to do this, but my estimate of just how long it would take to _select_ everything (tell the filter "show not installed" which is 20,000 and change packages, click the first one, scroll to the bottom of the list, shift-click the last one) was 52 hours. Based on timing how long it took to select 10, and doing math. Admittedly that's before I got enough memory I could afford to give qemu 384 megs rather than 196, but that still doesn't count the amount of time to download all the packages, or the time to install them. Which would have to be one big uninterrupted multi-day block. (Telling the command line one to install everything at once has a similar problem, but it times out and aborts after doing so much dependency resolution it thinks it's in an endless loop when it isn't.)
The reason I'm doing this in qemu is I need to actually _use_ my system. Installing some of these packages really screws up Ubuntu. I have no idea where KNetworkManager in the qemu image went, for example. Something disabled it during one of the big installs... And I just watched the current batch remove "cron", which it apparently felt it needed to do to install some other package, but which I'd like to have kept the man page of. Sigh. (There goes abiword and apache2... Possibly I should snapshot the man pages from a fresh install to add to the ones from installing everything, to make sure that at least the base packages are all there.)
The entire day was spent on moving. The "1-800-got-junk" people stopped
by to pick up the cats' favorite scratching post couch
this morning, and then we went to the storage space to rent a big truck
(and got an even bigger truck than we expected, because they were out of the
12 foot ones), and then spent the rest of the day emptying our storage space
into the truck. (We have many books, papers, boxes of old computer magazines
for computer history purposes... Heavy. Very heavy.)
Then it was time for the farewell dinner with the TimeSys alumni (none of which work at the company anymore, but that's where we met). Sushi at Pacific Ring on Murray.
Got home, finally got a chance to turn on the computer, but was too tired to do anything.
My laptop now has 2 gigs of ram. Yay! I need to repartition so it has enough swap space to suspend itself, which implies I should reinstall to get an actual 64-bit version of Ubuntu, but I'll see if I can hold off until we get to Austin.
Back in the Te Cafe, tackling documentation. This would be easier if I'd managed to get any sleep last night. (We thought the downstairs neighbors had moved out because their door was half-open all day with nobody in it, and the piles of garbage they've strewn throughout the yard and stairwell were slightly larger than normal, but they got home around 2 am and proceeded to be loud until about 6:30. Catching up on sleep didn't quite happen for me last night.)
Also, I would appear to be allergic to Pittsburgh. I was fine in California, but last night I had trouble breathing again as soon as the neighbors got home (maybe I was just asleep before that, but it seems they smoke something my lungs don't like). Also, I'm allergic to the cologne of the guy sitting next to me at the Te Cafe (it's making me cough), but that's probably not related.
Reeeeeally looking forward to getting out of this... ahem, "city".
Our plane home left at a bit after 9 pm yesterday, meaning the second half of the day was taken up with last-minute visits with california people, re-packing, arranging to return the rental car, and wandering through airport security.
Apparently, putting my flip-flops through the X-ray machine is no longer optional. Brushing aside the deep silliness for a moment, as I watched people voluntarily doing this a year or two back I wondered how long it would take for it to become mandatory. If so many people hadn't volunteered to do it, it wouldn't have become mandatory.
Anyway, a five hour plane flight to new york, plus three hours lost to time zone changes, plus a 2 hour layover, plus Delta losing our luggage (of _course_ they did, it's still a bicycle-powered carrier) got us home around noon, at which point I went to bed until Garrett wandered by to feed the kiggies and I took him out to dinner to say thanks.
Got nothing of use done today. Our seat on the 5 hour flight was right in front of the exit row, so it didn't tilt back, but of course the people in front of us tilted theirs back, so there was no room to pull out my laptop. I did read the last Harry Potter book (in a cramped enough space that I accidentally poked Fade in the eye with it at least once). Good book. Also watched "My neighbor Totoro" in the evening before going back to bed, which was also good.
Yay, internet! (Both of Fade's sisters have internet, of course. They live in Los Angeles, her older relatives live in Redlands.) Back near the airport for the plane back to Pittsburgh this evening.
Kay Sievers is STILL bouncing my email as spam. This is just SAD.
Still no internet access. Looked around for a coffee shop with wireless, but don't know the area. Spent most of the day back to Fade's relatives' house with the pool to be social.
Obtained the last Harry Potter book. Fade read it, I look forward to a lost day in the near future. (736 pages, or thereabouts.) The book I got is "Your Movie Sucks", by Roger Ebert. (I didn't MEAN to get a book, but I went into a bookstore.) Luckily, it's a bunch of two-page articles, easily interruptible...
No internet access at the hotel. (In theory it has it, in practice it's broken. Sigh.) Oh well, it's away from the stress of having to deal with sysfs developers, and I've got enough sotred material on my laptop to keep me busy for months. (And it keeps accumulating because _new_ material comes in faster than I can deal with it. Oh well.) Got a couple hours in summarizing old OLS papers before it was time to Be Social, but the pile of "check this reference at $URL" notes does tend to accumulate a bit working offline.
Went to the gathering of Fade's relatives we're in california for. Most of them couldn't make the wedding in Michigan, so they brought us to California for a reception. Fade's father made a speech, many introductions to relatives whose names I didn't remember immediately afterwards. (Fade's two sisters, one brother, and parents I can just about keep straight. Although I think I've forgotten her father's name again. The rest? Not a chance. I can't even tell her grandmother from the white haired aunt(?) who owns the house we were at. Unless they're the same person, I should ask.) There was a pool. Much swimming ensued. Also cake.
Went back on caffiene. On the road you either have diet beverages, or caffiene free ones, but it's hard to find diet caffiene free beverages other than tap water.
More fallout from posting hotplug documentation. Some developers are a touch territorial.
Memo: Helpful criticism is "XXX is wrong, it should be $Y". Less helpful but still helpful is "XXX is wrong, look at $Z". Unfortunately, "XXX is wrong, you're stupid, give it up" is in the NOT helpful category. Luckily, I already know I don't understand most of this and probably never will, and it doesn't bother me. Darn time consuming, though.
Also time consuming (but productive, and surprisingly educational) is the ongoing email conversation with japanese people about cultural differences that will probably eventually result in documentation to help language maintainers, especially in japan and china.
Fade's parents dragged us to Disneyland. I am sunburned, dirty, and exhausted, but it was fun. Got in maybe 2 hours of work today, most of it fretting about Kay Sievers calling me an untalented hack. (I agree with him, I just find it a bit impolite of him to say so. :)
Fade's sister (the professional actress/model) is of course highly cute. I wonder if her other sister is violating some kind of california statue by _not_ auditioning for acting gigs on top of her day job. (It's probably only a misdemanor if so. They give you a ticket and a stack of sides to practice reading...)
The new wireless connection daemon in Ubuntu is nice in some ways, horribly brain damaged in others. There's no way to tell it to rescan for available networks. If you mistype a password (such as entering a hex key as a passphrade), there's no obvious way to re-enter it until the connection attempt times out. (If you bring the list back up and click on the entry again, it restarts the connection attempt and makes the timeout longer. Right-clicking on an entry is the same as left clicking.)
Still, a huge advance over previous versions which didn't have one at all in the default config...
Yes, you can get jetlagged from flying _west_ three time zones, especially when you finally roll to a stop the equivalent of 4 am in your starting time zone.
Finally got the hotplug documentation in yesterday, and of course Greg KH submitted some sysfs documentation a few days ago. (I was working on it before my laptop died, and mostly finished it after talking to Kay Sievers at OLS, but I had to track down the old netlink version of mdev Frank Sorenson sent me back in 2005 in order to document the mdev bits properly, and that was a lot of back email to sift through. Might have been faster if I'd remembered who submitted it before coming across the post...)
The ram upgrade I ordered for my laptop is in, but I forgot to pick it up in time before we had to leave for the airport. Spent a largeish portion of the day on airplanes, which were of course late because we're using a traditional "hub and spoke" carrier, which means the entire operation is bicycle powered.
Feeling distinctly under the weather. (Possibly related to the fact I ate too much at Red Robin yesterday. Free refills on french fries!) Working from home this morning rather than going to the coffee shop.
Leftovers from the weekend: The toybox mke2fs has to support this. I'm not a fan of GPLv3 but Eben Moglen's lecture at Edinburgh last month is really good anyway.
Arranged to trade in my old laptop to double the memory in my new one (to two gigs). I'm pretty happy with the deal.
It's 2:07 pm and I have hit brain full. Brain is full. Must go do something else now. Brain full.
The tangents! The tangents!
Right, I got the _first_ patch in a series pushed out to the list. I also learned how to use diffstat, added more material to the sysfs-hotplug docs, identified and reported the server that's been generating spurious bounce messages on linux-kernel for the past six months, introduced sigaction(SA_RESTART) to some qemu developers, put Mel Gorman in touch with Don Marti of LinuxWorld (and got a copy of Don Marti's author guidelines which allow articles they write for the current incarnation of LinuxWorld magazine to be included in open source packages as documentation), got a line on some patches to the uClinux ld.so to fix sparc support in Firmware Linux, participated in the ongoing thread about translating kernel messages, and pinged Richard Gooch about updating the LKML Faq.
None of which was on the todo list I had at the top of this blog. That experiment apparently didn't result in a more updated todo list, so I've removed it.
Now to push out the _second_ patch for kconfig genericization, and to
get the cable modem and electricity set up for the move to Austin. And I
did the phone thing, but the kconfig thing turned into an argument with
Roman Zippel (as always), and I submitted a patch to MAINTAINERS
to add language-maintainers.txt to Documentation describing the position,
found a group of policy documents that might need to be translated,
got a link to a
Tangents. It's my daily routine.
Ok, I should finish stuff up and push it out today.
Let's start with the toybox patches genericizing kconfig. The toybox kconfig seems to be a snapshot of kernel hg 39604, so some rebasing is definitely required...
Back at the Te Cafe, trying to catch up on stuff.
Interesting bit of nostalgia, I installed Red Hat 6.2 in a qemu image today, and it rolled a critical failure attempting to detect the virtual VGA card, which is emulating a western digital/cirrus logic card picked for being very ancient and standard and supported by absolutely everything including Windows 3.1. Ah, memories: Linux hardware autodetect (specifically X11) used to TOTALLY SUCK, as recently as 2000. Wheee. (In its defense, it is running kernel 2.2.14.)
The reason I'm installing Red Hat 6.2 is it's the oldest ISO I could find on ftp://archive.downloads.redhat.com. There are bits of older distros, but not in an easily installable format. And the old 1.0 and 1.1 releases, which I might have been willing to put some effort into installing for historical reasons, are totally bit-rotted away.
Meanwhile, I locked up ubuntu 7.04 for about five minutes. How? On my shiny new system with a big disk I did: "dd if=/dev/zero of=blah.img bs=1M seek=4095". Guess what I forgot? "count=1". So it wrote 17 gigabytes into the file and then died with an error I'd never seen before ("File size limit exceeded (core dumped).")
But that wasn't what locked the system solid. Deleting that file locked the system up for five minutes. I could just about type stuff into my already open xterms, but trying to kill the rm process gave me a process in D state until it decided to finish.
A number of people have pointed me to sneakers with wheels in the heels (here and here). I agree this is probably what I saw on July 5, although they hid well.
Drove back from Eric's. Rebuilt the prebuilt FWL images for the 0.2.2 release along (I forgot to do this when I put it up), but mostly I was the one driving. I should add configuration options to the cross-compilers, so you can statically link them and so you can add c++ support. And I should statically link the prebuilt ones I put up so they're a bit more portable. Probably the qemu application emulation test should also be a command line option...
There are currently sixteen people in starbucks, every single one of which is male.
Eric is rewriting Emacs VC mode to understand changesets, so he can add a proper mercurial mode. This is apparently something he's been meaning to do for many years. He wrote the original VC mode code back before modern source control existed, and all the Version Control programs back then were file based and didn't have the concept of "changesets". Teaching it about changesets is a complete rewrite, so Eric gets to play with Lithp, and is happy.
Mark is apparently going cold turkey on alcohol, cigarettes, and caffiene all at once. Kind of puts my annual detox ritual in perspective. Still enough to knock me out of any sort of productive flow, though...
Set up a qemu instance with 16 gigs of disk space and set it installing Ubuntu 7.04. I'm trying to figure out how to install every package in the standard and Universe (but not multiverse) repositories. I tried "aptitude search . | awk '{print $2}' | xargs aptitude install" but that timed out after searching 5000 packages for dependencies. (Apparently aptitude decides it's gone into an endless loop doing dependency resolution if you give it too much to bite off at once.) I can't do anything useful with synaptic, so I told it to install kubuntu-desktop and I should be able to shift-click and select the whole list, right click and select to install, and then tell it to apply all changes. Maybe that'll work. If not, I can feed xargs "-n 1" and see if that works...
Yeah, 16 gigs of disk space probably isn't enough for this...
Off to the Matuszeck's again for barbecue...
Eric has reacquainted himself with C++ to bang on the Battle for Wesnoth code. (According to their source control statistics he's now their third most prolific contributor in the entire project's history, which surprised him a bit.) I commiserated about C++ a bit and pointed him at Garrett on IRC, who knows where all the bodies are buried in that language.
More productively, I talked Eric into converting the Doclifter repository to Mercurial. Then we spent rather a long time making this work, because the old doclifter repository was in RCS. He's been doing Doclifter since 2001, and it's apparently his last project left is this particular ancient repository format. (He assures me he got rid of the SCCS ones long ago.) RCS is essentially CVS without the networking half, but all the CVS to hg stuff we tried didn't recognize RCS. It's the same database file format holding the actual repository, but there are enough minor changes (like directories having different names) that it wouldn't recognize it. He wound up converting it to subversion (which is what he's been using, and apparently has contributed a number of patches to), and from there I tried to convert it the rest of the way with tailor (which totally failed to work, grr), and wound up digging up the old script I wrote to convert the busybox repository to SVN before I ever found out about tailor. THAT worked.
So there is now a doclifter mercurial repository, and Eric is off investigating how to get an emacs mode for it.
Meanwhile, I've been crashing badly from lack of caffiene. We went to the Matuszeck's house last night to play a rail game, read books, and watch Red Dwarf. Poked at my laptop a bit today, but didn't do much useful other than catch up on my email.
Driving to Eric and Cathy's and frowning at documentation stuff on my laptop on the way. I think the "Documentation/ABI" directory is mostly useless. The difference between "testing" (we may add stuff to these but shouldn't break existing programs using them) and "stable" (we may yank anything in here after 2 years) is pointless, and both directories are advisory at best. It smells like Greg KH's quest to extend stable-api-nonsense.txt to userspace is continuing, and I'm not humoring it.
That said, the "obsolete" and "removed" directories in there might be useful. Stuff they actually DID break, or that you can configure out of modern systems.
Catching up on the QEMU mailing list (since I'm not that far behind on that one).
Hmm, the "ignoring extension type g" thing should have been fixed in busybox svn 15660. That was a bug I'd fixed in the development branch of busybox before I left, and which I thought was backported to the stable branch...
Oh well, deal with it after work. (Day job time: now that I've got the documentation index up I need to shuffle stuff into it.)
A little blonde girl (maybe 8 years old) did the world's coolest sliding thing on the concrete sidewalk outside of starbucks for about five minutes. It looked like she was flying in a standing position.
She's wearing sneakers but she's using them like she's got roller skates, sliding for over 20 feet from a slow run (only a little faster than walking). She's putting the toe of one sneaker up on the heel of the other, raising the front sneaker so only the heel is touching the ground and slightly tilting the back sneaker so only the outer edge (and just the heel of that) is touching the ground, and sliding on these two points, straight forward, with almost no friction. You'd think it would wear the sneakers out incredibly fast, but apparently not. (Cheap white and pink sneakers, nothing special. Very definitely NOT roller skates, and they don't slide at all when she runs or stops suddenly.) She also did the sliding trick a few times inside the starbucks (stopping to say hi to what seemed to be her family, at one of the tables), on a tile floor with big squares with largeish grooves in them. It seems largely immune to terrain, as long as it's flattish. The concrete outside is _very_ textured, and on a hill that she slid _up_ as often as down.
Yeah, I'm amused by weird things but that is SO COOL. I'd try it myself but I have entirely the wrong type of footwear and I probably weigh upwards of five times what she does (which is probably significant to the physics of the thing)...
Oh well. Meanwhile I've been documenting sysfs. Wondering if I'm spending a bit too much time on the computer and not enough playing outside. I do cool things too, but I need to explain half an hour of background to most people before they have enough context to understand what it is that I'm doing, let alone why it's cool...
Happy gratuitous patriotism day.
So building Firmware Linux under itself brings up a few interesting edge cases. Last time I did this was in the old "x86-glibc->x86-uClibc with UML as the emulator" days. Now I'm trying to get the mini-native system for ARM to build a mips version. (Yes, native building on arm to cross compile a MIPS version. Because I can, that's why.)
So far, I have to mkdir /proc /sys and /etc, symlink /bin and /lib into /tools, mount /proc and /sys, run mdev -s to populate /dev, and set up the network for qemu (interface is 10.0.2.15, gateway is .2, nameserver is .3). Oh, plus switch to using the rtl8139 driver because PIO doesn't work on the qemu-system-arm PCI controller yet so I need something with mmio. Oh, and call rdate to set the clock because it starts at Jan 1 1970 although I'm told that's fixed in CVS now...
It built binutils without a problem. Died trying to patch gcc but that's because the busybox patch (in 1.2.2 anyway) didn't support "fuzz" and the patch is quite a bit off anyway. After I updated the patch it built gcc (which surprised me because you used to need more than 128 megs of RAM to do this, but I forgot I was setting CFLAGS to tell gcc to garbage collect its darn internal data structures faster).
Now it's dying with:
=== Building linux (mips) Extracting 'linux.tar.bz2'tar: Ignoring extension type g tar: Unrecognised file type ./cross-compiler.sh: line 1: caller: command not found
Wheee... Probably a busybox thing. I live in fear of having to make a major change to busybox, because I won't do it, and dropping everything and implementing "patch" or "tar" or some such in toybox would be a largeish diversion. Still, haven't had to yet...
Confirmed that the arm-scsi fix works for me, but it's not in -rc7.
Re-split the OLS papers (including the ones I'd downloaded pre-split) to put the OLS boilerplate pages at the end of the file. (This also means I can do a quick sanity check by looking at all the thumbnails konqueror puts in the directory; if the page doesn't start with a title and a gap I got the split point wrong. If the first page is always the OLS announcement page, it tells me nothing.)
Finally updated http://kernel.org/doc again, for the first time since my old laptop died. I still need to recreate some of the Documentation and htmldocs updating infrastructure (I _was_ doing that from git but my backup is pulling mercurial; version skew when restoring from backups is fun!)
Appended my skeletal index to the end of main page. I need to clean it up and start hanging other documentation from it.
Still going cold turkey on caffiene, although for values of this that involve having a hot chocolate at Tim Horton's yesterday, and not minding that decaffienated tea still has some caffiene in it. (Or possibly caffeine, I've never managed to lock down the spelling of that one for more than thirty seconds after looking it up, and I don't normally use spell checkers, so...)
The 2007 papers come pre-broken-up, just like 2004 did. However, there are some issues. The pre-broken-up papers come with three page prefix each of OLS boilerplate, which his good from an attribution perspective but means that the thumbnails are all the same OLS page. I think I agree with having those three pages attached to each split-up paper, but at the _end_ of the paper, not the beginning.
This means I have to do my own copies of the split up papers, and also re-split the papers I've already split up. But oh well.
YES! Somebody tracked down the arm scsi bug and did a patch for it. Which may be in the 2.6.22 kernel, hope hope hope...
So at the Gentoo BOF I was told that make -j should always be one more than the number of processors in your machine, because some things are I/O bound. So I benchmarked it using a kernel compile under /usr/bin/time (since the output of bash's builtin "time" doesn't want to redirect to a file, or work inside a subshell). I configured the 2.6.20.14 kernel for armv4l, did a make clean, "echo 1 > /proc/sys/vm/drop_caches" as root to emphasize any I/O boundness, and then built the kernel on my core duo 2 laptop. With -j 2, it took 4 minutes 9.85 seconds (4:09.85). With -j 3 it took 4:08.16. The difference is essentially noise. I tried again without the drop_caches first and -j 3 took 3:44.49, and -j 2 took 3:44.03. Again, noise.
I'm sticking with -j being the number of CPUs, without the +1.
Driving back from OLS...
Party at the blackthorn! I don't drink alcohol but I had six red bulls to show willing. (Last year I had five and was fine. Six is apparently too much. Time for my annualish caffiene detox, I think.)
Vague flashes of party involve Fade, Mark, and Wednesday (whom we dragged along for the free alcohol) trying lots of different drinks, me explaining the 64-bit paper to Adam Belay, a couple people poking me to finish up my mke2fs that works like an archiver, noticing that everybody who works on ext4 seems to be female (the talk was given by two oriental women, and Val Henson is also on that although she didn't seem to be at OLS), and although it would be highly impolite for me to "OMG-A-GIRL" over this I note that I didn't quite work up the nerve to go ask them what I need to add to my mke2fs to turn it into mke4fs, but I can do that in email (after I've finished ext2, and added mke3fs and htree support)...
Sushi. Shish-kebabs. Crab cakes. Much red bull. Accidentally grabbed from a passing tray without paying attention and found myself holding two live oysters, carrying them around until I found somebody who would eat them. Talked to guys from nokia about the N800. I can apparently get it for around $140 from Best Buy (although when I checked it was $340). There's some mystic certificate I can register for to get it for $100 because I wrote some of the software in it (if they're using BusyBox 1.0 or later, the answer to that is probably yes). Unfortunately, checking on the web the answer seems to be "no", since there were only 500 of these and they started emailing them to the Maemo developers in January. That was the important project to be on, it seems.
Oh well.
My brain largely shut down after giving my tutorial, and now I'm just trying to learn what I can. I'll read the papers afterwards and see what I missed.
There was a CELF BOF in the evening I have a couple pages of notes from. I vaguely recall jumping into the lightning talks to tell a room full of consumer electronics people that cross compiling is a bad idea, yet I seem to have emerged unscathed.
I find the Fedora port to Arm that Manas is doing hilarious, I should explain why when I have two functional brain cells to rub together.
Fade spent the day with Wednesday White of Websnark, and seemed to enjoy it immensely.
I need to pester Kay Sievers so he stops spam blocking my email based on the fact that a year ago the IP the server is using was dynamically assigned, before it was given to a static FIOS that's had it ever since.
But then my policy on other people's broken spam filtering setups has always been "I didn't have anything important to say to them anyway", so it's not high up on my todo list. I haven't managed to run into him in the halls yet, and if I can't I'll post the docs to the kernel list when I get them written up.
Stuff continues to happen faster than I can make note of it.
Left my laptop's power cord at the party Intel sponsored last night, but it was in the lost and found this morning.
Gave my tutorial on cross compiling Linux. I should have asked for at least 3 hours, I totally ran out of time at the end and had to skim the second half in about 5 minutes. Sigh. I feel sorry about intruding into the time of the presenter right after me. On the bright side, this might result in a few more Firmware Linux users. It seemed to work for most everyone, except for a few people for whom User Mode Linux didn't work. I should clean up the packaging state not to require that.
Spoke to James Bottomley (the SCSI maintainer) who confirmed there are no plans to port the ramdisk to go through the SCSI layer, and that the guy I need to talk to about the arm-scsi-qemu breakage in 2.6.21 is the ARM platform maintainer, Russell King, who is not at OLS this year.
Spoke to Kay Sievers and Greg Kroah-Hartman about the sysfs documentation, (almost wound up late for my tutorial because of it). Kay emailed me the notes from this, I can write it up when I get home.
Followed up on the language maintainer thing with Satoro Ueda of Sony, who might be able to find somebody in Japan who can act as a Japanese language maintainer. (The theory here is that if I'm collecting documentation in foreign langauges, this encourages the creation of patches from people who can't get them merged back into the kernel because they don't speak english, so there should be somebody who can collect patches and translate the descriptions, and translate questions and answers back and forth. I.E. a Japanese patch maintainer, korean, french, etc.) Again, I boggle that this problem has wound up in _my_ lap (the only class I failed in high school was German and the only D I got in college was French), but things get done by people who do them, so...
At the PPC BOF (where I didn't win a Praystation the third but H. Peter Anvin did), I spoke to Hugh somebody about finally building a PPC kernel I can boot under qemu. Here's hoping this turns into more than last OLS's conversations on the same topic did.
Saw Richard Gooch at the PPC BOF. He works for Google now and has a tennis racket from Taiwan that charges to 50,000 volts from batteries in the handle. He _is_ still actively maintaining the Linux-kernel FAQ, which is good to know.
People keep recognizing me, even when I wasn't wearing my badge. (Left it in the hotel room this morning, Fade brought it to me.) This confuses me, I'm really not important among this crowd...
I'm too fried to remember much else of what happened today. There was a lot. (Marvelous little mushroom things at the PPC bof, by the way).
Ok, it is VITAL that all Linux users know "rmmod pcspkr". Yay OLS.
I can has blackfin!
Robin Getz's "more linux for less" tutorial included a free blackfin board I got to keep. With a USB to PCI converter, even. I have a nommu test system I can actually _build_ stuff for now, and run what I built. (I have a coldfire I got sent last year, but I've never managed to run any code I wrote on it. I can try again when I get home though, based on what I learned about uboot at this tutorial. Bootloaders are still a weak point with me.
At the CELF BOF in the evening, the slides mentioned me by name. (They didn't even know I'd be in the room. Boggle.) The slides also mentioned Toybox. UBER-BOGGLE! I didn't know anybody but me was paying the slightest bit of attention to it. (I'd have put more time into it over the past few months if I did, my day job is now this Linux Foundation documentation stuff and the 10-15 hours of free programming time I have a week is mostly going into Firmware Linux because people are actually using that...
I also note that the problem of finding language mantainers for the Linux kernel seems to have fallen into my lap. When did this happen?
Found the bug in x86-64 headers from yesterday, although on further examination it seems only the missing semicolon is needed, the exports are optional. Anyway, x86-64 now builds and I can boot x86-64, i686, arm, and mips. Sparc builds but doesn't boot, and PPC has the ppc/powerpc thing as well as some bugs in qemu last I checked, but close enough for now.
Off to OLS!
Driving to OLS (well, I'm in the back seat, Fade's driving, and Mark's in the front seat playing Final Fantasy 7 on his psp. I'm trying out my new laptop which has a fairly impressive battery life so far.
Wow this thing is fast. Since it's a dual core I taught the firmware build scripts to autodetect the number of processors and feed the appropriate -j to make, and it's scrolling things by so fast it's a blur in places.
If you want to see all the gcc predefined macros, you can do this:
gcc -dM -E - < /dev/null
Which lists "#define __x86_64__ 1" in there, so the weird #include for asm/ioctls.h should suck in the one from the asm-x86_64 directory, so uClibc 0.9.29 shouldn't be dying with TCSBRK undeclared. Unless it hasn't got the proper #include, in which case why isn't it dying on other targets?
OOH! I want one!
Mirroring OLS stuff, banging away at my tutorial (oh yeah, I should mention THIS, now where would that go, must rearrange existing material...), and trying vaguely to figure out how to work in a monthly report for Linux Foundation when A) I don't have time before OLS, B) I need about another week for the heap I'm indexing to bear fruit.
Yesterday night the power went out for a few seconds, rebooting my desktop and costing me about 15 open windows with various pending things in them. (Yes, I save, but if you close all my windows how do I remember what I was working on?) Took it as a sign and went to bed.
I really, really, really miss my laptop. Dell's page says... Yay, it made it off the boat from Taiwan and is now in the hands of DHL in Tenessee. Likelihood of it being here before I head up to OLS on Tuesday: probably zero, but here's hoping. (If not, I can borrow a laptop.)
Anyway, what was I working on? Trying to build a recent qemu cvs snapshot that doesn't segfault, fixing more bugs in doclifter's conversion of man-pages, my OLS tutorial write-up... Actually I should stop right there and work on the write-up, it's on a deadline.
On the qemu front, there's a git version of the qemu repository which should help track development.
Linus replied to my question about translations, and I've merged the chinese translation of Documentation/HOWTO into the kdoc mercurial repository. (Hmmm, it should probably be zh_CN/Documentation/HOWTO instead of zh_CH/HOWTO. hg rollback and try again...)
(I am sitting here, waiting for this machine to swap. Still waiting. It's trying to open a new (empty) browser window so I can get a google groups link to splice into the above paragraph. I decided on a Google groups entry because I can base that off msgid which is A) easy to extract from Linus's email, B) unlikely to change over time. (I've had links to archives go dead before, it's annoying when you can't then find the message in some other archvie because the old link doesn't tell you anything about the message.) Ok, new window is open, fire up groups.google.com in it... swapping. I so look forward to my new laptop, and will probably upgrade its ram when it arrives to NOT HAVE TO WORRY ABOUT SWAPPING ANYMORE. I suppose getting up from my chair to pace while this thing swaps is good for exercise, but it sucks for getting anything done.
Huh, google groups appears to be broken at the moment. Looking up the message by msgid: not found. Searching for messages where author is torvalds@linux-foundation.org: nothing found. Database offline? Ok, I found it searching by his name. No trace of a msgid in this URL but at this point I've put too much time and effort into it already. Google, what's wrong with you today?
Finally got Fade's domain updated. That shouldn't have been nearly that much work either. (Half the problem is that dotster's UI is horrible. You enter two nameservers, and it tells you it either accepted them or it didn't. If it didn't, it won't say which one it had a problem with, or what the problem was. The problem turned out to be that one of the nameservers wasn't "registered as authoritative", a requirement that doesn't seem to be embodied in an RFC anywhere.
Couldn't find a hotel in Ottawa that wasn't booked for Canada Day, but David Mandala pointed me at a hostel that wasn't full yet, so life is good. OLS has apparently decided to go all pointy haired this year. I just got an email about how you can't access the wireless without WPA, and they're filming videos of all the presentations, as a profit center for the convention. Presenters will have to buy copies them if we want copies, and then we then won't be able to put them on our own websites or link to from the OLS site. I understand now why Thomas Gleixner decided not to come this year, this will probably be my last one.
Poked at Firmware Linux again, but QEMU CVS is broken. It's segfaulting trying to run qemu-i386 even with no arguments. Not sure why. Let's see what strace is doing... Ok, compare it with what "hello world" is doing. Link hello world against -lm and -lrt to make it match up...
Finally started the Mercurial repository for kdocs. Only a few things in it yet, because so much of it is half-finished slush or notes to self or things I want to rename. But it's a start.
I'm going through the kdocs web page and making sure I can reproduce everything there in an automated, scriptable way. (Lots of my automation infrastructure went away with my laptop, and the bits I had left either don't fit together are are notes how to do it by hand rather than actual scripts.)
Also spent too much of the morning fighting with domain name registrars. Apparently, dns.thyrsus.com is no longer an authorized nameserver for landley.net, and when I attempt to add it back, secure.ns.com says it can't find it. So dig can find it, but my domain name registrar can't. Why was it removed in the first place? Email in to support says I have to "register" my name server. What does that MEAN? When did this start?
Spent the evening wrestling with doclifter, recreated the conversion from man to xml and html, now for man-pages-2.55. Got it to do _all_ the directories this time, and checked the script for doing so into hg. Noticed that the symlinks are all wrong (absolute paths to where they were built), need to fix that in the morning.
I now have something like five different half-finished largeish chunks of documentation I'm writing. Need to finish and push out...
Tried updating all packages in Kubuntu and rebooting, but Konqueror bug where the "scroll tabs left" button doesn't do anything is still there. However, I figured out a workaround (the mouse wheel switches tabs when you're over the tabs). While I hate all "mouse over" behavior on general principles (work on what has FOCUS you stupid piece of...) the Kubuntu guys, like most desktop designers, believe they know what I want better than I do and have yanked the old way of doing things in favor of their personal preferences. Again. So even though I have policy set "click to focus", what the mouse wheel does is based on what it's over rather than what has focus.
Ha! The way to make pdftk split up a file is:
pdftk input.pdf cat 12-34 output output.pdf
If you don't specify an output, it complains "Input errors, so no output created" which is a HORRIBLE message for that error case. I should NOT have had to build this sucker from source to work that out.
Huh. I'd forgotten how thunderstorms screw up one's work when one has a desktop instead of a laptop. (Still on a boat from taiwan, or as Dell's website says "In Production". I'm starting to regret not having sprung for more than 1 gig of RAM, but that's something I can upgrade when it gets here. I did get the 160 gig hard drive, though.)
I had a conversation with Stu yesterday about how to properly organize the mountain of kernel documentation I've been sifting through, and I've decided I need to start up a mercurial repository for a web page. That way, people can download their own tarball of the most recent version of the web page trivially, and once I've got enough up they can send me patches against what's there.
Of course now that I'll be putting my build infrastructure out there for other people to use, I'm thinking about things like automatically downloading the man-pages source tarball and such. (I can yank existing infrastructure from Firmware Linux for that.) And documenting what prerequisites you need installed in order for this to work.
Since I did my clean install of Ubuntu 7.04 last week, I've installed a number of new packages. I needed:
Ulrich Drepper apparently removed LINK_MAX from glibc and insisted everybody use _POSIX_LINK_MAX instead. The first value is 127 (it's still in linux/limits.h but they go out of their way to #undef it), the second is 8. Linux stopped recursively evaluating symlinks in 2004, so the small limit is not what the system does. Once again, standards are seen as limits rather than minimums, if you can do MORE than the standard, you shouldn't be able to.
The limit is there at all to avoid recursive symlinks going into an endless loop when the filesystem (or something like readlink) tries to follow them. I'll just hardwire the 127 into readlink.c in toybox because the glibc guys won't give me a real define for this. (No, I don't care if my code works on Solaris or BSD: I'm not interested in that, and I don't write software for those platforms. I want it to be portable to future versions of Linux.)
Huh, realpath turns out to be nontrivial for the same reasons xabspath() isn't the same as realpath(), I.E. dir/.. isn't a synonum for "." if dir is a symlink. Appending the symlink value to the base directory of wherever this symlink is and then cleaning that up with xabspath() doesn't quite cover it either, because if a symlink expands to "dir/../blah" and dir is a symlink then xabspath() results in the wrong path. Also, the current directory it starts out with is effectively run through realpath().
Right. Punt on this for now. (And on "touch" which I notice I added a "hello world" for and then never implemented. Oops.)
Decided to clear the palatte a bit and do some test cases for the toybox test suite (in this case, for the new readlink command), and hit an interesting corner case with ln -f:
ln -s . link ln -f link link ln: creating symbolic link `link/link' to `link': No such file or directory
Now this is the old (gnu) version that's being odd. (I write test suites against what Kubuntu has, before replacing it.) The question is: does this count as "misbehaving", or is it supposed to do that? If there's an existing symlink that points to a directory, do the semantics of -f require removing that symlink, or following it and removing what's in it?
Ah, I see. There's a -n to go with the -f.
I'm a sucker for intellectual property discussions. I admit it.
I note that "in production" is Dell's way of saying "on a boat from Taiwan". Still figuring out how best to do my OLS tutorial without a laptop.
Heard back from Zack Brown, forwarded him to Mark. Fingers crossed...
Went through the IBM Redbooks today looking for kernel documentation. Found lots of stuff like this which seems useful at first glance, but isn't. If I wanted to know about FTP commands for the z/OS console or anything at all about DB2 or websphere, this would be the place to go. Kernel stuff, not so much.
Installed pdftk and xpdf-reader from the Ubuntu repository to pry apart the OLS volumes into individual papers. Unfortunately, the version Ubuntu packages doesn't _work_. Looked at lots of other things to such as Konqueror's "print to PDF" with a page range (which is both conceptually broken since it turns each page into an enormous bitmap, and buggy because once again KDE's print manager can't figure out where the edge of the pages are. This time the top 1/4 of each page is off the top edge...) and popler's pdftohtml, which produces near-unreadable output, although that's not entirely its fault. It's about as good as an OCR scanner would be.
Went back to pdftk, downloaded the source code... which is in java, so I have to install gcj... Ok, it built and that has the same problem as the Kubuntu version (not surprising) but I can stick printfs in the source code to figure out _why_.
Poked Mark about doing more kernel-traffic summareis, and emailed Zack Brown about potentially putting up the ones Mark already did. (Ok, technically I emailed Zach Brown, but he forwarded it... :)
How unbelievably stupid. The new version of Konqueror combined the "stop" and "reload" buttons. Now if you hit stop just as the page finishes loading, it starts loading all over again. And you can't hit stop on a page that's got a "meta reload" tag to refresh itself unnecessarily every few minutes (hello zdnet), because that would mean reload (which is what I'm trying to AVOID).
Also a fresh show-stopper bug in konqueror: if you have too many tabs the "move left" arrow doesn't seem to work. So there's no way to access tabs that have scrolled off the left edge, except by closing visible tabs.
I have a full system backup from right before I left for Austin (when was that, 3 weeks ago?), and incremental backups of my web page and my email directory that date from earlier in the day the laptop died.
The question is, what fell in the gaps between those two? (All of the research I did for the 6 month follow-up on the 64-bit paper, for one. Some of my documentation research that wasn't yet ready to go up on the web for another. Sigh. And my kmail set up is in the old home directory, but it's for the Kmail in 6.06 and this is 7.04. I have 39 filter rules to put my mail in all the right folders, I wonder if I can cut and paste this in kmailrc? Yes, apparently so, with one tweak to tell it how many rules there are...)
The regression in my mercurial repositories on Eric's server (the gz and bz2 links don't work) is actually a bug in Python 2.51. Not quite fallout from the move to Ubuntu, more fallout from the upgrade to a more recent version of Python.
It's going to take Dell two weeks to build my new laptop. (Translation: two weeks to get a preassembled laptop over on a boat from Quanta in Taiwan.) This means that the expected arrival date of said laptop here in Pittsburgh is the 28th, and Ottawa Linux Symposium starts the 27th. This is a problem. Hmmm...
Kmail still crashes on exit. Some things are reassuring in their consistency...
Vodz noticed I'm no longer working on BusyBox (yes I left last september, but he uses a russian to english translator and doesn't catch these details unless they're pointed out to him), and looked at toybox. Pointed out a bug (can't use va_start() twice on PowerPC 4xx, have to use va_copy()), which I've fixed in my copy but can't check in until I get the repository recombobulated from the backup. (I have the up to date version, but it's in the wrong directoy and the paths are all weird.)
Ok, I have _bits_ of my hotplug documentation. That's something.
So Saturday night I left my laptop doing something and came back a few minutes later to an unresponsive black screen and a very hot system (power button uncomfortable to touch sort of thing. No, none of the vents were blocked). So I hard powered the thing off and left it overnight. In the morning, it booted, but trying to replay the journal to mount /dev/hda5 gave "buffer I/O errors".
Ooh. Bad thing.
After scrounging up a boot CD I found that hda5 wouldn't mount as ext2 either. That was my home partition. Past tense.
Time for a new laptop.
I've been meaning to get a 64-bit system when I got to Austin in August, and buying a new 32-bit system seems deeply silly right now, but I'd planned on getting some stuff done Sunday, and have work on Monday. I need something to work on, and the other two systems in the house are Fade's Macbook, her old Windows laptop (which she's thinking of replacing with a mac), and the x86-64 server I bought which never did become stable enough to use. (It segfaults or panics if you do anything CPU intensive with it. Not sure if it's the power supply, the CPU overheating, or what. Memtest86 found exactly one bad bit on an overnight run and had to disable ECC to get there, but I can panic the thing just about all the time with three consecutive kernel compiles. Not useful.)
I phoned my brother who could get me an HPaq system preinstalled with Red Hat enterprise, but he's switched companies since the last time I talked to him, and I don't want to mail order something anyway. That could easily mean a week with no system.
So down to the the waterfront to look at the systems Office Despot and Beast Buy have on sale. I wind up with a nice little $600 number from Compaq that was cheap, had good specs (Core 2 duo, gig of ram, 120 gig hard drive), and had a nice form factor and a non-squished keyboard. (You must have decent cursor keys. You lift your hands from the keyboard to hit the cursor keys, if they're so narrow you miss this will become annoying and remain annoying. The other keys are less of a problem, you don't move your palm trying to hit them.) This means I can't avoid buying a copy of Vista, but at this point I'm desperate enough to stomach it.
Sunday night I unpack the thing. The kubuntu boot CD hangs during boot, at 5 different places on numberous reboot attempts. If I had to guess I'd say this thing is losing interrupts; it does start with a warning about a timer not setup properly by the bios. Monday morning, I've confirmed that Fade's windows machine can't burn an .iso to a CD (it can give me a CD containing the ISO as a file), but apparently burning CDs doesn't strain the x86-64 server enough to panic it, so that's ok. Knoppix boots up to a desktop on the first try (a squished 1024x768 on a cinema display, but it's way better than the current kubuntu live CD managed). And I run "lspci -v" and see... almost twenty "unknown device" messages. Oh beautiful.
Knoppix made it this far because lots of these devices are the genetic decendant of some chip family it can drive in a legacy emulation mode, but the wireless isn't on the list. That's ok, I have cat 5 around here somewhere. But the killer is that knoppix can't talk to the hard drive. (Nvidia makes a sata controller. Who knew?)
Now I remember why I always buy used laptops. If the sucker's been on the market for a year, Linux supports it great. If it just came to market Linux goes "wow, dude, look at all that hardware. I wonder what it does?
The Compaq box is covered with "please don't return it, call our help line instead" messages, so I call. I almost immediately get a nice woman with a thick indian accent who doesn't recognize the word "Ubuntu" but once clarified tells me "Compaq no longer supports the Linux" and that yes, I should take it back to the store despite what the box says.
Right, download the kubuntu "alternate install" disk. This boots (into a text mode installer). It has a driver for the broadcom wireless but it doesn't actually work (all ESSIDs it tries to set are one character long) but I plug in the Cat 5. It finds the network, finds the hard drive. It can partition and format the thing, and put Kubuntu on it. But will it have drivers? I already know the wireless is problematic, and the video is some new Nvidia nightmare that isn't even as well supported as the reverse engineered ATI stuff. And it's got all sorts of unnecessary crap like a fingerprint reader and who knows what else, none of which has drivers although most of which I probably don't want. If I'm going to return it, it would be polite to do so BEFORE formatting it. What are my options?
The http://dell.com/linux page still shows a 32-bit laptop on the first page. It's only slightly more expensive than the laptop/doorstop in front of me, and it has half the memory and 2/3 of the disk space of the said doorstop, but it's got Linux. Unfortunately, it's also got a 32-bit processor. It's 2007, why on earth would I buy another 32-bit system? Neither Intel nor AMD actually manufacture 32 bit chips anymore (outside of embedded lines like the Geode). I want a 64-bit laptop.
So I try their help chat, "Can I get a 64-bit laptop for Linux?" The little "submit" button does nothing in konqueror, so I download Firefox, fire it up, get in the chat queue, someone named Karen_C pops up alomst immediately... And X11 exits back to a login prompt. And when I log back in, firefox segfaults each time. Ok, the x86-64 system has done enough work for the day and needs a lie down. (I fought with this months ago, it's almost certainly a hardware problem but I'm not quite sure which.)
I plug the monitor back into Fade's windows box and use its web browser, but now the chat option's mysteriously vanished from the page I was looking at. I'm guessing the javascript doesn't recognize "Avaunt", which is the skin Fade uses to make IE bearable. Ok, there's a phone number. Call that. Get a directory tree with voice recognition. Give up trying to natigate it ("I don't HAVE a customer number you stupid flaming pile of...") and repeat "human" at it until it puts me in a queue for one, which takes about 10 minutes. Said human is a sympathetic tech support person who doesn't know anything about Dell's Linux offerings and transfers me to business sales when I say "home office, more or less". I seem to skip the queue this time and get a nice sounding man who positive BOGGLES at my question. A 64-bit processor? No, we don't sell those. Why would we sell those? 32 bit is going to be around forever, nobody needs 64 bit on a laptop. We won't sell 64-bit laptops for at least two years.
At this point I tell the guy I'm sorry his company's going out of business and that I'll go talk to Apple, and hang up on him. Ponder visiting the macintosh store, but they want over $1000 for their cheapest laptop and getting Linux to run on it's unlikely to be any easier than the Compaq. Maybe I should have asked to speak to the Dell guy's supservisor, but spending half an hour waiting to speak to an _idiot_ compared badly to Compaq's 30 seconds total to receive a polite brush-off.
I hit return and format the drive. Kubuntu installs, is happy all the way to the end, reboots... And the resulting system hangs during boot, just like the Live CD did. If it makes it as far as trying to bring up X11, it locks hard instantly with a black screen and no further hard drive activity. It only makes it that far about 1/3 of the time, it also hangs "loading essential drivers", attempting to configure the network, and sometimes elsewhere.
I fiddle with booting in "recovery mode" a bit, but at this point I've spent four hours monday morning wrestling with this box and I'm ready to say it won. I fire up the dell website and start pricing out one of their crappy 32-bit laptops... And the customization says Core 2 Duo!
Ooh! I can get a 64 bit processor in the thing! It's just hidden in the configuration options! I still don't want to have to wait for a system in the mail, but I want to support Dell's clumsy ham-fisted attempts at doing Ubuntu ("What type of Windows is that, again?" "The Linux kind." "Oh."), and if I can get the system I want... It's about $350 more than the Compaq but this one has an Intel 950 3D chipset and I got an extra battery and it should actually WORK, unlike said Compaq.
Time to return the doorstop. Beast Buy charges me a 15% "restocking fee", or $90. Can't say I'm surprised, although they would have charged me that if I hadn't reformatted the drive (and they have an install CD in there anyway, I saw it). I bounce off Chic-fil-A for lunch (if Microsoft's getting some of my money today, bible-thumping restauranteurs might as well have a crack at it too if they can serve me a sufficiently good sandwich), and then to atone for all this I go try to take Dell up on their offer of a Linux laptop that the darn manufacturer will SUPPORT.
I wind up hitting the storage space and digging out the old server I had in the closet in Austin, and a 17 inch monitor. It's only about 2/3 as powerful as the laptop that died, but I can squeeze Kubuntu on that and limp along until the new laptop makes it here.
Didn't get much else done today, though. And tomorrow, restoring backups!
My laptop died yesterday. More details if I can ever get kubuntu to boot on the new one (signs say no...).
Got up early today so Christian and Robin could drag Fade and myself down to the strip. Made the mistake of having breakfast food (a waffle with maple syrup) before noon. Now I remember why I don't do that; sugar crashed badly enough that I went back to bed when we got home.
Somebody emailed me trying to buy advertising on my old computer history page. I really need to find time to work on that again, but I've got too many other things to do these days...
The recruiter who emailed me earlier this week had easy questions (Linux geeks doing embedded development in C, local in Austin? Sure, here's contact info for three of 'em.) The one who just cold called me had a much more difficult question: he's looking for somebody who does perl for a job in downtown Austin. I don't do perl, so I don't know people who do.
Now that I updated my online resume to say booked through 2008, they're not asking me directly if I want a job (ok, they're strongly hinting but they're polite about it), but I'm still getting about a call a week.
Oh wow, this is surreal. That's the guy I was exchanging emails with at the Linux Foundation during my interview and the protracted contract negotiations. (He was also a significant contributing factor to the contract negotiations being long and protracted, and the entire process got unblocked shortly after he quit. Born bureaucrat, that man.)
Sigh, kmail's crashed on me again. For the crime of "cursoring around in an email I'm composing". Did I mention that the composer's auto-save function has some kind of race with its spell checker? (Which I TURNED OFF because it slowed down typing to one character per second which means I could easily get about two minutes ahead on typing and go get a beverage while it caught up. I had to rename aspell to aspell.bak in order to turn it off because the normal configuration stuff merely stopped it from displaying the results, not from eating CPU, but this is not my problem. Apparently, now when it tries to launch the spell checker and can't, sometimes it crashes. This is a guess, the crash dialog it brings up is totally useless for actually diagnosing what went wrong.)
Yes, I DO understand why Windows users put up with stuff like this. The alternatives are worse, unless I want to buy a Mac.
Today, the Google Alerts vanity search on my last name found two items: my livejournal's user profile page, and a seven year old linux-kernel post in a web archive that's been there since I made said post. Google Alerts is TOTALLY USELESS. Thank you.
And the coffee shop's nameserver is bad enough that I'm finally planning to sit down and write my own this evening. I've been just pointing it at Eric's, but for some reason during the move to Ubuntu it stopped handling recursive queries (I dunno, maybe that wasn't compiled into their version of bind). If you're only doing the UDP stuff, it's pretty straightforward...
From a freenode exchange, fallout from the recent system administration and a typo in a DNS entry:
[16:14] <landley> Let me know if new mail starts coming in... [16:16] <esr> Watching for it now... [16:17] <landley> You know better than that. :) [16:17] <landley> A watch mailbox never... dings, or something. [16:17] <landley> Whatever it is mailboxes do. [16:17] <landley> Recieves mail. [16:17] <landley> Possibly Receives, even. [16:17] <esr> I'm just running tail -f on the maillog :-) [16:18] <landley> It's still screwing up the quantum state of the mailbox. [16:18] <landley> (Queue Donald Southerland, "all these negative waves!") [16:18] <esr> IM IN YOUR QUANTOM BOX COLLAPSING UR STATE VECTOR [16:19] <landley> Kelly's Heroes meets lolcats. [16:19] <landley> Welcome to the internet.
Pondering Intel's new toy (which is probably a bit reason they're funding the Ubuntu Mobile effort). It's clever, but the reason the display usually flips up from the keyboard on a hinge is so the keyboard and display rest against each other. Sliding over the keyboard like that A) doesn't protect the display from being scratched when not in use, B) makes it easy for crud getting caught between the display and keyboard to jam stuff.
We probably need some kind of revival of the old IBM "butterfly keyboard" in order to fit a usable keyboard into something that folds down to the size of a cell phone. The proposed new palm thing seems to have two keyboards, one on the outside (to use while standing) and a larger one on the inside to use while sitting. That's expensive, but probably very clever.
Intel's speculation about creating a niche between the palmtop and the cell phone is stupid. The two are converging, and during this the cell phone will eat the palmtop because it's easy for cell phone to grow a bigger keyboard and display, but darn hard for a palmtop to hook up to the (still sadly proprietary) cellular network for internet access, not without permission from the phone companies that own the towers. Modern cell phones already have two weeks of standby and 8 hours of talk time, and the new generation of the suckers runs Linux.
This is why Apple's doing the iPhone. Just as IBM jumped on the micro bandwagon with the PC after noticing how the minicomputer undercut the mainframe, and everybody moved to the laptop since then, Apple wants to stake out the next form factor iteration while it's still possible to corner the market. The iPhone is positioned as "an iPod that makes calls", and since its introduction the iPod has turned from a dedicated music player into a video player and portable USB data storage device.
The big stumbling block (for years now) has been user interface, but between rampant cell phone text messaging on the part of anyone under 25 and blackberries for the older generation, I suspect that problem isn't really blocking things anymore.
Me, I want an arm palmtop that plugs into USB and presents itself as a block device. (When switched off, it presents itself as a USB flash key via ROM, so it's unbrickable. If you switch it on, the OS handles talking to the USB.) To boot, it jumps to the start of the first partition and starts running from there (so you just partition your device, copy your kernel to the first partition as a raw block device, and format the second partition to hold your files). It charges via the USB.
This is my ideal device. Display and keyboard and data storage are nice too, but fairly obvious. From a development standpoint, this is what a cell computer should be.
Dealing with passport people is INSANE. I wasted an hour over lunch at
the post office trying to get a passport, but they guarantee there's no way
I can get one through them in less than 4-6 weeks, and that's with the
"expiditing" bribe fee. But they say I could go through
a private "courier" service that will send it to Philadelphia and have it
back within 48 hours. Ummm... I was just 30 miles from Philadelphia until
we got back last night. Grrr.
So if I want to make an appointment and drive back to Philadelphia, what do I do? Call the number on the form... Which has been disconnected? This is just brilliant. Ah, I see, it was just overwhelmed and I got through on the fifth redial... And their queue to speak to a human is so long they won't even put me in it...
Oh well, I confirmed I don't need a passport yet when driving. This is canada we're talking about, if I wanted to go via back roads west of the great lakes, they haven't even got people checking. The current insane government's trying to put up a new berlin wall along the southern border, but not yet the northern one. And hopefully, they won't have time to...
Right, documentation. Working on hotplug at the moment.
Oh dear.
I'm writing up "the history of hotplug". This wasn't really the intent, but there's just so much backstory to explain WHY various decisions got made. (The special case of removable media is a couple pages all by itself...)
Ok, ubuntu has some _insane_ package dependencies. I just installed mailman on grelber (the server in Eric's basement), and buried in the enormous dependency list were firefox, gnome, the gimp, samba, vim, and metacity. Why does mailman depend on ANY of those? (I could see it depending on python, but this is a server tool. Come off it, guys...)
And yes, one of the anti-spam checks Eric added prevented me from sending outgoing email through that server. (Sender must supply a fully qualified domain name. Yes, even when you ssh into the server and send mail to 127.0.0.1.) Luckily, kmail has a configuration entry for that...
Apparently I haven't gotten any personal email in the past 5 hours. This is highly unlikely to be correct...
Ah, and another anti-spam check discards email from anyone using a charset like GB2312 (Alan Clark's email has that) or koi8-r (the charset used by Denis Vlasenko, the new maintainer of busybox). In general, I receive bug reports from people on the other side of the planet on a regular basis. Polite yelling ensues, and... Ok, he's reverted that one.
Eric is apparently tightening down the server until no spam gets through. I've always erred on the side of avoiding false positives...
I need my own server.
Spent the weekend at Eric and Cathy's in Malvern, still here through tuesday. One of the big things we did was replace the aging server in Eric's basement, which hosts landley.net and thyrsus.com. (The hardware wasn't so bad, but Eric wanted to get the last Fedora system out of the house, and we didn't want to take the old server down while setting up the new one, so we repurposed Cathy's old machine, which was recently replaced with a 64-bit system.)
Eric's first attempt to do this resulted in me losing half a day's worth of email and being unsubscribed from most of my mailing lists, so put the old server back and waited until I got here. So we set about setting up all the necessary services on a clean Ubuntu install, documenting what we did along the way. Setting up bind wasn't too hard, setting up sshd, even setting up apache (two virtual domains, plus mercurial repository support) was only moderately painful. I knew all that stuff, more or less. (Did it before, in the dim mists...)
But setting up a mail server was something I hadn't done before. (Although the smtp protocol is easy, mail servers are complicated nasty beasties largely for historical reasons.) I wasted HALF A DAY trying to figure out how the postfix configuration on the old box could POSSIBLY be routing my email (it was only binding to localhost!), before realizing that the old Fedora box was running sendmail. It had _been_ running Postfix back under Fedora 4, and postfix was still installed. But upgrading to Fedora 5 both overwrote some of the postfix config files with defaults, but it also reinstalled sendmail (part of the base install, after all) and since it installed it, it switched over to it. And the old sendmail config files were still there from before the switch to postfix...
And you wonder why we're getting rid of Fedora?
Oh, a fun little trick: if your machine isn't live on the net but A) has bind serving a domain, B) you want to check a virtual domain in apache or play with sendmail delivery, here's what you do. Add a DNS entry for the "net" or "com" net that points to localhost.zone, then tell the zone you're serving to return an A record of 127.0.0.1. Point /etc/resolv.conf to loopback. Then you can test your virtual domain and your mail server will deliver to itself locally. Also, learning from my experience, please point the zone's A record BACK to whatever IP it should be returning, because an A record of 127.0.0.1 confuses things rather a lot outside of a debugging context. (In general, removing debugging entries before putting the thing backc into production is a good idea.)
The frustrating part is that Eric's router (something verizon gave him when they installed FIOS) refuses to port forward port 80 from inside the firewall. It'll port forward all the other ports, but not 80. I'm guessing this is due to its' web management interface. This means I can't see if I put my email back up properly. Ordinarily the server I ssh to in order to test outside of whatever firewall I'm behind _is_ this one. I don't have access to TimeSys's servers anymore, the sourceforge servers are firewalled so they can't originate any connections, the DNS lookup for Mark's x86-64 server is failing (presumably because his dynamic dns hasn't updated in too long)...
Oh well. According to the apache logs, Yahoo's crawler is busily crawling my website. I presume this means the website is accessable from outside the router. If anything's weird on my server... Now you know why. :)
Meanwhile, I poked Eric about Doclifter, and he merged my patch and fiddled with some other stuff (which I already fixed but apparently wasn't in said patch), and hopefully this will result in a new release before I leave tomorrow. I'd like to say I poked at some other documentation stuff today, but to be honest this server administration crud ate most of the day. I'll make up the time later in the week...
Eric just wandered downstairs to enable some of postfix's anti-spam features. Be afraid. Be very afraid...
Eric got Fade addicted to "Battle of Wesnoth" on her Mac laptop. This cannot possibly end well.
Spending the weekend at Eric and Cathy's in Malvern, still here through tuesday. Ironically, not on my computer much while here. :)
After yesterday's big push I'm spending a couple hours working on toybox for palate cleansing reasons. Got it building with uClibc by the simple expedient of #ifdeffing out the strlcpy stuff, and then noticed that the "allnoconfig" version of toybox is larger than the "allnoconfig" version of busybox 1.2.2. The main reason is that the option parsing stuff is always built in (called from main() instead of the various applets), which makes toybox smaller when it scales to a larger number of apps, but makes allnoconfig look bad.
Caring about this one way or the other for the "allnoconfig" case is a bit dodgy, of course. It's a highly artificial benchmark. However, looking for an excuse, if no applets are built in that _use_ the command line option parsing stuff, then linking it in makes no sense. Of course I plan to reproduce the scripts/individual stuff here too and that would be the logical place to strip that out (which again wouldn't help allnoconfig of toybox itself)...
Anyway, I need a way to signal "applet doesn't parse command line options", and I have one: toys/toylist.h has "NULL" in the second argument of newtoy if there's nothing to parse. Unfortunately, this isn't quite right because lib/args.c copies toy.argv[] to toy.optargs[], and right now things like "exit" might use *optargs instead of toy.argv[1]. Ah, I see they don't. Well, easy enough to fix either way by adding else toy.optargs = argv+1;" to toy_init().
But what I _want_ is that when you build things like "sync" or "false" that don't use any command line option parsing at all, toy_init() doesn't call get_optflags() (which is currently true), but I want it to happen in a way that the compiler can figure it out and yank the code. Hmmm. I need a way of or-ing together all the toylist[i].options for i from 0 to TOY_LIST_LEN, in a macro.
Hmmm, I think I want a recursive macro. Unfortunately, gcc refuses to resolve macros recursively unless I give it the --traditional-cpp option on the command line which has 8 gazillion side effects. (Thanks again, GCC developers, for the non-orthogonal options.) How about a recursive inline function? (The sucker resolves to a constant, after all. The optimizer should be able to make it all go away...)
temp.c: In function 'main': temp.c:3: sorry, unimplemented: inlining failed in call to 'blah': recursive inlining temp.c:3: sorry, unimplemented: called from here
Great.
Ok, the fix I checked in does it.
On the Firmware Linux front, the default network setup should be:
ifconfig eth0 10.0.2.15 route add default gw 10.0.2.2 mkdir /etc echo "nameserver 10.0.2.3" > /etc/resolv.conf
And now, back to reading Documentation/*.
Ooh, somebody on uClibc mentioned deb-buildpackage and googling found this page which looks highly useful and I'll have to read it later. Busy today.
My attempt to pull an all-nighter to push out pending things so I can list them as done on my monthly report later today hit a snag when the cable modem went down at 3 am. Blah. There are still no 24 hour coffee shops with free wireless in Pittsburgh, that I know of. I might as well go to bed and get an early start in the morning. Yay california time, it means I have until about 8 pm eastern to send off my report...
It's amazing how hard it is to write up a "what I did this month" when so much of it is partially finished. Writing up "next I need to do this, this, and this" makes me go off and do it, and as a result I've written and then deleted entire sections on status because they became irrelevant.
One of the big ones is that I just got a http://kernel.org/docs web page, so the saga of figuring out how to update lots of small files in my people directory (apparently using /staging/upload.lock) just went poof.
Slight panic earlier today when my contact at the Linux Foundation emailed me asking for my report a day earlier than I expected, but he's ok with getting it tomorrow. I've got it half written up, but the real problem is I have so many unfinished things. I've _started_ a couple dozen different documentation projects. I've finished... well, finished is the wrong word. I can show results for about three.
My approach to this project so far has been largely investigative. Figuring out the scope of what I need to do turns out to be a largeish project in and of itself, one that will probably take the entire year. That said, I'm not being paid to spend a year figuring out what to do. It's just at the end of the year I'll finally know what I SHOULD have done, and at the end of an average day I've added 3 or 4 major documentation projects to my todo heap, some of which merge with others or change nature drastically a week later when I figure out "ah, this bit is a subset of that bit" and it all starts to make sense. I've scribbled a few paragraphs and an outline for each of them, but it's not going to make sense to anyone other than me unless I sit down and do a real write-up, or come up with a test program to confirm that what I'm recommending actually works, or I sit down and work out the actual syntax, or look up some piece of information, or read through three blog entries and a mailing list thread to distill out the five sentences of information people actually need to know...
And so, I'm on a "finishing things" rampage today, trying to get stuff done (or at least presentable, in some final form or other) and pushed out to the world so I can list them as accomplishments on my monthly report rather than in-progress investigations. No way I'll get through even a quarter of the heap, but for once the todo list shrinks rather than grows. :)
(Except that now I've been pointed at Documentation/firmware_class with the README and sample driver, and I realize that what I actually need to document is the /sbin/hotplug and corresponding netlink stuff, and that firmware loading is just a subset of that, but I need to test whether statically linked drivers really can load firmware out of initramfs via /sbin/hotplug called before /init gets launched, and that the ipw2200 driver in my laptop which USED to try to do this has moved to loading firmware at "ifconfig eth1 up" time, which is sad really... Yes, nailing this one point down could take all day. I've spent maybe an hour on it so far, altogether, scattered over the past week. This is not a new todo item, this is an existing todo item morphing int something else. Welcome to my world.)
So Matt Zimmerman got back to me about Ubuntu Embedded and pointed me to their wiki page. I've now subscribed to said mailing list, and joined said IRC channel.
BOGGLE. Ok, downloaded the mailing list messages so far, as a gzipped text file, accidentlly fired up vi on the gzipped file, and it decompressed it while loading it. Ok, vim officially has too many features. (It's sort of cool that it did that, but I think I'd rather have it fail reliabily than try to guess what I mean.)Working on documentation for firmware hotplug loading stuff. There are bits and pieces but what I really need to do is try it. The ipw2200 in my laptop needs firmware loaded at module insert time, and I'd really like to confirm that if I put firmware and an /sbin/hotplug in initramfs it can get called before init to service the firmware loading request of a statically linked device driver.
Three day weekend, mostly away from the computer.
So I want to create documentation from the menuconfig help in toybox. The Linux Foundation would also like me to turn the kernel's menuconfig help into web documentation. Both of these can share parsing infrastructure. The question is, what should it the results like?
In the toybox case, both interactive help and man pages are needed. (I can probably cheat on the man pages and make 'em html instead, but either way they're one file per command.)
Speaking of toybox, I just tried to build it with uClibc 0.9.29. Guess
what? uClibc is broken. It declares a strlcpy() when you define _GNU_SOURCE
and then #include
Wrestled with "make htmldocs" some more (specifically Documentation/DocBook/Makefile). Sent a patch to the list to do xhtml-nochunks (which is easier to print, easier for me to read, and makes an rsync to kernel.org suck less), and while I'm at it do a better index.html making use of the titles of each section as a description, rather than just a one word name.
I really need to document the toybox infrastructure. I've been away from it for most of a month and I'm head-scratching at some of the details (mostly having to do with command line option parsing, which is admittedly tricksy but so is getopt).
So QEMU for the PPC still isn't working for me. However, it occurs to me that the hercules emulator has been known to run Linux. And I should really dig up armultor and get that working, too...
Ah, nope. Hercules needs C++ to build and I never bothered to install a C++ compiler. (Just C.) Thought about firing up apt and doing so, but it's not really worth it...
Yay. I got paid. Ok, now to justify it...
Finally got the "request for birth certificate" mailed out. Should have done that last week but I was hoping I could find one of the two I already have. Of course if I'd done it last week (while I was in Austin), I might have it by now...
My daily Google alert vanity search on my name (sort of a microkibo thing) boasts "This as-it-happens Google Alert is brought to you by Google." All four of the entries it found this time have dates: the newest is from december and the oldest is from 2003. Thanks, Google!
Grabbed the linux manpages and I'm running doclifter against them. It looks something like:
wget ftp://ftp.win.tue.nl/pub/linux-local/manpages/man-pages-2.50.tar.gz tar xvzf man-pages-2.50.tar.gz cd man-pages-2.50 make MANDIR=../woot allgz cd .. wget http://www.catb.org/~esr/doclifter/doclifter-2.3.tar.gz tar xvzf doclifter-2.3.tar.gz cd doclifter-2.3 ./manlifter -I `pwd`/../woot -h
Unfortunately, that doesn't actually work:
I also had to install a number of things to make doclifter work (like xmlto). I forget exactly what, but it's the same stuff I needed for "make htmldocs" in the Linux kernel. Next time I do it, I should write down what it was. :)
Extensive pestering on the topic (and a patch for three of the above issues) sent out to Eric's mailbox. Probably a doclifter 2.3.1 release forthcoming. :)
Meanwhile, I got about 2/3 of the sucker converted and online, at least on my website. I need to refresh the kernel.org copy of that, but A) The URL for it is insanely long, B) I still need to figure out the "polite" way of updating 8 gazillion small files without triggering the bad mirroring behavior they warned me about.
After having used Mercurial for most of a year now, I believe I can say with some authority: git's user interface sucks rocks.
I'm trying to update. One web page says "git rebase" (which I do, and it backs out my local changes). Another says "git pull origin" which I do and it says it's already up-to-date. But "git log" says the last change was yesterday, and the web page says the last change was 2 hours ago. I tried pull with the full URL, it made no difference.
[an hour and change of asking stupid questions on #git on freenode goes by...] Ah. Turns out there weren't changes, I had a stale web page open in the browser. Right. In other news, the user interface for git changed drastically in the past 10 months, and compiling it from source is a largeish pain. Oh well.
Struggling with the "inline" doc so I can put something final up today. Except my questions about it have spawned a thread on linux-kernel I need to wait for the end of...
The OLS schedule is up and my tutorial has an hour and change gap in the middle of it for the lunch break. Oh well, I can work with this...
I poked Jeremiah about why I hadn't seen any patches to FWL from TimeSys since I left. (Wondering if they had their own fork.) He's been too busy to work on it. Why am I not surprised? I suppose it's probably no longer strategic for them, or something...
So I'm moving files around in the Documentation directory trying to inflict some kind of order on the thing. It's not easy. There are lots of files in the top level Documentation directory that shouldn't be there, My initial urge was to create an "obscure" directory for hardware most people don't have, but that's everything from "zorro.txt" (the bus in the Commodore Amiga) to . So far I've got:
mkdir -p arch/amiga mv zorro.txt arch/amiga mv arm cris blackfin parisc powerpc s390 x86_64 uml arch mv sx.txt stallion.txt specialix.txt rocket.txt riscom8.txt computone.txt \ hayes-esp.txt moxa-smartio serial
Which is enough to push upstream and relatively uncontroversial, I'd think. But there's no sane way to do that as a patch. I'm going to have to install git, aren't I? Sigh...
Went through two boxes of papers this morning looking for either my old passport or my birth certificate (to get a new passport with). Thanks to the Office of Domestic Surveilance, I need a passport to go to Canada for Ottawa Linux Symposium at the end of next month. World's longest unguarded border but if you're dumb enough to be _honest_ about it, you need to spend a month on paperwork. That's too stupid for words...
Yup, have to install git. But it _is_ possible for it to send sane diffs to move files. Doesn't do so by default, though.
Hmmm... Ctrl-alt-escape brings up the "kill a window" function, turning the cursor into skull and crossbones. Attempting to click on the background to make this go away is NOT a no-op. Just FYI.
Back in Pittsburgh, at the Te Cafe. I still hate Pittsburgh (this became quite apparently as I hit the edge of the city limits, driving back), but I like the Te Cafe and Squirrel Hill's not so bad.
Jeremiah pointed me at a writeup about building gentoo against uClibc. It uses an obsolete version of uClibc, and I'm not really targeting gentoo anymore, and when I asked on the #gentoo-embedded IRC channel Solar said that it was one of the things that had inspired himm to start the Gentoo Embedded project. During the conversation I also stumbled across this, which could be cool.
The vanity search I have on Google alerts finally brought up something I hadn't already seen, a picture of me making liquid nitrogen ice cream from Linucon 2004. From here.
Got sunburned walking to pick up the car. (Mark got his car and picked me up from near the Northcross mall. I walked about 80% of the way and went "ok, now I remember why I do this at _night_". But the UT shuttle doesn't seem to run when there are no classes and it's a Saturday. Who'da thunk? And the regular busses won't take a debit card...
Still haven't gotten the first check from the Linux Foundation, so still living off savings. (I wonder what address they sent it to, and when?) Updated my resume to mention my new day job, though.
Signed paperwork for an apartment in Austin, starting August 1st. Time to drive back to Pittsburgh until then.
Broke down and got the air conditioning fixed in the car. It'll take them until tomorrow to get it done.
So I need to write up better documentation for the Linux kernel firmware loading API, and especially how to make it work for statically linked drivers via initramfs. This may involve writing some code to get the kernel to do what the documentation says, but I'm game. :)
I'm going to try to get as much of this can written and submitted next week, so it counts towards my "achievements for this month". Also get the website updated, and the scripts to keep it automatically updated, and finish the triage of Documentation, and the "make htmldocs" makefile changes (whether or not they get submitted upstream, I need a clean patch I can apply to each new version)... You know, I have a huge amount of work put into this already but it's all half-finished things that don't show results yet. Deadlines are good for finishing things and piling up results. Yay end of month reports...
Mark's graduation today. Unlikely to get much else done.
Mark's father was in town and we all went out to the sequel to Mr. Sinus Theatre in the evening. The old Mr. Sinus troupe broke up (Owen and Jerm left, John recruited two new comedians under a new name). The new group is called "Master Pancake Theatre" and they were making fun of Pretty Woman this time. I like that movie, but they're good enough to mock even good movies. (They did the original Star Wars once.)
Muchly email, late in the day. hg 49575 built. I am reminded that the qemu guys were discussing an arm interrupt problem, but I can't find it on linux-kernel and it's apparently not fixed yet. hg 49540 built.
I'm starting to get requests to do documentation emailed directly to me. (And I haven't even seriously launched the effort yet, this is still prep work. I publish and announce what I have around the end of the month.) I'm all for it...
Didn't get any sleep at all last night, drove up to Waco with Mark to meet David instead. David had to leave early to avoid Dallas morning rush hour, so he was going to get to Wacko around 8 am. Mark asked me when we should leave and I went "eh, 5:30, 6" with the appropriate hand wiggle, and then decided that the question had been answered precisely. So at 5:36 precisely, we headed out. Got back around 1 pm and I collapsed into bed until 9 pm.
Still tracking down the arm-scsi-qemu breakage in the kernel: hg 49500 build break, 49750 built ok but had the scsi problem... (This is obviously a background task due to the long compile cycles between each attempt, not currently automated by anything other than "cursor up" and a long shell statement with lots of && signs in it). 49650 built...
A pretty girl named Nikki told me she liked my Linux t-shirt, and we discussed distributions for a minute or so. (She was using Mandrake but friends were pointing her at Red Hat. I suggested Ubuntu, of course.) Even in Austin, pretty girls noticing you because you have Tux on your t-shirt doesn't happen more than about twice a year, and remains noteworthy. I'd have given her one of my business cards, but I ran out _last_year_. (I should do something about that.)
I miss Fade.
Today I learned it's possible to create a hard link to a symlink.
$ ls -l total 0 lrwxrwxrwx 2 landley landley 6 2007-05-16 18:54 boing -> walrus lrwxrwxrwx 2 landley landley 6 2007-05-16 18:54 walrus -> walrus
Hey, a symlink is a real filesystem object with an inode. You can create hardlinks to any non-directory inode (and _that_ limitation's only there to prevent some nasty infinite loops in things like "find" and "fsck").
Also, man 2 path_resolution is incomplete and slightly wrong ("." and ".." haven't been in the filesystem for a while, they're handled by the vfs and actually maintained as properties of a process)...
Beautiful. Continuing yesterday's hg bisect, it goes straight from unable to build on arm because it doesn't know what cmpxchg was (hg 48076) to the build break due to "undefined reference to dma_free_noncoherent" (hg 48077). Ok, find the _end_ of that run (the checkin that fixes it so it builds again), and then backport it to the checkins in between to get something that builds, and see if it boots.
Sigh.
Ok, day job stuff: reading about logfs, I should do a taxonomy of filesystems...
Andrew Morton said:
> + buf = kmap(page); > + ret = logfs_write_buf(inode, index, buf); > + kunmap(page); kmap() is lame. The preferred approach would be to pass the page* down to the lower layers and to use kmap_atomic() at the lowest possible point.
I should ask him about that. I don't _think_ it's listed under Documentation, unless it's in the "make htmldocs" stuff I haven't read most of yet.
Somebody piped up on the tinycc mailing list with an interesting problem in libtcc (a part of tinycc I've never used), but my reply to them was spam-blocked because they don't like the static IP that Eric's FIOS line has had since last year. This is not my problem. (I don't mind when somebody's spam filter eats a message and I have to resend, but inaccurate blocking based on IP address is too stupid for words. No, I'm not going to go chase some brain-damaged realtime blackhole list. People who use that kind of thing obviously don't want to hear from me, and I have nothing to say to them.)
I wrote some documentation for __always_inline today. Not quite sure where to put it, it's in the kdocs index for the moment.
Also tracking down what the heck broke arm scsi under qemu. Current tip (hg 56757) has it. 2.6.20 (hg 46373) did not. Let's try -rc4 (hg 50954)... Bad. -rc2 (hg 50211) Bad. -rc1 (hg 49644)... Bad. Yup, it went in during the merge window. hg 47373... works. hg 48373... build break. 48400... Build still breaks. This could take a while.
I have overloaded my poor laptop so badly. Swapping its' guts out. I'm pondering a script to regularly try various things (build and boot the current -git kernel nightly on all the FWL platforms, and make htmldocs from same). Next laptop needs more memory, more hard drive space, and a 64-bit processor. (And a clean keyboard with a working C key. And smaller/lighter would be nice.)
So I want to not only put the "make htmldocs" output on kernel.org, but I want to update it regularly. Currently it's producing xhtml, with lots of little files, which is a pain to update on kernel.org because it's really not set up to update lots of little files (it drives the mirroring scheme nuts). Instead I want it to use xhtml-nochunks and produce one html file for each "book", which is easier to read when the suckers are as small as they are right now anyway. Of course getting it to work turns out to be outright fiddly. Have I mentioned that basing kbuild on gnu make instead of something sane (bash? python? Heck _awk_...) was a mistake? Oh well.
Taking a break from indexing other people's stuff to write some documentation of my own. (Palette cleansing, maybe?) This'll be the first major chunk that the Linux Foundation gets the copyright to, rather than me, but oh well. Not like I was thinking of writing a book someday anyway. :)
Ok, wound up working on my OLS presentation instead. Well, that's useful and it is a weekend...
Hmmm. In tcc, I moved the i386 files to an i386 subdirectory yesterday, adding a dependency of i386/* where appropriate. And this is broken, because when I do a "make" followed by a "make install", the install rebuilds the project because .o files showed up in the i386 directory and they're new. (In theory they shouldn't be newer than the tcc executable, but the dependencies are multi-level and something intermediate is getting confused.)
I hate make.
Spent far too much of the day grinding through linux-kernel and the kernel's mercurial changelog.
Decided it was time to put some of my scribblings for my 2007 OLS tutorial up online, and as a result I moved the recordings of Ottawa Linux Symposium 2006 into an OLS subdirectory so I could create an ols2007 directory to hold the new stuff. (Didn't want to pollute the root directory of my website.)
Argh. After all that, I only signed the last page of the contract (I also had to sign page 3) so now I have to fax it again.
I've been pointed at the LF travel fund which is unrelated to the fellowship and I'm not sure I qualify (I don't consider myself a "rock star" of the open source world, I'm more of a backup singer, or maybe a roadie), but I suppose I could ask if they'd fly me to the big australian thing everybody keeps talking about. Or maybe one of those European ubuntu meetings. But for OLS, I can drive. Assuming I get a passport in time.
So, since yesterday there were 330 new changesets in the linux kernel mercurial repository (a mirror of the git repo I find easier to track), and over 1000 messages to linux-kernel. I'm am trying valiantly to keep up, but OW. Drinking from a firehose here, need to do it for a week or two before I remember how to skim this context properly...
Reverting to 2.6.20 made armv4l work again. Obviously, I need better testing of each image after upgrading packages. I also need to bug the linux kernel mailing list to figure out what happened, but first try the current repository version...
Hmmm... When I went out this morning to fax my contract to LF (last night's attempt to make this whole "fax" thing work was... interesting), I left a build running on my (suspended) laptop. Now that it's resumed, I don't remember what exactly it was trying to prove?
So, Mark's taking me to see "28 weeks later" tonight. I'm ambivalent about zombie movies (I liked "Shaun of the Dead" and "Polar Express"), and didn't see the first one, but it seems worth a try. We've both seen Hot Fuzz. I'm vaguely interested in seeing Spider Man 3 (Emo Parker!) but Mark's already seen that.
Ah, build is using a snapshot of the linux kernel to see if the SCSI thing got fixed for armv4l. Right. And the build broke because the squashfs patch no longer applies. Wheee...
Yay, I have a draft of the LF contract I can sign. Now I need to confirm the dates and find a Kinko's...
Digging up my old documentation notes. Wow there's a lot of this. Back to plotting strategy: I think the best thing to do overall is create a master web page, index everything I can find, identify the holes, and start filling them. Every one of those sub-clauses is a can of worms, of course. :)
I am reminded that in Texas, you can't keep the little tiny sugar ants out of a first floor room except by making the place airtight. I'm reminded of this at Trianon, a wireless coffee shop on Far West. I'm also reminded that Tolber1 bought in this country doesn't taste nearly as good as the fresh ones Thomas Geiexner brought over from Europe with him.
Re-reading the Statement of Work attached to the LF contract, one of the most time consuming things I'm expected to do is stay current on Linux Kernel, so I've started at January and am trying to catch up to the present. This could take a while. (I've done it before, but I'm waaaay out of practice.)
And for variety, during the last few days on the long walks I've been taking in the evenings (5 hours sunday night, 4 hours monday night, took a break last night but I'll probably do it again tonight), once it gets too dark to read my TCC source printout I've been working out my OLS tutorial outline. On paper at the moment (you try walking with a laptop), but it's a start. (I need a new laptop.)
Hmmm... armv4l broke, booting it hangs resetting the scsi bus in a loop. Now is this a problem because I updated my qemu cvs snapshot, or is this a problem because I went from 2.6.20 to 2.6.21.1? Hmmm, re-updated my cvs snapshot of QEMU, and it's still unhappy.
The gelato place at Burnett and Braker has free wireless. (And really nice potato things, although their beverage selection is pepsi products and the Mountain Dew isn't diet.) Gelato (with however may t's it's got... oh, it's on the sign on the wall: one) is apprently italian for "extremely fatty ice cream", although Mark insists it's italian for "not actually ice cream but something else". I should look it up.
Links: Confessions of a Recovering NetBSD Zealot: http://www.onlamp.com/pub/a/bsd/2006/09/14/netbsd_future.html
So Eric Raymond finally upgraded his last machines to Ubuntu, including the mail server my email is on, and now I can't read my email. Why? Because he upgraded to Ubuntu 7.04, not 6.06, and the /bin/dash thing broke my filchmail script. (I guess the lesson here is "don't try to use shell scripts on an Ubuntu system, do it in Python or something instead, it'll be less likely to randomly break.")
Ah, it's more broken than I thought. I haven't gotten any email since 9 am, apparently his new postfix isn't recognizing my domain. Fun. And my website is down too. I have no email and no website. I should really get my own server...
[some time later] Eric wound up reverting to the old machine. I'll visit in person after I get back from Austin and we can swap machines then, but for the moment I've lost a day's worth of email (the server being down is not the same as the server being up but saying the user doesn't exist), and I have to re-subscribe to all my mailing lists. Oh well, that's system administration for you...
Of course this would happen when I'm on the road. :)
Ok, David Wheeler's now broken up all of the non-windows hunks of grischka's big "build gcc with tcc" patch, and I've applied them all except for the macro strigify behavior (which David agrees needs a rewrite, and which I'm rewriting). Still reading through the tcc source code...
Also, I've caught up on my livejournal friends list for the first time in... Dunno. 2007, probably. It's been a while.
I'd write about the long IRC session I had with David and tglx and Mirell, and the phone call with Linux Foundation, but I need to go get food. (Yay being within easy walking distance of a Wendy's that's open late.)
Weird Al's song "Do I creep you out?" was apparently based on a song called "Do I make you cry?" I have learned this because it's playing over the speakers at Wendy's, and I am having a MASSIVE attack of cognitive dissonance over WRONG LYRICS... I really liked the Weird Al version. The original is straight and sappy and my brain is not wired to enjoy it, but keeps going "ooh, good song... wait."
I am at Metro! I have chai!
Oh wow, I've missed this.
I see that Beth was laid off from TimeSys on Friday, while I was driving cross-country. It's possible she was the last survivor of Manas/David's old engineering group. (Did Jeremiah report to Al back then? I forget.) Talked to David a bit about trying to fill the vacuum TimeSys's collapse would leave at the low end of the embedded Linux market. We have the skills, the question is do we have the funding to get big enough fast enough? We could always accept venture capital funding, but that comes with _ropes_ attached, not just strings. I'd much rather fund from operations, but that's a chicken and egg problem...
Oh well, it'd be Mark and David shouldering the bulk of the work for the next year. I expect to be busy, if I can ever get this darn fellowship contract signed. :)
Spent a few hours catching up on teh intertubes, then went to Fry's, which also refused to sell me a laptop without Vista. (I'm disappointed in Fry's now. I'm going to put Kubuntu on my next laptop, which should be 64-bit. I'm only mildly annoyed by buying a copy of XP I won't use and will immediately wipe, but I _refuse_ to add to the Vista sales figures if I'm not using the darn thing. Guess I'm going with Dell.)
Walked back from Fry's, which took 5 hours. I needed that. I need to do more of it. I'm also just a little sore now. Pondering how to find a swimming pool to get more balanced aerobic exercise, and thinking star ranch.
And uClibc 0.9.29 is out! Yay! I've updated firmware and am building test versions. With that change and the powerpc thing (need to finish that) it qualifies for another release.
Stopped at a "Flying J" on I-40 in Tenessee that had a big billboard about free wireless. They lied, their login screen wants me to purchase it. I still feel the same way about pay wireless as I do about pay toilets. They're spending more on electric lighting than on internet, is that going to become coin-operated too? I wouldn't have stopped here (and bought lunch) if not for the false advertising.
And now you know why my blog's mostly migrated from livejournal to a text file I an update on my laptop. And from SVN to Mercurial...
Off to Austin.
Juggling things to head to Austin, probably head out tomorrow. I think I can get everything but the oil change done today.
A month back I set up a "google alert" on my last name, because doing a vanity search on myself always pulls up such strange and interesting hits (and goes on for many, many pages). So far, it's fed me back my blog entries (not this one, just livejournal), and a few 5-year-old posts I made to linux kernel and such. Last week I found found a dozen or so mentions of my name in Penguicon con reports out on livejournal, but the Google alert hasn't found one. So very deeply impressed...
(Note to self, finish reading this after rebooting to unfreeze Konqueror AGAIN.)
Nah, that drama belongs on livejournal. Yes, I know I wasted most of last week tracking down Penguicon con reports. As I said, I got a bit excited...
New yardstick of irony: George Orwell's house under 24-hour surveliance from 32 of britain's state run closed circuit TV cameras.
Also, there's a patch to QEMU that can export an emulated USB device to attach it as a client to the host system. Yes, you can run a piece of software that hotplugs a totally fake USB device into your computer. Life is good.
Merged more of the patches David A. Wheeler's chipping off the old grischka mega-patch. Serious kudos to both of them, I should figure out how to tell gcc to build with tcc so I can properly test this...
Ok, messing with the powerpc build in Firmware Linux, my first attempt at a fix (copying include/asm-powerpc/Kbuild to include/asm-ppc/Kbuild) didn't work because asm-ppc apparently hasn't got ipc.h. Not entirely surprised, but it falls into the category of "relatively clean fix" by my strange little definition here.
The _simple_ thing to do is patch my script to specify a different arch= to the linux kernel source code for "make" and "make install_headers". But that's ugly. The next most obvious thing to do is patch the kernel Makefile to test for arch=ppc in make headers_install and reset it to arch=powerpc instead. But this hits exactly the "declarative/imperative" mess that makes me hate Make; when does variable assignment happen and how do you restrict/sequence it? My first attempt was:
headers_install: include/linux/version.h scripts_basic FORCE
@[ $(ARCH) == "ppc" ] && ARCH=powerpc; \
if [ ! -r $(srctree)/include/asm-$$ARCH/Kbuild ]; then \
echo '*** Error: Headers not exportable for this architecture ($$ARCH)'; \
exit 1 ; fi
$(Q)$(MAKE) $(build)=scripts scripts/unifdef
$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.headersinst obj=include
But that doesn't work because the assignment redefines the shell variable (in a child process), not the make variable. But in the $(Q)$(MAKE) lines the subshell has exited and we're back to make invoking new processes.
As to assigning make variables, conditionally, in targets, I dunno how to do that. If it's not indented, it's not run conditionally. But when it is indented, it's a different syntax (shell commands) that doesn't let me assign to make variables. (Did I mention I hate make?)
Keeping multiple balls in the air. As soon as I get the LF contract signed I'm going to have to focus a bit, but that's not a bad thing...
Heading to Austin this weekend, working out the details...
Oh good. About time. I need a new laptop.
David A. Wheeler has been emailing me about my tcc fork, and we're trying to get grischka's old mega-patch broken up so I can merge it. I'm all for this. The hard part is figuring out which bit does what.
I'm also fighting with powerpc/ppc in Firmware Linux. George Boudreau fed me an updated config file a few days ago that builds a powerpc kernel that qemu can boot up to the "can't find root=, panicing now" bit, which is highly cool. Except that it make headers_install needs to use arch=powerpc and the rest of the build needs to use arch=ppc, and this is a clear bug in the linux kernel. (Although the bug is actually that arch=ppc is old and obsolete and all the functionality is migrating over to the new arch=powerpc, but in the mean time some bits work in the old implementation and some work in the new, but neither does everything and that sucks.) I don't want to teach my build scripts to work around a platform-specific kernel bug, but it's not exactly a trivial fix, either. Or wait, maybe...
Hmmm... Shouldn't be too hard to make the kernel's "make headers_install" recognize arch=ppc as a synonym for arch=powerpc. The question is, why didn't they do that already? (Yeah, the want people to migrate, I know...)
A comment I made to a blog post where someone pointed out that they stay at Microsoft because they see that company as the best hope for figuring out how to engineer large scale development projects:
Didn't the Linux guys already figure out a workable process for engineering large scale software? The Linux kernel is almost 300 megabytes of source code, and manages to have quarterly releases. (And each of those quarterly releases now merges more lines of code than went in during the entire 2.3 "development series" that turned the 2.2 kernel into the 2.4 kernel over a period of about 3 years. To increase scalability the kernel developers went to a four-tier structure of developer/maintainer/lieutenant/architect and they also make use of modern distributed source control ala bitkeeper/git/mercurial, which is _nice_.)
Developers develop for the same reason bloggers blog ("pixel-stained technopeasant wretches" that we are). And 99% of it's crap (sturgeon's law), but fighting off sturgeon's law is an ancient business model, from book publishers going through the slush pile to what project maintainers and Linux distros do. Cream skim, cherry pick, polish it up a bit, and put out the next issue. The editor's job is to say no, to reject 99% of the slush pile (sometimes with notes on how to fix it, which newbie authors don't always realize is a gold-plated invitation to resubmit).
What Linux has done is nest the layers. Individual developers get three layers of editorial filtering and cleanup to get their code into the kernel: first from the maintainer for a given file, then from subsystem maintainers (I.E. lieutenants), then from Linus himself. And then the distributions have their own editorial passes, which make local changes to get the release out but feed those changes back upstream through the normal submission process. It's all extremely decentralized and scales marvelously.
There are only four business models on the internet: create content (bloggers), host content (youtube), index content (google), and let users interact with each other (guild chat in World of Warcraft). In Linux, we have individual developers dumping patches onto mailing lists, maintainers do an editorial pass to make packages they dump on sourceforge, and distributions like ubuntu do another editorial pass selecting the packages to put out distro releases with. The chat part (freenode and lots of mailing lists) comes in with both development and tech support.
This scales well enough for Ubuntu to put out two complete releases a year, with daily security updates in between, on less than 0.1% of Microsoft's staff. I know Microsoft is trying to figure out how to do that itself, and you tell me it honestly believes it would be coming up with anything new if it did. But this really isn't the case, Linux is doing it now.
So I've got the beta of kubuntu 7.04 installed under qemu, and there's so many things wrong with it I need to just wipe it and start over. Konqueror can't display any web pages, at all. (Whatever the site is, "could not connect", but I can wget there just fine.) The dhclient3 that runs on startup accomplishes nothing, I have to kill it and run "dhclient eth0". It's sad.
I should do a release of my tinycc fork... done.
Wow. So readlink() is more broken than I expected. If the buffer to hold the name is too short it won't report an error, it'll just silently truncate. And it doesn't say how long it SHOULD have been. The only way to find the correct length is to iterate until you've allocated a _bigger_ buffer than it needs. That's a horribly designed API.
I also notice I'm fighting the compiler. The obvious way to do xrealloc() is ala xrealloc(&ptr, newlen) but since the type of ptr can be arbitrary (anything from char * to an array of structures) xrealloc wants to be "void xrealloc(void **ptr, int newline)" but if you then feed it a char ** as the first argument gcc complains about a type mismatch. And I can't typecast in the call ala &(void *)ptr because that breaks strict aliasing. I can declare it as just a void * but then I have to add another line to xrealloc() to copy and typecast it into a variable of the type I actually want to use, PLUS the type you pass in is then confusing.
I miss the days when the C compiler didn't try to be clippy-style "helpful".
Ok, toybox now has readlink. Wheee...
Sigh. It's 10:30 pm and I'm finally just turning my laptop on today. Unfortunately, all the coffee shops around here are closed. I'm seriously Looking forward to getting back to Austin.
Inexplicably tired, but it's been raining here and I'm still just getting over the whole "winter, must hibernate" thing. Took a nap, might possibly have made it worse.
I finally read "Kitty Takes a Holiday", the book Fade got me at Penguicon. The ending was disappointing, the last third the book spent a lot of time building up to... nothing. The old storytelling maxim that if there's a gun over the mantlepiece in chapter 1, it will be used by the end of the book? This is because the narrative doesn't follow the protagonist into the bathroom: you're on a word budget and you have to make each one count. Spending half the book on a series of wild goose chases, introducing dozens of new dangling plot threads, and then pulling a massive cop-out that resolves NOTHING and makes the second half of the book might as well not have happened, is not a satisfying ending.
Specifically, you can skip from page 217 to page 287. Nothing in between makes the slightest bit of difference. The books 303 pages long if you include the epilogue. I think I blame the editor.
Decided I've spent enough time collecting Penguicon con reports, and posted what I had. There are more, but I think a week of this is enough.
I also promised somebody in irc a working mdev (fixed for the symlink issue in the 2.6.20 kernel) in toybox by monday. I should work on that. (The hard part isn't fixing it, it's retracing the history to make sure no copyrights I can't license under GPLv2 snuck in there. I know there's no Bruce-tained code in this one, but it's still a bit of a job adapting it to the different framework.
I have spent waaaaaay too much of this week collecting Penguicon con reports. I have 42 so far just from livejournal. (Let's see, somewhere between 830 and 850 attendees, what percentage of them are on livejournal...)
Coming back from Penguicon I have books I want to read, dvds I want to watch, stuff to read on line, technical projects to complete... Enough to keep me busy for a couple months, I expect.
Far and away the most flattering photo of me ever taken. Oh yeah. (That's Fade in the foreground, holding the book with the cover Randy Milholland whipped up for us in photoshop. Underneath was his copy of Gurps Basic which we got back to him during his Burrows and Bunnies under Miskatonic University campaign. Yes, combining a setting loosely based on Watership Down with a setting based on the Cthulu mythos of H.P. Lovecraft. I saw some of the drawings for this beforehand.)
What if Penguicon's not really a good con, but merely a conspiracy of a little over 800 people all _pretending_ it's a good con? How could you tell?
Watching the Dr. Who dvds we got for the wedding. Tonight's episode can be summed up as "queen victoria vs a werewolf".
I've mentioned before my theory that the best science fiction shows tend to have the science fiction element get you into a situation at the start of the episode, get you out again at the end (but not before), and be of little or no help in between. Sliders and Quantum Leap are classic examples. Star Trek had the perennial problem that the Enterprise was too powerful, and they keep finding excuses to disable it (transporter malfunction, has to leave orbit, darn communicators stolen again).
This is why, in Dr. Who, the Tardis has no armaments. It's transportation (although sufficiently unreliable in the steering department you don't want to use it mid-episode), and a refuge (if you can get to it), but that's it. When they used the Tardis and the time vortex as The Great McGuffin at the end of the ninth doctor's season, they made it cost the ninth doctor his life because it can't become a regular plot device without destroying the series. Keeping the sonic screwdriver sufficiently low-power has been a challenge too. (It should be versatile, but should never resolve actual plot points. In the tenth doctor's first episode, the christmas tree and evil santas were scared away because the doctor woke _up_, not because of the sonic screwdriver.)
I'm really looking forward to the episode with Sarah Jane.
Grelber's down. That's the machine in Eric's basement that hosts my email and my website. Once again: Mercurial is a marvelous program. (Ah, it's back up. Possibly Eric was just rebooting it.)
More Penguicon stuff: a flickr photo group, con reports from the con chair... I should post an index to the Penguicon LJ community.
Lots of stuff to catch up on, and what am I doing? Reading Penguicon con reports. Hey, I got married on saturday. Not that it's related, but it's a marvelous excuse this week...
(I'd be working more seriously if I had a contract. I have recruiters calling me, another one friday morning while I was at Penguicon, who called back Monday and has been forwarded to David and the new company. But the fellowship thing has yet to produce a contract and money, so I'm considering my start date to be whenever they get me something I can sign, and the work I've already done before that's a freebie.
Now THAT was a fun con.
Went to lots of panels, got married, bought some stuff, cookout sunday night... I'd write a con report but I'm too tired. Luckily, other people beat me to it...
I believe Fade and I are the only people to be married by Steve Jackson so far. Eric Raymond was best man. Randy Milholland whipped up a quick "Gurps Matrimony" book cover in photoshop, which he and Steve signed afterwards. There was cake. (And then more cake later, we got three that were blue green and white only because we didn't think to make the white one red. Then when we had dinner with the relatives at Red Robin across the street from the con hotel, they brought us a chocolate cake and balloons. It was GOOOD chocolate cake. The hotel ceiling was covered with sharp pointy bits and took out two of the baloons on the way back to our room. I also think I left my electric shaver in the bathroom when we checked out.)
I recorded panels, and all six of the recorders came back at the end: none were stolen. (Yay!) Of course on several occasions people tried to help (returning a recorder to ops that was duct-taped down to the table, for example). I think that the first 80% of Eric Raymond's "economics of open source" panel was missed because the thing was counting _down_ (which it does when it's playing back, not when it's recording) and it ended near the end of the panel, when I restarted it. I think the previous panel (which removed the tablecloth it was duct taped to, for reasons I don't know) stopped it while removing the tablecloth and then thought they'd restarted it but actually set it to play back what it had recorded, at zero volume. Sigh. Pity, it was a good talk. I missed even attempting to record a few others, and I make no guarantees about the sound quality, but this year we should have something to put on the web. Steve Gutterman has them now.
The chaos machine was a little too dense for my tastes this year, so many people were adding complications that it was all squiggles by friday night. It needs to be bigger and more stretched out, meaning they need more infrastructure (blue tubes) and more space. (Not that anyone was complaining about the complexity, it's sort of the point, but I like to be able to see it more easily.)
The sunday night cookout at Sal's place (the head of Aegis Consulting, and the guy who runs sword camp) was FUN.
I came back to 150 megabytes of incoming email, most of which was of course spam. Spamassassin and kmail spent most of an hour grinding through it, the last 15 minutes of which involved my laptop swapping its' guts out. (I remember when half a gig of memory was a lot...) Now to _read_ it...
Hmmm, apparently the author of Dropbear has switched to the Mac. Gee, what a surprise... Eric Raymond and I had a panel about that on sunday, possibly the recording will go up soon.
Note: I posted a slightly different con report on my livejournal.
I've asked to be removed from the Penguicon concom list. It's just no fun anymore. I'm still attending and giving a few panels this year, and I may still attend in future years, but I'm done with organizing any content for it.
Talking to a new guy about the fellowship contract. He should have something for me next week.
So, a todo item that's bubbled near the top is to switch the firmware build over to using initramfs to mount squashfs, meaning the ext2 image can be hdb. Of course the way I want to do that is with toybox driving the initramfs, and it's nowhere NEAR ready at the moment, but one of the nice things about not doing this for a living anymore is I'm not on any kind of deadline. So, back into the thick of toybox, my copy of which doesn't currently compile because mke2fs is halfway finished. Right, clean up just enough of that so I can get the pieces to fit back into the box...
Hmmm. Complicated mess I left off in the middle of, isn't it? The hard part's figuring out what order to do things in, but I knew that. I think all TT.entries should be filled out by main() before calling init_superblock().
The other todo item is following up on Jason "nuclear" Wessel's help (sorry, but that's how I remember his name) getting ppc working under qemu. I shrank the .config he gave me to a miniconfig (about 250 lines, I can clean it up after I get something to boot). I've also updated my qemu snapshot, and I'm rebuilding my ppc cross compiler as I type this. (I notice that qemu just grew alpha emulation. Whee, another target platform to try. Gotta be more relevant than sparc...)
Nope, no matter what I do qemu segfaults trying to decompress the image (or possibly immediately afterwards). This isn't "the thing running inside qemu doesn't work", this is "the emulator exits with a segfault".
Hmmm... This code snippet:
if (sizeof(this->st.st_size)>4)
in->dir_acl = SWAP_LE32(this->st.st_size >> 32);
Produces a warning about the size of the shift being greater than the size of the datatype. But I'm _TESTING_ the size of the data type in the previous line (I added that to make the warning go away). The sizeof becomes a constant and the comparison is between two constants so the dead code eliminator zaps the assignment. And yet the warning is still generated, which is sad.
I hate unnecessary #ifdefs... Great, #ifdef doesn't understand sizeof(a->b). Grr. Ok, time to get creative...
Sigh. Ok everybody, repeat after me: Assault is not theft. Perjury is not theft. Working as a salaried employee is not theft. Giving donations to charity is not theft. Infringing copyrights is not theft either. If you want to talk about which it's _like_, feel free. But stop talking about "theft of copyrighted material" unless somebody stole a physical book, ok? Sheesh.
So whatever that problem with KDE being totally frozen (repainting, but ignoring all keystrokes and button/scrollbar clicks) resurfaced this morning. It's one of the tabs I have open, I've been launching KDE without net access so those tabs open as "no such website" and then I can hit reload in 'em later one at a time when I get around to viewing them. Well I shutdown and restarted my laptop, and it reloaded all those tabs, and I suspended and resumed and it resumed frozen. Wheee.
I'm waiting for spamassassin to finish processing my email download before I can ctrl-alt-backspace X and fire up kde again. (I could just re-open konqueror but it'll open empty and I can't figure out any other way to get my list of open tabs back than to have kde give me the list it remembered at the last desktop shutdown.) While waiting I went through the list of pending open email replies (I'll open a reply window and minimize it to remind me to reply to that later), and accidentally closed the main kmail window when I was done, despite the fact that what I'm waiting for _is_ kmail. Of course kmail crashed (it brings up the KDE crash handler window about half the time I close it, I think it gets confused trying to pack some of my enormous folders ala linux-kernel), but the download-with-spamassassin process is still chugging away happily in the background expanding the temporary mbox file it's downloading into, and I've hardlinked it to another name in case some part of this proccess decides to delete it (unlikely, but hey). So I can force kmail to manually process this inbox when I restart the desktop.
Yes, I understand exactly how windows users cope with crappy software on a daily basis.
Ok, that's my limit of blow-ups with Matt Arnold, so this is my last year having anything to do with Penguicon. I may attend in future years, but I'm not pushing any more content into it. Not my problem. If I feel the con-running need, I can re-launch Linucon (under the name "Y'allnix", because Usenix is up north) when I get back to Austin.
There is a uClibc 0.9.29-rc out! (Ok, it's actually an _rc which is kind of silly, but that's nitpicking. It's out! YAY!)
I think the konqueror freeze is related to the netscape plugins. Not sure.
More glacial progress on the contract for the fellowship.
On the road, delivering a cat. Couldn't fire my my laptop (the battery lasts about seven minutes). Wrote down lots of bullet points for a rant on why to replace Make with shell scripts, with a pen on the back of the envelope my taxes were done in. Need to find that to write it up.
Fade and I are indulging in a very brief road trip to Arkansas to drop off the last kitten. I'd hoped to use my laptop for lots of this trip but although the adapter's working fine, I forgot that the cigarette lighter in the car isn't. Sigh. (I don't think it's the fuse, I checked the fuse.)
And the battery lasts ten minutes...
Breakfast at Uncle Sam's Subs (have I mentioned yet that putting a fried egg on a cheesestake was a stroke of genious?) and then off to the Te Cafe to get some work done.
Apparently, the PowerPC version of Linux doesn't quite match the spec. Page 38 of the spec says that return values from functions and system calls and such should be stored in r3. Page 44 of the spec gives the register layout main() can expect, putting argc in r3. Since exec() is a system call, it leaves r3 0 when it succeeds (and argc is in r1), but the dynamic linker shuffles things around to where page 44 of the spec says it should be so all is well. (Unless you're running a statically linked program, in which case crt1.o has to do this fixup stuff for you.) The Linux way of doing it is explained here.
Except that qemu application emulation is making exec return the register layout from page 44 directly, which is not what Linux actually does, which is why my statically linked "hello world" was segfaulting on exit. I don't know if the dynamic linker would figure out it didn't need to shuffle the registers around, or whether it would scramble them trying to fix things up. My "hello world" didn't look at the arguments to main().
Sigh. And it turns out that the people who have been making working prep kernels were using ARCH=ppc and I've been using ARCH=powerpc. I don't understand the difference between them, other than that ARCH=powerpc lets you select prep but won't build a zImage for it, and running "make oldconfig" with ARCH=powerpc on a .config from ARCH=ppc spits out pages of unknown symbols and then prompts you for pages of new ones.
Another thing to ask when I get around to "stupid question time" on linux-kernel.
Backing up to the 500 gig USB hard drive I bought a while ago turns out to be a bit of a pain. The external drive is vfat formatted, and I'm leaving it that way so Fade can use it too. So I was trying to back up my home partition with "tar" and it kept stopping early; took me a while to realize that vfat can't make a file larger than 4 gigabytes. (Oh yeah, I remember those sort of limitations...) So now I'm doing a cp -a into a directory, which is going to be horribly inefficient (but hey, 500 gigs) and is printing a warning every time it can't create a symlink. Oh, and there's a "can't create directory, file exists" because I have an upper and lower case version of the same directory name... Ah, and now it's complaining it doesn't like files with colons in the name, kde made rather a lot of those in its konqueror cache entries...)
It's a partial backup, at best. I pondered "ok, can I make multiple 4 gig files, loopback mount them, and raid0 them together" but for now, a partial backup is better than leaving the drive empty, and I've got other things to do today.
Ah! "tar czf | split -b $[65536*65535] - home-split.tar.gz." Of course, it's so simple. (Ducks.)So, the City of Pittsburgh has an income tax, which TimeSys never deducted from my paycheck, so I owe the city a couple thousand dollars for the priviledge of watching people spit on the sidewalk and driving around potholes for the past year. This is on TOP of a sales tax as big as Austin's, and presumably property tax for people unfortunate enough to own something here. I hate this city so much. I'm so glad I'm moving out soon...
Installed The Ubuntu 7.04 beta into a 4 gig image file under qemu, so I can reproduce a tcc bug that only happens under glibc 2.5. In the process I tried to create a mercurial version of the qemu cvs repository (because dealing with a cvs repository is icky and it went "boing" on the last update because I'd made changes to the code and forgot to back them out before updating, and I dunno how to clean up the mess with that toolset other than deleting the directory and re-checking it out), and of course I found a bug in tailor when I did this that's going "this cvs repository has 2000+ changes in it! Create a fresh hg repository, now copy the last 256 changes over! Ok, done!". So I'm still using a somewhat out of date qemu cvs snapshot at the moment, which is ok for x86 but not for tracking down what's crazy with ppc support.
If, by the end of that paragraph, it's wandered through enough different tangents (debug tcc, update qemu version, debug tailor, debug ppc...) that you forget where you started... Welcome to my world. That was about 20 minutes. It's not that I don't get stuff done it's that I spend so much time trying to close out tangents to get back to stuff I stopped in the middle of that my pile of half-finished projects continually grows...
Speaking of which, I should dump the todo list up top. You wonder why I have gazillions of unfinished todo lists and keep starting new ones to track the stuff I'm _currently_ doing, in hopes I can finish it before the tangents take over...
It's easier with deadlines, which let one spray for tangents.
So Firefox market share is now above 30%. Interesting.
I've put together a uClibc-0.9.29-pre1.tar.bz2 and uploaded it to my home directory on uclibc.org. It's just plain been too long. I'm trying to pester Mike into releasing it officially so I don't have to, because I fell for the "I'm not maintainer, I just review/commit patches and do releases" thing once and I'm not falling for it again.
I tracked down the page that's been making konqueror eat 100% CPU. (Ok, technically Konqueror eat 2% CPU and making X11 eat 100%, even when the tab wasn't actually being displayed. Great design there, X11.) It's Slate's blogging the bible series. (Not that page, but the ones under it, probably the navigation javascript thingies in each-sub page.) Yeah, it's a Konqueror bug for allowing javascript to eat CPU at all when the page isn't being displayed, but still.
Ordinarily I open a few more tabs than I manage to read and close every day (especially when they're long series like that one), and then software suspend will silently fail and my laptop will reboot instead of resuming (or sometimes not so silently, the "X11 is trying to acess the keyboard hardware directly" infinite panic loop's always fun. Of course right now the kernel's got an endless loop going (about 3x/second of "sda: assuming buffer cache, write-through.", and every half-dozen of those it throws in a "Buffer I/O error on device sda, logical block 0". Makes all the text consoles unusable until I reboot. This happened because I plugged my ipod shuffle into the USB to charge it (telling the pop-up NOT to mount it), and then unplugged it again. It'll keep inifinitely looping until I reboot, trying to access the removed device it was never asked to look at any data from.
Did I mention that portions of Ubuntu 6.06 "Long Term Support", and Linux in general, are buggy? I'd build and install a new kernel (potentially fixing software suspend and the moronic "route USB through the scsi layer despite a total lack of SCSI devices in any system I've ever owned" thing, but unfortunately the ipw2200 wireless driver has a binary firmware blob I'd have to upgrade and that's black magic I've never gotten to work. (Plus intel had an insulting contract-o-matic on its download website for that last I checked: no, I'm not agreeing to any terms and conditions in order to use the hardware I purchased, thank you. I didn't when I installed Ubuntu and got the old versions. First sale doctrine. Fair use. Go away.)
Finally got the x86-64 versions of the cross-compilers built and uploaded for the 0.2.1 release of FWL.
Didn't get out to the tax guy today. Didn't get through the kernel Documentation/ review (got farther, though). Moved kdocs into the www directory so it gets backed up as part of my web page upload. (It's the kind of thing where having to go to a whole-laptop backup from a week ago would suck a lot.)
Huh. According to AMD's product outlook page, all their server, mobile, and desktop processors are now 64 bit. They've still got embedded 32-bit stuff (Geode), but nothing outside of the embedded space. I wonder how official/complete/up to date this information is? Of course they've been touting the energy efficiency of their 64-bit stuff for a while, so there's not much reason to keep it out of laptops. (In the embedded world arm still has better power/performance than just about anything...)
Finally set the watch I got at the waterfront last week to the correct day. It's easier to spot these things when one has a commute to an actual office.
Spent the day dealing with the IRS. A 1099 for work I did christmas of 2005 and got paid for in January 2006 went to the wrong address, and I foolishly thought the IRS might know the right amount to fill in on the tax form, since it had to be reported to them in January. (The logic here is that as long as the numbers I give them match what they _have_, they should be happy.) But apparently, they don't actually have this information. Giving them information in January doesn't mean they have it in April. (The excuse was "300 million people". My reply was "Computers: look into it", but it wasn't helpful.)
This was a block from TimeSys, and I thought about saying hi but wound up not doing so. Walked home for the exercise (5 miles). Stopped at a restaurant and a coffee shop along the way, and got some work done but neither had working internet access. Ah, Pittsburgh. Spent the evening cataloging bugs in Spider Man 2 for the Praystation 2. (If you're charging jump while you get a heal token, it doesn't register. It's possible to trigger a "green question mark over pedestrian" quest while fighting bad guys from one of the "purple dot on your map" quests; of course both use the square button. I may have to restart because I've got a save game where the objective list says I've completed all the objetives for the level (the funhouse in mysterio's bedroom after you beat up the statue of liberty; don't ask, it's not as stupid as the sports arena thing) but the next chapter simply didn't start. Fade assures me this second spider man game isn't nearly as buggy as the first one. Did Microsoft buy Activision while I wasn't looking?
I also had a printout of the tcc source code with me in a 3 ring binder to read on the walk. I'm abou 15% of the way through. There's rather a lot of it. If the documentation fellowship ever results in actual money (their lawyer is looking at comments I made on the contract), I need to buy a printer so I can print out lots of the documentation I'm reading. Much easier to spend several consecutive hours reading and taking notes when it's on paper rather than a screen. Plus I can get some exercise.
Sigh. I finally got Firmware Linux 0.2.1 out this morning, and of course a bug report came in right after that. (Actually it was sitting in my inbox but the poster wasn't subscribed to the list so I had to go sort through the pending "probably spam" entries via web administrative interface in order to see it.) I think it just means a bug that didn't manifest for me didn't actually get fixed by my first try, either. (I built everything and it built, so...)
I'm building the x86-64 versions of the cross compilers right now, on Mark's machine. Considering that there are now eight of them, this could take a while...
Somebody wrote an x86 assembler in javascript. As Fade says, "that's faintly wrong".
Yeah, it's about time to add a big endian mips config, isn't it? Ok, guess #1 on how to do that... Hey, it worked. Checked in.
Paul Graham has declared Microsoft dead. Interesting. Unfortunately, he seems to be seeing mostly Apple these days (he mentions Linux once, as an afterthought). This has to be giving Dell and HP warm fuzzies, but so far they seem determined to hold on to the Vista anchor all the way down. (I wonder what AMD thinks of this?)
My slog through the kernel's Documentation directory continues. I'm going to have to ask lots of stupid questions on linux-kernel once I get to a good stopping point. (The whole PNP thing: is that just legacy ISA crud? I don't think PCI uses this at all, and I don't think the modern hotplug/udev stuff has anything whatsoever to do with this. It claims ACPI support was pondered but not implemented.)
Fade and I went to the waterfront today. (It's the place in Pittsburgh that has parking.) I bought three little lecture recording things (essentially a 256 meg flash stick with a microphone, possibly an ARM processor in there too). They're too cheap to have a USB plug so the only way to get the data back out is to play it back through the audio out, but it can record 30 hours at its' idea of "high" quality, which is more consecutive hours of panelage than Penguicon has. Best Buy restocks on Thursday, and if I can get three more I should be able to record six tracks worth of panels.
Yay, my darn command line fix patch finally got applied somewhere between 2.6.20.4 and 2.6.20.6.
Sigh, no it didn't. Wrong patch. Put it back, yank the _other_ one...
I should get 0.2.1 out one of these days. Really... :)
It was only a matter of time until somebody started a class action suit over "Vista Ready" stickers.
Going through the kernel's Documentation directory, categorizing the mess. Does a bitmap graphic of a penguin really count as documentation? How many multiport serial drivers does the kernel _need_?
Web pages viewed in Konqueror regularly go nuts eating 100% CPU. You'd think that tabs that aren't being viewed would be suspended and not able to eat CPU, but that would make too much sense. You might there there was also a way to see which of the 40 open tabs I've got is the one eating all the CPU, but again, that would make too much sense.
Konqueror's been a nice web browser for years, but it's crippled by the design decision to bolt it to KDE. People who don't want to use KDE can't use Konqueror, and thus attention goes to Mozilla (the current regeneration of which is Firefox, but I'm sure somebody will declare that too bloated and fork off yet ANOTHER iteration, given time). Firefox is a pretty sad browser in comparison (ten times as much code to accomplish the same thing), but it has the advantage that you don't have to deal with the whole of KDE in order to patch it. (I.E. Firefox is actually _less_ code if you take into account the elephant of KDE on Konqueror's back.)
The sad part is of course that Konqueror doesn't show up as the CPU hog, X11 does. I'm guessing Konqueror's reloading a double buffered bitmap in a loop or something similarly silly.
I'd like to single out Documentation/sched-domains.txt as an example why documentation written by somebody too close to the material is a bad thing. Read that (skimming is fine), and then show me where it answers the question "What is a scheduling domain?" No fair cheating explaining it based on prior knowledge, point me to where that thing says it. (Hint: it doesn't.)
Stared at documentation for rather a lot of today. Didn't really accomplish anything. There's a lot there and it's a bit hard to figure out a good place t insert the shovel. I'll probably break down and write the bits that interest me until I have some traction, and justify it later. (It's amazing how demoralizing the "statement of work" has been.)
In the meantime, I'm getting poked about tcc from a couple directions. Huh. I don't really care whether it builds on windows or not, but when I do:
echo '#include "tcc.h"' > blah.c gcc -c blah.c -o blah.o
Then A) it shouldn't give me warnings, B) "blah.o" shouldn't be 20k. Time to clean out tcc.h...
I have an account on master.kernel.org! Woot! (Insert fanboy bouncing.)
So, putting stuff in the staging directory, then moving it. Hmmm...
Ok, my sad little start at this is now online. Not the world's greatest URL to find docs at. I'd prefer if it was more like kernel.org/docs which A) shorter, B) isn't tied to one specific person's login handle. But hey, it's a start. I've taken the landley.net/kdocs link down so as not to confuse google.
So in order to get the kernel to build a zImage with ARCH=powerpc, you need to select one of CONFIG_PPC_{PSERIES,MAPLE,IBM_CELL_BLADE,PS3,CHRP,EFIKA,PMAC} and no CONFIG_PPC_MULTIPLATFORM doesn't cover it although it would seem to. There are no help entries for this.
Subscribed to linux-doc@vger.kernel.org, an ancient mailing list that I plan to rehabilitate as part of the fellowship. Also subscribed to git-commits-head and mm-commits although keeping up with either one is likely to be hit and miss at best. We'll see...
Tweaking the slushpile. Today I added
for i in *.html
do
x=`sed 's/.*HREF="\([^"]*\).*/\1/' $i`
rm $i
ln $x $i
done
to the htmldocs preparation invocation thingy. Why make a wrapper web page when
you could just as easily generate a symlink? I should push that upstream, but
the whole htmldoc generator needs to be rewritten in something that isn't
perl. (I dunno, bash maybe?)
Downloaded the mercurial version of the Linux kernel current development branch today. According to du, the database is 410.8 megabytes. That's pretty impressive.
Heh, the linux.com article on RaveHD makes me nostalgic. Once upon a time I worked at BoxxTech, a company forced into doing Linux by its' customers, against its' will. (I was head of Linux development, and my boss, one of the founders and on the board of directors, told me he hated Linux because he thought it was "communist". I didn't stay there long.) Anyway, that's where I learned the difference between modeling, rendering, video capture, editing/compositing, and playback.
Modeling is creating new 3D content, done interactively by artists with large monitors and expensive graphics cards. It's also working out scenes with them (somewhere between puppetry and the audioanimatronics in old disney theme park rides). Rendering is running through such a 3D scene to create a series of finished high-resultion screen-ready images, via batch jobs submitted to compute clusters, often running overnight.
The alternative to modeling and rendering is video capture, which is digitizing video from the real world, coming from a camera or broadcast signal, turning it into a big file. (The hard part is saving gigabytes per second at real time, HDTV is something like 150 megs/second uncompressed, and movie quality to show in a theatre is even bigger. This is where all that realtime stuff comes in, 30 frames/second and you can't drop even one.)
Once you've got the video clips into your computer, editing and compositing is a huge chunk of work requiring a good user interface. Editing is the process of splicing them together in some order (possibly with fades or wipes or some such, but mostly you don't want to draw attention to the cuts), and it's generally treated as a simple case of compositing and done by the same software, although technically they're slightly different. Compositing is overlaying multiple images simultaneously to produce a finished image, everything from blue screen effects through making a light saber glow, to putting a picture in the TV your characters are watching. This is how 3D and captured video get combined with 3D characters like jar-jar.
Then there's playback, which is just video capture in reverse with all the same issues. Ok, capture has to deal with block allocation in your filesystem, but no serious filesystem used in this space has had a problem with that for years: the simple case is we have ONE process writing LOTS of data, deal with it. Don't glitch every 5 seconds because you went to update metadata and seeked the heads out of position so the writes can't keep up. It's one big linear write, preallocate ALL THE SPACE to this file and then give it back when you close the file. Of course if you have eleven cameras capturing your big special effects shot from different angles in slow motion, things can get a little trickier. :)
Ah, that was fun. Too bad the people I did it for were crazy, but this isn't unusual...
Yesterday I found out how to build a PPC zImage, today I'm trying to figure out how to boot said PPC zImage under qemu. This would be easier if the person doing the PPC work in qemu wasn't spam-blocking the IP address my mail server's on. (Because it's not at a big ISP, therefore it must be a source of spam. Sigh.)
Heh. Ok, the news is out. (From last Tuesday's meeting, the bit at the end of the day where I should have predicted it because it's so obviously what they'd do.) Atul is the new CEO of TimeSys. Ok, one more post about TimeSys to get it out of my system, then I move on.
Atul is the guy that Adams Capital brought in as an advisor months ago, apparently based on his ability to "play dumb" and ask lots of questions. His exposure to embedded Linux was that he'd previously worked at a company that had used some of it (not any of our stuff) so he brought an "important customer perspective". Because at that point something like 2/3 of timesys was sales and marketing (and most of the rest were people like Dan the CFO and Beth the sysadmin/janitor), so obviously our main weakness was lack of familiarity with our customers.
Atul's been around for a while, on and off since shortly before the new year I think. I didn't mention him much here because I didn't think he was important. The second half of the big engineering meeting when Thomas and Ron were in town devolved into "explain everything to Atul, at great length, using small words". It got to the point where Michel was explaining stuff to him. Michel is the french salesbeing from california. He's apparently a good salesbeing, but his windows laptop is a virus-infected spam spewing monster, and Beth once had to fix his blackberry due to what she described as user error. Watching _him_ explain a technical issue to Atul (using an explanation he'd picked up earlier in the presentation Atul wasn't following)... I had to leave the room for a bit.
He's now CEO.
The reason I didn't exercise my options on my last day is because Ed hired a new full-time Windows system administrator. (Beth was training him when I walked into the server room.) Ed also wanted to outsource all the Linux servers.
So the engineering department reports to a man who never used Linux before this job, has Windows on his desktop machine, and is expanding the use of Windows within the company. He reports to someone who once had to have some technical issues explained to him by a notoriously nontechnical member of the sales staff.
They're hiring.
On my last day, on my way out after staying late into the night cleaning off my desk (throwing out papers, untangling cables, etc), I turned out all the lights.
So: Documentation. I should get up a website.
Oh dear. I decided it would be a good idea to clean up my linux-kernel mail folder, which now has well over a year's worth of messages in it, so I made a "2006" subdirectory and told kmail to move everything before new year's into it. It's now been eating 90% of the CPU time for half an hour. None of my own windows where I was composing email messages are even repainting. While I'm not a big fan of threading this is just SAD. (The problem is if you do mass actions in a folder that has focus, it does the action one message at a time and tries to update the folder's entire state each time. Going "empty all trash folers" from the pulldown menus is almost instant: unless you have the trash folder open. Then it can take an hour. (There's really no excuse for this, especially since it's so insanely CPU-bound to move data from one file to another.) Unfortunately, many actions can't be done without viewing the folder. Especially since right clicking on a folder to open its menu ALSO left-clicks on that folder.
Hmmm, drill down to .kde/share/apps/kmail/mail/.inbox.directory/.linux-kernel.directory and "grep '^From ' kernel-2006". (wonder why I love mbox files and hate maildir? Now you know...) Ooh, it's up to October 1. Ok, almost done then.
When doing "make htmldocs" in the kernel, kernel-api.html takes an inappropriately long time. I should probably find out why.
Yay, flu is finally being less evil. Having a water bottle next to the bed last night helped a lot, especially not being woken up at 6 am by dehydration.
I was away from my computer until 9 pm, don't expect any development in today's entry. I doubt I'll even catch up with my email before going to bed.
My goodbye TimeSys lunch (sort of a tradition among the alumni, Piggy used to organize 'em) was all you an eat sushi at the Pacific Rim, on Murray. The group included Piggy, Christian, Robin, Beth, Jeremiah, Mike, Garrett, Fade, and myself. They have reeeeeally good eel.
For some reason, konqueror's frozen after resuming from software suspend. Not "won't repaint" frozen, it's doing that fine. And ps says the process isn't blocked. It's just ignoring keystrokes and mouse clicks. The scrollbars, tabs, pulldown menus, URL entry bar... all frozen. All I can see is the portion of the web page it was showing before suspend. The window manager's still working fine (it'll change its title bar color to accept focus, the little pulldown menu from the top left border icon lets you switch desktops and such). And resize will actually re-wrap the page so the rendering engine in konqueror is fine, parsing the data tree. As far as I can tell, all the QT widgets the thing uses are frozen.Hmmm... I was hoping if I logged out of kde, konqueror would get to save its tab list (full of "right click, open in new tab, look at that later" items). Unfortunately, it seems that a single hung app can veto the kde logout indefinitely. I could always ctrl-alt-backspace to exit X11, but the purpose of logging out was to get konqueror to snapshot its' state so it re-opens the same tabs next time I launch kde. No idea if it hung before or after doing this. Oh well, I've given it ten minutes...
Ooh, you can go to the K menu and "end current session" again, and then it kills hung processes. Nice feature. And yes, it saved the list...
I'm getting really tired of this cold. (Note: I just got email from Ed pointing out the Center for Disease Control page on distinguishing colds and flu, clarifying that there is such a thing and I almost certainly have the latter. My question is, "is there more than one flu"? There's a cold, but _the_ flu. Can only one person have it at a time? If so, I feel honored...)
Ooh! Horst Von Brand sent me a mercurial patch to my tinycc tree! I got an excuse to learn how to use "hg import", which worked! Woot.
I need noise cancelling headphones (thanks to David Lang for the link). It's not just that the Te Cafe is currently playing The Bee-Gees "your everything", it's that the guy in the back corner is singing along with it, loudly.
I've also been pointed (via a LWN comment) at three more links with background info on Stallman's attempt at a hostile takeover of glibc a few years back. I suppose the way the FSF works isn't any siller than the crusades (spreading the philosophy of turning the other cheek, by the sword!) and the current silliness in Iraq (imposing democracy by force). It's no secret I find the FSF's "how dare you be so ungreatful for all we've done for you" attitude a bit off-putting, and any definition of "free" that doesn't leave me free to ignore them seems a bit hypocritcal at best. I used C long before I used gcc, and back in the late 80's I was part of an independent open source community (the WWIV .mod community) that had nothing to do with Stallman or even Unix. They rise of the internet had nothing to do with them, and Linux came out of Usenet and Multics. I personally worked on alternative command line tool implementations in BusyBox for three years; it's not hard.
Perhaps my real objection to the FSF is that it's actually a religion. If you disagree with Linus or the OSI, they'll think you're wrong. If you disagree with the FSF, they think you're a bad person.
The "low fat biscotti" is actually quite nice. This is the first full day I've gotten to spend at the Te Cafe in weeks; my last week at timesys I came in to the office every day.
Oh, here's another relevant link. Plus Stallman's attempt to hijack ispell, see here and here. Oh, and then there's his signature in this letter to The Register:
Sincerely, Richard Stallman Principal developer of the operating system often inaccurately called "Linux"
And you wonder why I try to keep a wall between myself and the FSF?
I'm not accomplishing much today, other than a little computer history research and inviting someone to be a GoH for next year's Penguicon. I didn't even reply to the half-dozen pending emails from the Linux Foundation guys (although in my defense, I don't technically start that job until tomorrow, or possibly monday). I hate the flu.
Here's an interview in which Stallman explicitly explains his freedom-as-religion position.
Ooh, note for toybox, from #gcc on freenode:
[17:06]
Which has been a todo item of mine. (And has to be a shell builtin like cd since it affects the current process and descendants.)
Huh, reading through the pigdog.org interview I notice by page 6 one of my big disagreements with Stallman. He doesn't seem to believe it's possible to lose ground. I've seen this due to the hardware you work on being obsoleted, due to the rise of new data formats you don't support (which show up and spread faster than you can reverse engineer them), and due to things like World of Warcraft and this year's Turbo Tax which didn't even exist five years ago but are now indispensible to a lot of people today. I guess that last one is "interaction with services" and a better example would be Google Maps (which the Konqueror in Ubuntu 6.06 still doesn't support, nor do writely or gmail quite work with that. Probably fixed in 7.04, but that's my point).
A big part of the 64-bit paper was that sometimes tactics are of more immediate importance than strategy. It's great figuring out tomorrow's chess move, but if you're in a burning building while you do it, the board can burn away while you ponder.
Last day at TimeSys. After today, I never have to go into downtown Pittsburgh again! Yay!
If anything, my cold is worse than yesterday. Nevertheless, I'm putting out a 0.2.1 release of Firmware Linux, on the theory that tip has been a significant improvement over the last release for a couple weeks now, and my todo list is no reason not to just checkpoint development at a "known to work" version.
Of course PPC isn't "known to work". Somebody asked on the qemu list if anyone has ever gotten a ppc kernel to boot under qemu, and nobody's replied in a week. I should reply with a "not that I know of"...
The fix I made to uClibc a couple days ago wasn't correct, and I reverted it yesterday. It's quite possible that the problem is actually in qemu's application emulation, although the powerpc linux kernel "make" in 2.6.20 is dying because it doesn't create a zImage file, but then expects to find it. So apparently, you can't build a powerpc kernel without specifying a non-default make target. Good to know. Maybe 2.6.20.4 fixes this? (checking...) Nope. Ok, ping linux-kernel...
Ed insists I have a flu rather than a cold. There's a difference? (This just doesn't come up in Texas. We have don't have a "cold and flu season", we have allergy season instead.)
I've decided not to exercise my stock options. Today was my last day to do so.
I put up the video of my OLS 2006 presentation on initramfs. (It's a bit of an abuse of busybox.net, but that's where the rest of the presentation materials are and it's not any bigger than the Red Hat 9 image I put up for regression testing.) Just finished listening to it: am I really that scatterbrained in person?
I actually have a directory with audio and video from 21 OLS 2006 panels which I found as a bundle on bittorrent some weeks ago. (Somebody pointed me to it on IRC.) I wonder if the OLS organizers are interested in hosting any of this?
So yesterday's sore throat and general irritability has turned into a full blown cold. Runny nose, coughing, sneezing, joint and muscle aches, the lot.
So I looked it up and OLS says "Final papers must be formatted using the Linux Symposium LaTeX template. No other format will be accepted. If you send your paper in another format it will result in your offer to present being revoked."
Since I've never used LaTeX for anything ever, and the rules don't _require_ a paper for tutorials, this pretty effectively prevents me from sending them one. Might not prevent me from writing one, but if so I'll just do it in HTML, put it on my website for the attendees (maybe print out a few copies at Kinko's or something on my way up for handouts), and just not tell the OLS administration about it so as not to annoy them.
Printed out the gdb manual last night and am slowly reading through it. Debugging PowerPC assembly dumps is not my idea of fun, but trying to figure out the order of execution of the code that runs after main() exits isn't fun either. (Remember __do_global_dtors_aux() from last week? Finally got back to debugging that and it gets to the end and exits happily. Something after that is dying. So I'm looking at "powerpc-objdump -a hello", wondering where I can find a good reference on PowerPC assembly (google would know), and pondering actually making use of qemu's -g option...
Yesterday I had some questions about health insurance. (The best option for health insurance during the fellowship seems to be to use COBRA to extend my current health insurance, but I'm getting married next month at Penguicon: can I add Fade to my COBRA when I get married, after I quit? Turns out: yes I can.) I visited HR in the process of asking this, and picked up my "so you're leaving TimeSys" letter, which reminds me that various clauses of my contract remain in force after I leave, such as the non-compete clause.
This is hilarious. "Does this mean you DON'T want me to continue working on Firmware Linux after I leave? I've been thinking of spending more time on ToyBox anyway..." Ed assured me this was not the case. Not that they own either project (the first predates my involvement in TimeSys by years and I can prove it, the second was never remotely work related), and not that non-compete clauses are even enforceable in Pennsylvania or Texas anyway. It was only in the contract in the first place for intimidation value (maybe that should have been a hint). And my new day job is doing documentation, which I haven't done here in most of a year anyway. (I stopped to avoid getting annexed during Laurie's year-long power grab where everything was a marketing function, which resulted in more than half the company reporting to marketing shortly before Joe reshuffled things. Yes really, Beth counted heads.)
Apparently I'm grouchy today. My guess would be a combination of the kitten waking us up a half-dozen times through the night (her midnight kitty crazies lasted until at least 5 am, although in here defense the food dishes were empty so maybe she thought the world was ending), and getting new glasses.
The new glasses are fallout from the doctor's appointment a week ago: as long as they were examining my eyes they wrote down my current glasses prescription (mucho diopters), and the new glasses I ordered with that arrived yesterday. Considering my previous glasses were over five years old (I had newer ones than that, but they broke during the move to Pittsburgh and I've been using the previous pair for a year now), I have some serious adjusting to do here. It's kind of surprising I haven't fallen over yet. I can see the end of the street clearly, but now I have to focus in to see my laptop screen which is something I'm apparently out of practice with. (Ok, I'm used to eyestrain but this is _different_ eyestrain. Quite possibly the good kind, if there is such a thing, but still.)
Oh, and I woke up with a sore throat, which I still have post lunch.
So, can the x86 version of gdb debug PPC assembly, or am I going to have to compile it from source? Hmmm... Nope. "set architecture" only lists i386 variants, and I'm not adding anything as ridiculous as gdb to the base build just yet. Ok, time to brute force this problem. Step 1: back to staring at assembly dumps...
Ok, according to "powerpc-readelf -a ./a.out", the entry point address is 0x100000268. According to "powerpc-objdump -d ./a.out", this is the beginning of "_start", which does things and then jumps to __uClibc_main. That's in libc/misc/internals/__uClibc_main.c which (in the current svn version) makes the actual call to main() at line 400, and feeds the return value of that straight into a call to exit(). That's in libc/stdlib/exit.c but all that does is #include "_atexit.c" (I've noticed this is common in uClibc and have no idea why: what a mess). So exit() is defined in there on line 318... (aside: this is the March 27, 2007 nightly snapshot. I'd note which svn version it was but when I just ran svn info in the directory it said "This client is to old to work with working copy '.'; please get a newer Subversion client", to which I respond "nuts to your white mice" and move on with my life.) Ok, that does a few things before calling _exit() (which we know works), and I've configured out the locking without hiding the bug so it's probably one of the function calls. There's __exit_cleanup(), __uClib_fini(), and _stdio_term(). Stick lots of write(2,"exit1",6); calls around and check cross-compiler.sh to see how to rebuild this...
Ok, maybe I should figure out how to split cross-compiler.sh and mini-native.sh into a bunch of individual steps I can run one at a time to reproduce individual build stages, because when the reproduction sequence is:
make CROSS="${ARCH}-" KERNEL_HEADERS="${CROSS}/include" PREFIX="${CROSS}/" \
RUNTIME_PREFIX=/ DEVEL_PREFIX=/ all install_runtime install_dev
Substituting all those variables by hand is _way_ too much like work. Ok, I can hit cursor-up to rerun it after the first time, but when I suspend my laptop halfway through debugging and it decides to reboot instead of resuming (as has happened twice while I've been working on this problem)... Not fun.
Where was I? So: Fast wind index, V. Vogon constructor fleets, enter their code, see what it says. I'll keep watch... (Or perhaps it was the PPC segfault? Oh yeah.)
Ok, why did the uClibc .config file revert to saying the target it's building for is Alpha? Oh yeah, the build does a distclean before building readelf and ldd. Right.
So it gets up to "exit4" (right before __uClib_fini()) before printing out the __do_global_dtors_aux bit I stuck in the compiler earlier, and never makes it to writing "exit5" which is right after that. Ok, that's in libc/misc/internals/__uClibc_main.c (line 249), and the last thing _that's_doing before the segfault is calling __rtld_fini... Which is a function pointer. Right, where does that get assigned to? In __uClibc_main. Ok, back up to there...
I would REALLY like to be able to stick a printf in here. Ok, the arguments to __uClibc_main come from crt1.o, which the comment says is called from assembly. Wheee. (Wasn't I here a few paragraphs ago?) Ok, grep says that lives in libc/sysdeps/linux/powerpc/crt1.S and that's using... _fini. Right. What's _fini? Whatever it is, it's being loaded into r7, so back to the objdump: the low half of r7 is 5564 which in hex would be 0x15bc and that... Ah, ok "_fini". Haven't found the source yet but it's a small thing calling __do_global_dtors_aux by name, I wonder if anything in the code does that...
You know, a comment in the code (crt1.S line 77) says that it's using r3 instead of r7 since linux 2.6 clobbers r7, but the comment doesn't match the code. What the...? Ok, ask about that on #uclibc.
The Alamo Drafthouse is not only showing Killer Klowns from Outer Space as its' Weird Wednesday free movie, but it tracked down the original actor who played one of the alien clowns. Yes, it brought in a guest speaker for a free showing. How cool is that?
HA! Khem Raj on #uclibc knew what to do! Change the move into r3 from r8 to r7, and it stops segfaulting! (No idea if it's the right fix, but it's working for me now.) Ok, time to catch the bus and wait for tonight's snapshot before checking the powerpc config file in.
Even though my OLS thing this year is a tutorial rather than a presentation, I feel I should do a paper. Writing the paper is easy: in HTML. But they don't want that, they want papers in TeX or PostScript or some such. Maybe I can do it in docbook, which is essentially HTML with different tags and some bureaucratic boilerplate at the beginning.
I realize the docbook guys just cringed hugely there, but I don't care. Show me a wysiwyg word processor that produces docbook, and I'll show you a nonprogrammer who can use it. This is a programming language for people who want to write databases by hand in XML, and the narrow niche of programmers to whom that appeals are already using docbook. I don't see any growth in this, and I _do_ see it potentially being displaced by real tech writers finding a real document preparation format that can be manipulated visually. Go ahead and bemoan the fact that most HTML these days is written using visual tools like FrontPage, which admittedly produce horrible HTML. The point is that the visual tools are an option (so at least they're not doing it in Flash), and I have yet to see one for DocBook.
From my point of view, DocBook is an easy way to make both a PDF and HTML version from the same source. Nothing more. Yes, you can tag a stretch of text as being about root vegetables, sub-tag Rutagaga, but that's a lot of work and how do you debug the result beyond "well, the tags nest"? I have yet to see an actual use for any of these tags other than presentation (a big default style sheet), and any time I see infrastructure in search of a use I feel I should run away screaming before it collapses on top of me. If I want to know something I google for it, and nobody had to tag that information for google.
Now it's possible for docbook to replace man page format via doclifter, but they're replacing something ancient and horrible that only about a few old-timers still understand, and they're using it as just a source format to produce html, pdf, and text from.
Note to self: when running five parallel builds in the background, kmail's composer becomes unusable. Renicing all the builds down to -20 doesn't quite fix kmail, but makes it almost tolerable. After giving up on replying to email, watching youtube videos isn't necessarily the logical next choice, but it works surprisingly well. I'd call this a damning indictment of kmail, personally...
The announcement at today's meeting will be made public April 10 and they don't want me talking about it until then, but I really should have predicted it. (I wasn't surprised in the least, I'm just annoyed I didn't twig to it myself weeks ago.) Oh well, not my concern anymore, it doesn't take effect until after I leave. It does make me glad they didn't hire Mark, though. I'd still be working here if they had, and they're on course to have the last vestige of geekiness scrubbed out by summer...
Got a phone call from a Google recruiter this morning. (The last one was a woman, this one was male.) Told him about the position I'd accepted last week at the Linux Foundation. Presumably he'll call back in a year.
Trying to figure out if I should just drop 0.2.1 now and work on a 0.2.2 this week. The reason to do it is that the current build's working on an x86-64 host and has several fixes to arm. The reason not to do it is that neither Sparc nor PPC work right yet, plus I need to fix mdev.
I now have a timesys meeting scheduled for every remaining day this week.
Apparently, the CELF guys have me scheduled to give a uClinux presentation. I didn't submit a paper for CELF this year, although Thomas Gleixner poked me about it repeatedly. (This was partly procrastination, and partly it ends the day before Penguicon this year and I'm getting married at Penguicon, so there may be some prep work involved in that.) I think they've got me confused with the uClinux guys. I keep meaning to play with uClinux more, but so far I think I've booted it twice (both times under armulator).
Offline this weekend, went to Michigan for the next to last Penguicon concom meeting for the year. Didn't turn my computer on once, spent most of the rest of the time hanging out with Sal (the guy who runs Aegis sword camp). There was a cookout. We made extensive use of the "wrap stuff in aluminum foil and throw it in the fire" school of cooking (yes it works with bananas too), and then roasted marshmallow peeps.
The Penguicon web page continues to suck rocks. It has wiki-itis. (Ooh, these are LINKS! Let me put a little .gif of a globe next to them because having links in a web page is such an unusual thing nobody will ever notice if there merely blue and underlined and your mouse cursor changes when we go over them! This web thing is so strange and unusual, it's all new to us won't they be excited that these are LINKS! On a WEB PAGE! Why has nobody ever thought of this before?) Speaking of which, if you go to the "links" tab in the menu on the left, it goes "have you seen our news page?", which is A) the home page, B) in the menu on the left, C updated an average of _less_ than once a month so far. Yes, this frustrates me. Why does everybody insist that you need a database back end for static content? It's very very much gotten in the way the past few years. (Not just inability to put up content without an act of congress and the result looking horrible, but at the end of the year we want to archive last year's site and that means keeping the old version of the database running...)
I'm not a fan of plone. (Not that drupal or mason are improvements.)
Gene asked me to do an interview for his podcast yesterday. (That gets added to the index in April, I'm told.) If it seems like it's me blathering on for half an hour without a topic... that's pretty much what they asked me to do.
Finally tracked down the IBM Object Code Only Policy announcement (February 8, 1983, IBM announcement 283-016) and added it to my mirror. I'm tracking the rise of the proprietary software industry. The first rumblings occurred in the late 70's but it didn't become big business until 1983. This announcement was IBM joining the proprietary software industry, and announcing that distributing source code for its software would no longer be standard policy.
Fade and I head out with Robin and Christian to go see FreezePop tonight.
Did QEMU just merge a patch to add support for the "PowerPC SPE extension", I.E. the Cell processor? Boggle. (Only IBM could work "Syngergy" into a processor architecture TLA.) Ah, I see, not a "Synergistic Processing Element" but "Signal Processing Engine". Apparently a Freescale TLA, not IBM.
I emailed Ed my resignation yesterday, and he forwarded it to the company today. Still here until the end of the month, though. And I've been working on Firmware Linux for years, don't expect me to stop now. Less focus on bootstrapping Gentoo though, because that was Mark's area of expertise and since Ed declined to hire him that part's not my problem anymore. Still plenty of work to do on the generic emulated native build environment part. Getting networking to work, for a start...
Testing the build under x86-64 (Mark's machine is back up, and I'm borrowing it via ssh while he's in class). And where did I leave off on the PowerPC debugging again... (My darn laptop rebooted instead of resuming from suspend this morning. Again. Luckily, I have notes. What did you _think_ this tech blog was for?)
Darn it, according to my email I'm missing the Monty Python and The Holy Grail Killer [Rabbit] Feast. The Alamo Drafthouse has done one of their theme feasts for Montie Python ic den Holie Gralien and I'm MISSING it. Sigh. I've gotta get back to Austin...
Updated firmware build to not attempt to build qemu, since the "work with gcc 4" patch doesn't apply to 0.9.0, or on a non-x86 host. Someday, Paul Brook's QOPS patch will be finished and applied, and then we can build qemu with an arbitrary C compiler. Until then, as much as I hate host dependencies I can't sanely get away from this one.
Found yet _another_ thing wrong with dash. If you fork off background tasks with &, they still get killed when you hit ctrl-c in the shell. (Because they're a child process, so the signal still gets sent to them, even though they're backgrounded.) THIS IS NOT A REAL SHELL!!!
So tracking down the ppc thing (still partway through) has driven home the fact that building and reinstalling these things is harder than it should be, especially from a debugging perspective where you want to patch/rebuild/install/test repeatedly. For example, gcc needs the headers moved from their default location and the wrapper reinstalled each time.
One way to deal with this is to build out of tree, and have the appropriate build snippet written into each build-package working directory. Then after you patch you just rerun ./rebuild.sh and it updates your cross-compiler-x working directory.
The cross compiler needs binutils, gcc, uClibc, and the Linux kernel. For gcc and binutils it's already doing that, and I've made it work before for the Linux kernel, but uClibc doesn't document how to do it and the normal ways (run configure from the working directory, feed O= argument to the make command line) don't work. It's got top_srcdir and top_builddir in the makefile, but when I set top_builddir it's looking for Rules.mak in that directory (which is clearly a source file).
For the second stage build, I'd need the above four packages plus busybox, make, and bash. Shouldn't be too hard.
Of course the new issues this raises are: when should the old source directories get deleted? I can't compare the date stamps on the file because those come from the tarball. Not unless I run touch on the directory to give it the same date as the source tarball. Which could work, I suppose.
Should the extracted tarballs go under source or under build? Really they're a cache, so I guess build is the right place for 'em. build/sources perhaps?
And if build gets cluttered enough, should the various "output" files (all the *.tar.bz2 and *.ext2 files) go into an output directory, or even at the root level? Putting 'em at the top level would imply moving the scripts to a subdirectory (mixing 'em in the same directory makes manual cleanup irritating), is that justified?
Yeah, going to need a README soon, I think.
Phone call's not until this evening, so I'm banging on powerpc. All the source tarballs and build directories can be left behind after a build by just setting CLEANUP=echo in the build script. (This is intentional, and I should document it better.)
So the segfault happens after returning from main(). It happens if I disable threading support and atexit() support. In fact it happens on a powerpc version of allnoconfig, with every option disabled. It also happens if I call exit(0);, but not _exit(0);.
Asking around on #uclibc on freenode, they say that the code that takes control after main() is crtend.o, and grepping around in the gcc source a bit that turns out to be produced from gcc/crtstuff.c. I can't stick printfs into this code (since libc is in a sort of wonky state during exit and using stdio FILE * stuff is highly inadvisable) but I can stick in write(2,"hi\n",3); and that's close enough... (Ah, in order to rebuild and slap the new one into the cross compiler I have to shuffle the directories around and reinstall the wrapper. Right. Maybe I should have the build write out the shell script snippet it uses to build and install each chunk into the source directory? If it takes _me_ more than one attempt...) Ok, __do_global_dtors_aux() in crtend.o is being called before the segfault. Yea checkpoint. Time for the phone call...
So Mark forwarded me this email he got Sunday:
Hi Mark-
Thanks again for making the trip to Pittsburgh and meeting some of the team.
While we believe you have many strengths, at this time your overall set of skills and experiences is not a match for what we are seeking in filling our Engineering positions.
I wish you the best of luck in your future endeavors.
Thanks, Ed
Ever since David Mandala left I've had a can of Slim-Fast on my desk, because on his last day I decided I'd stay as long as my Slim-Fast supply lasted. That was the first little temporary deadline I gave myself, such as staying until Christmas vacation or staying until Thomas Gleixner's visit. The most recent one was that Mark was interviewing, and I was looking forward to working with him. Then I'd be able to go back to Austin in August and telecommute from there but still have somebody local to discuss the project I was working on with.
I drank that can of slim-fast today, in front of Beth who knew what it meant.
The above email made my decision about what to do with the unexpected opportunity a couple friday's back a whole lot easier. The opportunity is that the Linux Foundation offered me a fellowship to do documentation for a year (at 1/3 more money than I'm making now although that's not particularly important). I applied for it back in November (during the great engineering exodus), and had a phone interview and a performance test since, but the process was sidelined for months by OSDL merging to form the Linux Foundation, which is why it was somewhat unexpected to hear back from them. I hadn't immediately replied because I'd been worried about potentially leaving both Mark and Timesys in the lurch. With TimeSys about to hire Mark, I couldn't just leave. But Ed was kind enough to make that stop being a consideration.
I told Joe repeatedly that he should hire back David; he hired Ed instead. That was his right as CEO, of course. I told Ed he should hire Mark, and Ed went so far as to interview him, then said no. That is of course his right as VP of engineering. My advice in the matter could only be a suggestion, a single data point from one engineer, and what's that worth? (I also told both of them they should really read "The Mythical Man Month", to get the authoritative source on that whole "programmer time is not fungible" bit. I don't believe they've done that, either.)
Now Mark and David are starting their own company. Even though I can't do documentation work as part of that company (and yes I asked, they wouldn't go for it) it still made me look around and go "why am I still working here?" Timesys had a great group of people when I signed on, but Garrett, Piggy, Christian, Greg, Walt, Sam, Bill... All gone. Even Tony, Pragnesh, and Chris, who I didn't particularly get along with but still respected. I don't really work directly with anybody who's still left. Yeah Ed said he'd let me telecommute from Austin, but so will the Linux Foundation.
I'm still at TimeSys until the end of the month, then I go to work for the Linux Foundation. I've missed writing, and now I get to spend a year doing it. I'm looking forward to this... :)
Ok, build's working again. Need to test it on x86-64 but the new build system Mark set up got reformatted with Gentoo when he got fed up with Feisty Fawn and my account hasn't been restored yet, and I powered mine off after the second kernel panic. (Possibly I should compile my own kernel and see if that's happier.)
Back from the doctor's office. (Dialysis drops! Now with _quadruple_ vision, just try driving home. Bonus points: it's raining, and I got lost. Clever little sunglasses inserts, though.)
So my retinas are apparently fine, although my glasses are 5 years out of date. (I knew this intellectually but going through an eye exam does sort of confront one with the enormity of it.) So the problem could be eyestrain, with possible complications from my darn sinus infection. I should have that looked at, but first I should get new glasses.Should I be worried that not only am I editing patches by hand in vi, and that the results apply, but that I take this in stride? Possibly I've been doing this for too long. The patch in question came across the list and might fix the PPC segfault, might not. Trying it now... Nope. Darn, still need to debug it.
"Sweet Home Alabama" has _mellow_ electric guitars. I think that's relatively hard to do. (Still loving the ipod shuffle. I should invest in some high end noise cancelling headphones.)
Ed declined to hire Mark. This makes my decision about what to do with the unexpected opportunity a couple fridays back a whole lot easier.
After leaving memtest86 running overnight on the x86-64 machine, it found one bit off, at location 1A707450, which is the 423.4 megabyte mark. I've booted the kernel with mem=400M to see if I can get stable builds out of it, and I'm pondering applying the bad ram patch to mask out that page. It only found one bit off, and it had to disable ECC to do it, so it would be a shame to have to order more memory. But first, I'm rerunning my script to build everything in parallel, from yesterday, to make sure that the system stays up with mem=400M. I want to confirm it's just memory and not the processor overheating or something...
Sigh. Three of the five builds have already died, with various errors. This is weird. Lemme check and make sure everything still works on x86... Ok, building on x86 is dying with the same errors. Right. The single bad bit (that's probably masked by ECC anyway) seems like a red herring. (Although I'm leaving the server dialed down to 400 megs for now, but the only thing I could potentially attribute to that is the hang/panic, not the build breaks. And with the electicity around here the hang could be anything.)
Darn it, the hang on the 64-bit machcine happened again even with the reduced memory. Most of the panic message had scrolled off the console but I saw "Soft lockup detected on CPU #0" with a stack trace on updating the timer interrupt. Need to figure out how to hook up a serial console.
I've explained how I hate Pittsburgh, and every time I bring it up my friends try to "console" my by either finding fault with Austin (where I lived for 10 years, so I suspect I'd have noticed by now) or by telling me something supposedly good about Pittsburgh. The "great hospital system" comes up in the second category. By "great" they mean "large", of course. Not good. Because it's closed on Sundays.
I mention this because I've been having eye trouble, and I googled the symptoms and it came up with "possible retinal tear/detachment, treat as heart attack, seek medical attention immediately". Probably someting I should look into, so I called the number on my insurance card for University General Medical, navigated the "press 1 for english" directory tree to the point where I got a human (their answering service), which said that kind of thing is serious and they'd probably get me an appointment quickly, so I should call at 8 am monday morning because they're closed until then. Apparently "treat as heart attack" means "call back monday morning". Whether or not two asprin are involved is left as ane xercise for the reader. (Sigh. I don't even know what I have, I'd just like somebody to look at it.)
Ok, backing up to the last checkin before the uClibc upgrade, all the architectures built fine. Now to try just updating the uClibc snapshot without the miniconfig changes.... That works too...
Doh! Ok, when changing the name from config-uClibc to miniconfig-uClibc, change all the config files to GENERATE the new name, not the old one. Yes, I have a patch to add error checking to kbuild's miniconfig read support, no I'm not using it and it's not pushed upstream yet.
When one's laptop's memory is full and the disk is overloaded with I/O, 'watch -n 10 sync' is a bit of a dirty trick, isn't it? But it makes the web browser happier. I'd poke at the "swappiness" value under /proc, but that makes the OOM killer restless. (I point out that vim is kind of stupid, it does an fsync on its' .swp file every few dozen characters, which causes the cursor to freeze for upwards of 15 seconds when the disk is saturated.)
Also, Ubuntu's getting pedantic. If you type "vi" you don't get vim but a broken version that's dumber than the busybox vi (so broken you can't use the cursor keys while in insert mode). And it's pointed /bin/sh at dash which is unusable in all sorts of little ways (it doesn't understand blah/{file1,file2} syntax, it supports "." but not "source"...) and if there's a way to tell make to use something other than /bin/sh I don't know what it is. I know Ubuntu is trying to do this to make users change their behavior, but what it's made me do is stop recommending versions of Ubuntu beyond 6.06 because you have to fix so much intentionally broken stuff by hand now to get a usable development system. (Don't wave "SUSv3 compliant" at me for something that's been a de facto standard for the entire 15 year history of Linux. It worked, you broke it, that simple.)
Sigh. I left the build running on the x86-64 machine last night, and although armv4l seems to have built fine (I can boot to a shell prompt in qemu and everything), the x86-64->i686 cross compiler segfaulted trying to build an i686 native gcc. (Yes, gcc died trying to build... gcc.) So, building x86-32 target on an x86-64 host, not so tested. I guess most people are still doing the other way 'round.
Migrated uClibc to miniconfig, and checked it in. (To FWL, not to uClibc.) Yay! Gotta push some of this upstream.
Fun little trick I've probably mentioned before, to run a process detached:
(./build.sh armv4l mipsel x86_64 sparc i686 > out.txt 2>&1 </dev/null &)&
The two-fork and exit thing is daemonizing, so if the parent process goes away it's unaffected. This time I'm building i686 at the end of the list because I already know that one breaks on x86-64 (although I'm trying again to reproduce it). And armv4l is at the beginning because that built ok last time but now it's with the new uClibc and miniconfig stuff. The stuff in between, I have no idea.
Finally got my playlist on the ipod shuffle Fade got my for Valentine's day. The sucker hotplugged right into my old Ubuntu 6.06 laptop but the files on it were all ipoddy (m4a?) so I installed gtkpod, which was a bit crotchety but in the end did work. (I had to create a symlink, "ln -s sda /media/ipod" which most end users wouldn't have figured out, but the error message was informative enough I only needed to make a level 1 geek roll and didn't have to resort to strace.) It's a MUCH more useful ipod with my playlist on it. Yay ipod.
[After returning from a party] Hmmm... The mips build died too, with some kind of assembly error. Something is very weird here.
A variation on the above trick is:
for i in armv4l mipsel x86_64 sparc i686 do (./build.sh $i >out-$i.txt 2>&1 </dev/null &)& done
And when I did that it fired off several processes, and all the .txt files were getting larger for a bit, and then ssh hung for no apparent reason, and I couldn't ssh back in either. That smells an awful lot like a kernel panic, which would smell an awful lot like bad ram, which could easily explain the build breaks as well. Hmmm... Fade just went to bed so borrowing her monitor and keyboard again... Black screen, unresponsive. (Sigh. Panic should unblank the screen. Not a new observation.) Ok, Ubuntu installs memtest86 so I'll leave that running for a bit and see if it can find anything...
(12 minutes later, no memory errors found yet so it isn't something OBVIOUS...) It could also be overheating (insufficient CPU fan?), or have an understrength power supply. It produced incorrect results when I told it to do something processor intensive, and when I did something in the neighborhood of make -j5 the whole system went bye-bye in a minute or two. But just sitting there waiting for somebody to ssh in, it was up for a couple days...
One really nice thing about my laptop is I can beat the heck out of it CPU-wise and it doesn't get unhappy. (As long as it's plugged into the wall, anyway.) It monitors its' own heat quite well and changes the fan speed or even clocks back the CPU if necessary (as long as I don't close the lid while it's number-crunching, that will overheat it). It's got a built-in UPS by way of the battery so it can use fairly dirty wall current without being upset by this.
My laptop's getting overloaded. I'm rebuilding all the FWL targets (of which there are now something like 8 although two don't work yet (ppc and sparc) and variants like i586 vs i686 aren't all that exciting). Although my laptop has half a gigabyte of ram, between the compile, kmail showing threaded views of mailing lists with tens of thousands of messages, the 40 or so open tabs in konqueror, and the fact I've done suspend to disk over a dozen times since the last reboot and fragmented the heck out of my swap file... Slow, unhappy laptop.
Still debuging the uClibc makefiles to get it to accept the new kconfig. I hate make. Finally got it to build, but vapier doesn't want to merge it (he wants to use the upstream version verbatim). So to get it into uClibc I have to break it into patches and feed it upstream into the kernel, which is yet _another_ tangent. (But eventually, Colonel Potter will get his tomato juice! M*A*S*H reference, don't worry about it.).
Finally got the new kconfig working for uClibc, fed it into the build to get my x86-64 server to build uClibc without curses installed, and now it's building. And BOY is it slow. My 32-bit Pentium M laptop kicks my 64-bit Sempron server's ASS at compiling large software packages. Part of this is that the laptop's 2 ghz and the server's only 1.6 ghz, but I think most of it is that my laptop has 2 megabytes of L2 cache and the server only has 256k. When it comes to compiling, it doesn't matter what your processor's doing if it's sitting waiting on the memory bus all the time.
Still, OUCH. The gcc build got into tree-ssa-*.c and I actually killed the build the first time because I thought it was hung. Nope, just sitting there with gcc eating 99.?% CPU and very occasionally moving to the next tree-ssa file...
Memo to self: when I buy a new 64-bit laptop, make sure it has more than 256k of cache...
Need to wait for tonight's uClibc snapshot before checking in the new kconfig patch because svn 18129 makes that patch _much_ less painful. Then I can convert all the uClibc configs to miniconfig, and then worry about A) updating my kconfig to 2.6.20, B) breaking up my kconfig changes to feed back to the kernel guys, C) breaking out generic config stuff into a separate file.
Mark was at work today interviewing for a job (he'd be telecommuting from Austin). Fade came in on the bus with me this morning to say hi, and is sitting at Garrett's old desk working on her laptop. (The bus was less crowded than I've seen it in weeks, this is the first time this week I've even been able to stand behind the white line when I first got on. Could be the rain, could be that we were catching it an hour earlier than I usually do, or maybe we were just lucky...)
David Daney on the uClibc mailing list responded my my ranting about gcc (I've gotten just a bit touchy on that subject, haven't I?) by pointing out that my arm bug may indeed have gotten fixed for 4.1.2, and maybe I can put squashfs back in now. That would be HIGHLY cool, and I'm testing it right now...
Ok, first pass of that worked but then I realized I hadn't added CONFIG_SQUASHFS back to the various kernel .configs. I should probably have some kind of global section appended to the end of each one, and have the config file just have platform specific stuff. (But I need to get uClibc switched over to miniconfig first.) Actually COMPILING the squashfs patch (and not just applying it and leaving it configured out) breaks in SLAB_KERNEL, but this is because the cross-lfs.org squashfs still contains that but it was removed from 2.6.20 (I grepped the whole 2.6.20 tree). Googling for SLAB_KERNEL 2.6.20 brings up some patches replaing instances of that with GFP_KERNEL, so let's try changing it to that... And happiness. Pinged the CLFS guys to make sure they know.
Applying a different patch to a different kernel doesn't prove it was fixed in the toolchain, but I'm not going to quibble. It works, life is good, moving on...
HA! And this is something I can point my boss to and go "here's why Firmware Linux is open source": James Newell just sent a patch in to the Firmware Linux mailing list fixing soft float for arm! Cool! Testing now, and then I can take it off my todo list.
Eric has finally decided that the correct way to describe what's going on in the software market is to use the Black-Scholes theory of option pricing and frame the discussion in terms of the benefits of modularization. Ok.
Mark is here. Yay mark. Off to dinner.
Spent far too much of the day on an email thread arguing about the definition of "commoditization" with Eric Raymond (who found a wall street analyst presentation and is now convinced that what open source is doing to the software market can't be called "commoditization" because Red Hat's margins remain too high. The software's a free download and the margins are too high. Right.
Still working on getting toybox's cleaned up kconfig adapted to uClibc. Keep getting interrupted. I'm also re-converting the mercurial uclibc mirror with a better comment format, and I've got that auto-updating now (added tailor to my shell script that rsyncs my laptop to my web page).
Another alumni dinner this evening, like the one for Joe last month, only this time for Ed. Half the attendance of last time, and focusing more on history than technical decisions (since the new build system has been approved already). Still, Ed seemed to get something out of it...
So the reason the server wasn't responding is that the bios doesn't switch it back on after a power outage, and one of the cats jostled the plug. (Yet another fun thing about Pittsburgh is the prevalence of two-prong outlets predating the Tennesee Valley Authority, which you need to screw an adapter into to get a three prong outlet, and since the position of the screw in the middle of the base plate isn't quite standardized they aren't always the world's most stable connections.)
I'm finally encountering some of the broken things about Ubuntu Edgy that people have been telling me about (my laptop's still on Dapper), namely /bin/dash. Not to defend bash, but dash is a fairly horrible shell that doesn't work for lots of stuff. For the moment I've just redirected the symlink to bash.
They're also using a broken (or as they prefer to say, "newer") version of bash that's screwed up tab completion. To grab the most recent snapshot of FWL I do "wget 'http://www.landley.net/hg/firmware?ca=tip;type=bz2'" which produces a filename with both a question mark and a semicolon in it, and then I go "tar xvjf firmware[tab][tab][tab][tab]" but it's just not going to tab complete that one because the filename doesn't end in any recognized tar extension. I DO NOT CARE. I didn't ASK it to. It's trying to be way too clever, and I want it to STOP DOING THAT.
I need to get back to writing my own shell...
Fun number 3:
$ gcc hello.c collect2: cannot find 'ld' $ ld ld: no input files
Yes, this is why I wrote a wrapper script to lobotomize gcc's path logic (it's too broken for words) but this is NOT THE GCC I BUILT. This is the one that comes with ubuntu, and it's not working out of the box.
Not a fun morning. (I'm glossing over the darn IP masquerading session timeouts in the new linksys freezing my ssh session. I set my laptop's keepalive at 2 minutes and maybe that'll work. Found one of the keepalive settings in linux is undocumented, made a guess at what it does, might send in a patch for that if I go through the source and confirm my guess.)
I'm guessing it's confused becaue I installed gcc 3.4 in a futile attempt to build qemu, so I tried to remove gcc 4.1. With dpkg. If there's a "rip this out and everything that depends on it I don't are how stupid", I haven't found it yet. Yeah, it's dangerous, but I can reinstall the darn thing and at this point I'd probably do so with something OTHER than ubuntu edgy.
HA!
cd /usr/lib/gcc/x86_64-linux-gnu/3.4.6
mv collect2 broken
ln -s /usr/lib/ld collect2
Now the bastard's working. For values of working that involve "it builds hello world". And possibly qemu... Yup.
Next fun bit: uClibc can't do "make oldconfig" unless you have curses installed. Sigh.
Walked Khem through building FWL. A strange build break turned out to be an idiosyncrasy of his environment (in his bash profile) breaking the User Mode Linux makefile of all things. Just because it's not my bug doesn't mean I don't have to fix it. :)
I didn't MEAN to spend today upgrading uClibc's version of kconfig, but that looks like how it's going to turn out. I should probably resync the toybox one to 2.6.20 before propogating it, though. Maybe push some cleanups back upstream if they haven't already manifested there.
I still hate makefiles. Mixing imperative and declarative syntax is horrible. "You must make these things happen in the right order, but you have no direct control over what order things happen in." So people have makefiles call other makefiles, which is an evil hack to work around the fundamental brokenness of the language.
So I finally got the server hooked up yesterday, SSH was working, life was good... And the IP of the cable modem seems to have moved overnight. I sort of expected this, and can work around it with dyndns, but honestly.
Ok, the rumor of IPV4 address space exhaustion just isn't true (even ignoring the way what has already been handed out is so horribly allocated, or that youtube's pretty conclusively proven that multicast will never amount to anything). They're already _giving_ me a world routeable address, and when they rotated it they gave me another. (If they didn't, bittorrent wouldn't work and that's how World of Warcraft downloads updates.)
I'm sure if I asked them they'd say it's a vital anti-spam measure, holding up the general crappiness of Windows machines as a defense. Because none of the viruses setting up phishing sites could possibly master dyndns themselves? Because it's not possible to spew spam through a masqueraded connection? No, they're just being annoying.
They want me to pay extra to get a static IP, even though it doesn't cost them extra and I'd HAVE one already if they didn't go out of their way to screw it up. And although it's nominally $10/month in reality I can't get it except as part of the $95/month "small business" thing.
We need more competition in the broadband space. DSL, Cable, and fiber to the home all have the problem they have to lay cable, which is a big enough moat around the business (forget the digging, securing right-of-way is insane) to keep the small and hungry out so far. The cell phone system showed some promise but it's owned by the phone companies who don't want to compete with their own DSL. (Last I checked they still wanted to charge $250 for a cellular modem with horrible throughput, and only give unlimited minutes for half the day. The internet may be mature, but the last mile isn't.)
I suppose I could just take the box into work and ask Beth to hook it up in the server room. I _am_ trying to get a 64 bit server to test builds on, which work hasn't managed to do yet. (We have several x86-64 machines, all of which have 32 bit kernels installed on them. Beth has tried repeatedly to fix this, but isn't allowed.)
Huh. Fade tells me that the new router is dropping connections for her, and when she read out the connection info it's still using the old IP. (Waste of a good rant, that.) So what the heck is broken?
The problem she's having sounds like the IP masquerading timeout is set to 10 minutes or something ridiculously low. Spent an hour with the linksys website and with google trying to figure out how to increase it to 6 hours. (Do they call it "session timeout", perhaps? Nope.) I found a web page saying how to edit the registry keys in her windows box (connections are dropping for her on windows, not her mac laptop) to send keepalives, but she's not comfortable editing the registry (and who is, config changes being persistent by default was too stupid for words...) and I'm at the Te Cafe rather than home at the moment...
Still doesn't explain why I can't ssh into the server I set up last night.
I run sync (by hand) before each suspend, but after suspend I tend to want to pull back in more of the working set than resume is willing to give me. I've decided if I ever do figure out how to implement such a prefaulting script, it has to be called "thwim". You have been warned.
Spent most of today reinstalling my x86-64 server, which involves borrowing the monitor from Fade's PC. (I could buy a spare CRT from work for $5, but I don't want to have to throw it out when I'm done with it. Seems ecologically impolite.) Of course this involved hooking a new system up to the 3 year old linksys which has been going strange for months (even before the cat threw up on it) and finally gave up the ghost, so Fade bought a new one from Radio Shack. (This is the first thing we've gotten from Radio Shack in over a year that actually worked. They've managed to sell me _cables_ that didn't work. Luckily, they have a liberal return policy, and the mom 'n pop computer store down the street was out of routers. I toyed with the idea of setting the 64-bit machine up _as_ a router, but it hasn't got wireless.)
Still can't find sshd in the kubuntu repository, but dropbear is easy to install.
I'm also still boggling about the phone call from the end of friday. Talked to Mark, Dave, and Fade about it. "Huh." And you can quote me on that. It's good news, it's just the timing is really strange. I think we're going for it, though. Now I need to talk to Ed.
Dear Coca Cola corporation: you're screwing up my attempt to recreate my wall of cans.
I mentioned my attempt to rebuild my stash of coke zero and pepsi one cans after somebody at work threw them out. But apparently, coke's now driven pepsi one out of the market (or at least our distributor can't get it anymore) and to celebrate they've turned their once-white coke zero cans dark metallic grey. I have 32 black pepsi one cans, but only 2 of the old white coke zero cans, and the new shipment is dark grey.
Oh well, at least I got two pictures made while it was still possible to do so...
The copyright navigator is cool, at least if you follow intellectual propertly law as a hobby, like me.
I have 120,000 messages in my linux-kernel folder and I just told it "thread messages", and now kmail is thinking. I remember back when kmail used to lose its marbles when the message count in a folder went over 32767 and its' indexes started having negative numbers in them. Uphill. Both ways. (Ah, 2004. So long ago...)
Finally brought the ipod "teensy" Fade got me for Valentine's day into work. (I mentioned it in the morning rant about how much I hate my commute, which I am now in the habit of deleting after writing. You can just assume I start every morning I'm actually in the office in a horrible mood, and stay late because I really don't look forward to going Out There again, which is kind of inevitable on the way home until they get a teleportation booth installed here.) Anyway, the ipod mostly has beatles songs on it (since Fade's long since lost track of which songs of mine
Dear Amarok developers: When I minimize amarok, I want it to do what it does now when you hit the big X "close" icon, namely go to the little mini-icon in the panel right edge of the task bar, and drop out of the window list in the main task bar. Don't show up in BOTH the task bar and the panel. And when I hit "X", close. Don't pop up a window telling me that closing is not what you're about to do, and then minimize instead. Quit it. If KDE wants to add an entry to its global configuration saying all apps should minimize when I hit X, fine. If all apps that register a panel icon want to consistently minimize instead of close, fine. But that's not your job to make that kind of decision unilaterally, and a pop-up saying "I'm not about to do what you think, and there's no button on this pop-up telling you to do it anyway, you have to UNDO what I'm about to do by hand and then go do it the way I tell you to, nyeah" is NOT an improvement. Thanks.
I really should update the kubuntu on my laptop to something more recent, but I should really get a new laptop too...
Heh. Beth the sysadmin says our Microsoft representative called and asked why our servers weren't running Server 2007 yet. "Because we'd like them to keep running?" He offered a "complimentary consultation". Didn't say how much of it we'd have to sit through before we got the free luggage set.
Yes, the exchange server is still sitting there (festering) but there are some very nice alternatives out there now, which Beth is looking into. The server room may yet wind up windows-free later this year.
It's sad, actually. Stu tells me that Microsoft has announced the End of Support date for Vista Ultimate is 2012, but XP will be supported until 2014. Something tells me it's not selling really well. And Laurie of Marketing just sent a company-wide email announcing that the Exchange server will be an hour off from the start of the new daylight savings time until when daylight savings time would have started before, that there's nothing the Windows consultant can do about it, and that Microsoft's officially recommended fix is moving your appointments an hour. (I almost replied "Logging in as the administrator and resetting the clock by hand (or changing the time zone to Chicago), and then changing it back again by hand 3 weeks later, isn't an option then?" But I decided against it.)
Unexpected news this evening. I'd write about it, but I have to figure out what to DO about it first.
The reason I'm awake at 5 am belongs in my livejournal rather than my development log, although it's likely to prevent me from getting too much done today.
I set tailor copying the uClibc svn repository to a local Mercurial version at around 8 am. It's now 9:15 and not quite at svn 5000 yet. The repository is up to about svn 18000 now. The slowness isn't on the mercurial side.
The arm native gcc is still dying immediately with a bad mmap, but coincidentally Khem Raj posted a patch to the list yesterday to fix an mmap issue, and what do you know: this was the fix I needed. Confirmed that gcc can natively build "hello world" on all the other current platforms except sparc (where I still can't get a shell prompt). Updating to tonight's snapshot should fix arm. Now to bang on ppc some more.
Tailor took SIX HOURS to create this. Now I just need to figure out how to keep it up to date...
Ooh, after this morning's drama waking me up at 4:30 (and me being awake ever since), my cat proximity has spiked. Time to take a quick nap before the 4 pm conference call.
Today I'm fixing uClibc. Since uClibc hasn't had a new development release in a year and a half, I'm using an svn snapshot. (That and toybox are the only things I'm doing this for, and toybox is very optional at the moment and _will_ have a release when I start to actually need it.) I was staying with a known-working snapshot of uClibc from February 14 up through yesterday's release of Firmware LInux. Then I tried to add PowerPC support, and it segfaulted (after printing "hello world", in the exit-from-main cleanup code.) So I updated to last night's snapshot tarball in hopes this had already been fixed (and so if I wind up producing a patch, it applies to the current version). Guess what? The current snapshot doesn't compile, and apparently hasn't done so for a while. The last one that built for me was svn 17940.
I need to make a mercurial version of the uClibc repository.
A few reasons I prefer Mercurial over Subversion:
Even when the project's maintained in SVN and all my mercurial tree does is mirror it, it's still WAAAAAY easier for me to have a local copy so "hg update -r 12345" takes 2 seconds (instead of the full minute it sometimes takes svn when the server's bogged or DNS is being stroppy) when I'm binary searching for the checking that broke the build. And if the SVN server goes DOWN for an hour (it happens) I can still make progress on mercurial, but can't even view the history with SVN.
So the nice trick an IRC user pointed me at yesterday, aimed at reducing gcc memory consumption, breaks the arm build. Sigh. Temporarily reverted just long enough to get a darn release out... Ok, GCC is still dying with "I/O possible", which is just weird. That's a non-fatal (and presumably largely ignored) signal. I'm confused.
Ok, piping the output of gcc to "tee" apparently causes gcc to _break_ now. Brilliant. Just brilliant. Can't _anybody_ sanely handle short writes to stdout? (It seems that gcc registers a fatal signal handler for SIGIO. How do you screw up that badly?)
Firmware Linux 0.2.0 is out. Pausing for a bit to poke at Power PC support.
The last words in the blog of Ray Ozzie, the new head of Microsoft after Gates and his assistant Ballmer retire in 2008, were "Let's keep the momentum going..." This was written 11 months ago. You can't make this stuff up.
Got up to a late start today. Late enough there was actually somewhere to sit on the bus and use my laptop. Weird. It's snowing again.
David Lang linked me to two places to get Linux laptops. I think my brother's still offering a better deal, but part of that is I find both sites slightly disingenuous. (For the cheaper model of both they advertise the processor as "Core Duo", but when you click through you see it's only an upgrade option and the price they mention in the default system is Pentium M. The difference is do you want a 64-bit processor or a 32-bit? Yes, there are weasel words in the description. I don't care.) Still, good to know.
Release today. I've got i686, armv4l-vfp, mipsel, and x86_64 booting up to a shell prompt under qemu. Ideally I'd like to get sparc doing the same (mostly I just haven't tried it) and get mdev into toybox before doing the release, but that's optional. Won't hold things up for it.
I talk to Ed today about moving back to Austin. There are three factors at work here:
Eric Raymond's been trying to get OSDL (or whatever they're called this week) to sponsor a documentation project (different from the one OSDL already interviewed me for) which he'd like me to make time to work on if funding goes through, and I haven't even talked to the CELF guys yet. I've also been reading things like Paul Graham's essays on entrepreneurism (linky linky linky), which haven't really told me anything I don't already know (although "don't learn things from teachers who are bad at them" is a great statement of that rule), but they are reminding me that if I'm ever going to start my own company now is probably the best opportunity I'm going to get.
Cool! Ed is ok with me telecommuting for TimeSys from Austin, and he's interested in interviewing Mark for a similar telecommuting position. This is highly cool.
Doing the ritual "build a clean checkout before release" thing. Forgot to check in the kernel patch to fix UML packaging. Oops. ("hg diff" is a great way to show you what files you've touched and not checked in yet, but it doesn't help if you never did an "hg add" on them in the first place.)
While talking to my brother this weekend we both noted the rise of $600 laptops. "They're turning into stereo systems," I said. "They're turning into VCRs," was his reply.
And I need to make sure my sister knows about the wedding.
So I'm still looking for a new 64-bit laptop, but every one I've seen in local stores comes preinstalled with Vista. I actively don't want Vista. I object to paying for Windows in general, but if the price of the laptop is low enough to make it the best available option, I can ignore XP. But there's no way Microsoft is trumpeting my Linux purchase as an indication of the success of Windows Millenium Second Edition. That's "insult to injury".
Eric, Cathy, and Fade are swordfighting in the backyard. (There are three "C"'s in that sentence and I had to hit each one at least three times.) I'm sitting on a chair with my laptop. I should go to sword camp next year.
My father and brother (and their respective female acomplices) showed up and took Me/Fade/Eric/Cathy out to lunch. (First my brother demonstrated to Eric that he's been swordfighting for 14 years for the SCA, Markland, fencing, etc.) My brother is a salesbeing by temprament, and currently he's doing so for CDW. He related to me how there's been a surge in Macintosh sales since the release of Vista. Good to know. He also says he can get me a 64-bit laptop without Windows on it, which is also good to know. I expect to take him up on this.
Got the mips build building. The kernel config was indeed big endian (note: if you build a mips kernel with a big endian kernel config, using a little endian compiler, it'll boot up about halfway under the big endian mips qemu before panicing. The little endian mips qemu will just exit immediately saying it can't load. This can cause some head scratching. Now it's going all the way to the command prompt. Three down, two to go.
Still need to figure out what to do about the soft float patch. Apparently, arm soft float only works if you --enable-shared in gcc 4.1.2. The patch adds it to libgcc.a, which lets --disable-shared find it. If you apply the patch _and_ --enable-shared, the functions are in there twice and it goes "boing" halfway through the build. I've put that back on the todo heap for now.
The only thing wrong with x86_64 (preventing it from getting to the initial shell prompt, anyway) was that it needed a symlink added so it could find ld-uClibc.so.0, rather than ld64-uClibc.so.0 which the uClibc build is inconsistently making. Took a little while to track down where in the uClibc source code it was going strange (Rules.mak), but I found the appropriate hammer to hit it with (overriding a makefile variable from the command line) without even having to patch the source code.
So the patch to make armv4l soft float work for the --disable-shared cross compiler breaks the --enable-shared native compiler. If I remove the patch between the cross-compiler and mini-compiler build phases, the result seems to be a working soft float compiler (although how to actually _test_ this is an open question, since there's no way to tell qemu to disable the floating point coprocessor, or i686 instructions when I build a 586 toolchain, or nommu...)
I'd really rather not add infrastructure to conditionally apply a patch. I'd much rather figure out how to make the patch have its own #ifdefs. But it's an area of code I'm really not familiar with in a project that has a configuration system approximately as compliated as the US tax code. Hmmm...
I really really hate dealing with the gcc sources. I have a patch that applies to the file "gcc/config/arm/t-linux". Where is that called from? There's no way to know, because tracing _back_ where any of this stuff is used from is virtually impossible and goes through a dozen layers of brick-wall indirection. Also, the existence of ./configure --with-float=soft, -msoft-float, and -mfloat-abi=soft is starting to remind me of perl and Rube Goldberg.
Eric's working on getting up a tree of Ubuntu man pages converted to HTML via doclifter, for a definition of "working" that involves playing Battle of Wesnoth a lot. (It's a weekend.) We finally convinced him that both Fade and myself are fairly informal types and thus spending multiple days designing a wedding ceremony for us wasn't really something either of us were partiularly enthused about. He considers this a shame, but it's our wedding.
I'm poking at mdev.
I thought gcc wasn't building libgcc.a, so I removed --disable-shared and rediscovered that the build then breaks halfway through because it wants to link libgcc_s.so against crti.o, which is part of the C library (uClibc in this case), which needs to be built with the compiler. How the heck was THAT supposed to work? You can't build the cross-compiler without the target version of the library which you need the cross compiler to build. So --disable-shared is mandatory for cross compilers. Right.
Meanwhile, the other cross-compiler directories I already built have libgcc.a in them, but the gcc build directory that just completed "make all-gcc" didn't. (Although I tried it again and now I've got one. Dunno what happened there.)
Sigh. For the distcc accelerator I'm vaguely planning to work on next week, I need to get the cross-compiler doing C++ (so I can farm out whatever work the native compiler has back to the cross compiler through distcc). I suspect what I'm going to have to do is build _another_ cross compiler (after I have the target uClibc built) in order to add C++ support, because there's no way libstdc++ is going to be sane about anything it does. (Or perhaps I can use uClibc++? Hmmm...)
Anyway, this weekend I have to finish adding mdev to toybox and updating it to work with linux-2.6.20.
So, the libgcc.a it's making with --with-float=soft does _not_ contain floatsidf(). It has floatdisf(), floatdidf(), floatdixf(), and floatditf(), but no floatsidf(). So it's providing double precision but no single precision logic? Maybe? Asking on irc...
Ooh: How open source projects survive poisonous people.
Hmmm. The x86-64 build boots qemu and can run hello-static but not hello-dynamic. (If it seems silly to include copies of "hello world" in the image I build, I need to write up documentation on troubleshooting a new platform.) It could be a library search path issue, the darn thing's probably looking in "/lib64" or some such insanity even though it's a pure 64-bit system.
I wonder how much work it would be to get "strace" to cross-compile?
As for mips, I'm getting a kernel panic halfway through boot. Probably not a toolchain issue or I wouldn't get _that_ far, and userspace can't be involved yet because it hasn't gotten that far either. So the #1 suspect is kernel config. (Could also be qemu, but how?) Ah, the fact that it's building as mips-little-ending but booting as far as it is using the mips-big-endian qemu probably has something to do with it.
Off on the 5+ hour drive to Eric and Cathy's. Yay DC-to-AC converter to power ye olde laptop.
Boo darn cigarette lighter outlet thingy being dead and rusted (and it had a dime in it, but we fished that out at a rest stop.) It would be nice to figure out how to fix that for the trip back.
So, the mips thing seems like an easily fixable clash between the kernel config (thinking it's little endian) and the compiler and userspace thinking they're big endian, the x86_64 thing seems like a library path issue (probably the dynamic linker path having a "lib64" in it, so what am I doing? Trying to do arm soft float again, of course. It broke linking uClibc (again): "difftime.c:(.text+0x8): undefined reference to `__floatsidf'" and I still have no idea where this function is supposed to come from (gcc? uClibc?) but other people have made this work. According to google it's probably part of gcc, but it doesn't seem to be building it.
It's always interesting how selective and associative memory is. I never think about the car alarm that goes off for an hour between 1:30 and 2:30 am unless I'm about to go to sleep, so I never get any of my large hammers out of storage. I keep trying to remind myself to do this, as I drift off to sleep. Went outside when it went off sunday night (technically monday morning) to see which car was announcing "hey, the owners haven't been paying any attention to this thing for 45 minutes now!". It was several houses down, the sound carries surprisingly far.
Did I mention the hating Pittsburgh part?
Decided not to try to teach build.sh about dependencies because I just don't want to complicate it and because sometimes (such as a build being interrupted halfway through) it would just plain get the wrong result. I'll probably revisit this issue in future, but in the meantime I've gone back to teaching it how to build multiple architectures in sequence from a single command line. When I get an actual SMP test system I'll worry about optimizing parallel builds, but right now it's actually slower to build in parallel on my laptop due to cacheing issues.
So has anybody other than me started calling Vista "Windows Milennium Second Edition" yet? It's not selling. I mean it's really not selling.
Emailed Joe (the CEO) explicitly about my plans to move back to Austin. We'll see how this works out. I was willing to move to Pittsburgh for this job, but if I can't stand the city more than a year after moving here, no job is worth spending my life in a city I actively hate.
Went in to work rather than telecommuting today for an 11am all-hands meeting to introduce Ed, the new VP of Engineering, who has never used Linux before. All the engineers reporting to him met with him again at 3, and Jeremiah and I meet with him tomorrow at 2 to give him the presentation on the new build system. I'm getting a bit sick of giving this presentation.
Mark graduates from the University of Texas in 65 days. I should attend his graduation (although the only reason he's going is for his mother, same reason I went to mine). But A) I can get a head start on househunting in Austin (well, apartment hunting, Fade needs to be there for anything permanent and it's the wrong season to look for anything to buy anyway), B) if I don't make a move to hire Mark I'll lose him to another company, and that would suck rocks.
Nerve-wracking moment when the build system died spectacularly today (one build segfaulted and the other complained about a missing file halfway through the build), I thought my laptop might be packing up (the C key is one thing, bad memory is another), but it turns out that build.sh unconditionally calling host-tools.sh was a race condition waiting to happen. (Duh. I knew that, but when I wrote that multiple builds happened serially, and this time I ran them im parallel. All the targets use their own temporary directories, but the host tools tempdir is shared.) This race wasn't easy to trigger before I added User Mode Linux to the host tools build, turning a 3 second race window into a minute-plus. Yeah, one UML build died when the other process did an rm -rf on the linux source directory, and the other one segfaulted because its build was being modified behind its' back by the other process. Wheee.
The guy behind me on the bus ride home has been coughing loudly all trip. I wonder if it's contagious?
Jeremiah pointed out that on arm, -hda adds a scsi device even though the help text (and the name of the option) indicate otherwise. Got it booting, now on to mips which is being stroppy. (Specifically, my procedure for building readelf and ldd doesn't work on mips. Ok, fixed.)
Well, Fabrice wants to keep tcc on Savannah and wants to keep it on CVS. My tree is officially a fork. Now I just need to figure out how to name releases so they don't get confused with his version. (Perhaps "tcc-rl1" or some such.) And it turns out he did commit the broken-out patches I sent back in October, he just didn't mention it on the mailing list. (There's no such thing as cvs log -v, you have to look at each individual file to see if anything's changed. Did I mention CVS is obsolete?) I don't really want to maintain my own version of this, but I want one I can use. Hmmm...
The mips kernel isn't booting. I vaguely recall that qemu's -kernel loader wanted the vmlinux (ELF image) instead of zImage. Need to tweak the build for that.
All hands meeting at work tomorrow, looks like I'll be in the office instead of getting work done...
Interesting news on the commoditization front: John Carmack says 3D accelerator hardware is now a mature market filled with interchangeable "good enough" components he doesn't even bother to track closely anymore. And I quote, "...it’s hard to go wrong nowadays. The prices are so low relative to where things used to be and the performance is great... it’s been a long time since we’ve cared enough about the exact performance stuff to go and make exhaustive benchmarks on all the different things that we’ve done." Good to know. (Carmack's the founder and lead programmer of ID software, the people who do Doom and Quake and such.)
Hmmm. Poking at mdev on the bus ride home, and I'm not quite sure how
to deal with the makefile including xregcomp(). It's something that should
only be parsed when a command is configured in that uses regular expression
support. Otherwise, the C library may not have regular expression support
and it's a build break to #include
Caught up on tcc stuff today. I've now merged all of David Dodge's patches, gotten David Wheeler's float high-bit initialization issue fixed, I've asked about the Debian patch, and I've downloaded the gcc-on-tcc patch but haven't poked at it yet. I've also got reproduction cases for something like 3 different bugs that I have to figure out how to fix.
Fabrice Bellard offered me commit access to the official tcc cvs repository on Savannah. Much pondering at this end. I have philosophical objections to Savanah, I don't like CVS (even subversion's an improvement, and I much prefer Mercurial to either), I'm not particularly enthused about the idea of sifting through the past 6 months of patches in my repository and re-applying them to an out of date tree just to get back where I am now (especially after I already broke most of them out for Fabrice months ago, with no effect). And then after all that, I'd still be the only one actually committing anything on a regular basis...
Spent about half the day playing Kingdom Hearts 2. (The bit where you go up against 1000 low level heartless is _fun_.)
Headed out to the coffee shop around 5, and had the strange experience of really wanting a long walk (I haven't gotten enough exercise for months, and I LIKE walking), but getting to the end of the street and hating the cold and the hills and the dirt and the salt and the traffic and generally despising Pittsburgh so much I didn't want to walk through it. So I drove to Starbucks instead. (I want a nice long walk, but I don't want to be outside in Pittsburgh. I wasn't too thrilled about it when it _wasn't_ winter.)
Adding mdev to toybox. It's fair game, I wrote it, although I'm checking the revision history to see who else has touched it. As usual, I'm starting from my first checkin of the file (well, the last one that I was the only committer to touch) and then looking at each diff since and applying anything important by hand, usually in a different way. Plus random other cleanups from taking a fresh look at the code after all this time...
Now arm isn't building? Ah, I'm still trying to build ldd and readelf. The ulibc build for that is broken for cross compiling: either it builds a target binary (which you can't run on the host system) or it tries feed the compiler flags for the cross compiler to your host compiler (confusing it) and then link against the uClibc you just built (mixing target binaries and host binaries). Solution: bypass the uClibc build for "utils" entirely, and build ldd and readelf by hand.
Hmmm... Ok, the old kernel (the one in the mini-native-armv4l tarball from the 0.1.1 release) boots under qemu 0.9.0, so that's not the problem. The same command line running the kernel generated by the current build dies immediately with "qemu: fatal: integratorcm_read: Unimplemented offset 0x1f1018". So it's either the kernel, gcc, or a kernel .config issue. Doing an "hg annotate" on sources/configs/armv4l gives... Hmmm, the kernel miniconfig is mostly from hg-33 but several lines were added in hg-94, and three of them change the machien type, "CONFIG_ARCH_VERSATILE_PB=y" plus MACH_VERSATILE_AB and CPU_ARM926T. The old config was doing integratorp1026 which is apparently an unusual board (which the qemu developers picked because it had the largest feature checklist for them to implement, other boards are mostly a subset of that modulo new stuff just coming to market now like armv6). Ok, "qemu-system-arm -M ?" says I've got versatilepb and versatileab options (what's the difference?), try one... Hey, I get boot messages! Both work.
Now I need to ask Jeremiah how the heck he attached a blok device to his qemu instance earlier. 2.6.20 claims to have ATA support, but I don't know if the Versatile board QEMU is emulating has one. The command line options of qemu-system-arm only mention attaching IDE disks via -hda and such, no scsi command line arguments, but I thought Jeremiah was using SCSI? (I'm not familiar with these boards.)
Oh well, I can ask about that on Monday. In the meantime, I have a much improved failure to check in. :)
Finally got the packaging step checked in. The arm kernel isn't booting under qemu, which could be due to gcc 4.1.2, linux-2.6.20, or qemu 0.9.0. (I don't see how it would be the new uClibc snapshot, but that's also been upgraded since the last time I tried this.) Wheee...
Hey, 2.6.20 finally broke mdev. I've been waiting for that. (The old mdev ignores symlinks, because following the "device" symlink used to cause infinite loops. Now all the entries under /sys/class are symlinks into /sys/devices.)
A fun little detail about packaging is that the script that runs inside UML hasn't got include.sh so it can't pipe the output of tar to dotprogress, meaning I either have pages and pages of filenames scroll by, or I tell tar to be silent. The latter, I suppose.
There's a new version of dropbear out. Except my build system isn't using dropbear, that would be part of the stuff built by gentoo. I should get that messed with soonish...
My C key is sticking horribly. It's really annoying. It's now only responding about 1/3 of the times I hit it. This is really annoying, but where in pittsburgh does one get a laptop fixed (or buy a new one)? I'm still somewhat surprised this city has electricity.
I want to go home.
Overheard in the Te Cafe today: "If the end of the world comes I want to be in Pittsburgh because it'll take it another 5 years to get here."
Remembering lots of old things I fixed in Firmware Linux 0.8, I.E. last time I was playing around with User Mode Linux. Submitted a kernel patch for an issue I fixed in 2.6.13 that's _still_there_. I'm sure I submitted this patch before...
(Now the conversation's migrated to busting on Bill Clinton and/or Monica Lewinsky. Checking my watch he's been out of office for over six years. I'm tempted to quote sigmund freud, "Sometimes a cigar is just a cigar", but I'm staying out of this conversation too. I'm also reminded that my favorite coffee shop in Austin was open 24 hours so I could hang out there from midnight through 8 am, and that it was full of college students who generally don't harp on stuff that's a decade old now, and that Austin's liberal enough to have a federally recognized nude beach.)
Taking a break from gene2fs and adapting my old user mode linux hostfs trick to do quick and dirty ext2 packaging for now. (Time to get the firmware project unblocked, this is taking too long.) Found a bug in oneit while I was at it. Oh well.
I need to do a proper help system for toybox. Currently, the only help is in "make menuconfig", which is silly.
Stumbled across the wikipedia entry for tcc, and merged a few things it pointed to. I should probably spend a weekend on that at some point, it's too good a project to be languishing like this.
There's something in the hot water pipe of our shower, making the water spurt-stop-spurt-stop unless you turn it all the way up. Judging by the smell it started to give off after a few minutes of cooking in the hot water, I'm guessing "dead rat". I do _not_ feel clean right now, and I'd just like to repeat that I HATE HATE HATE HATE HATE PITTSBURGH!!!!!!!
Right. August. Six more months.
So, I've got a directory tree listing all the files to write out. This would be easier to do if that statement was true, but I'll get to that in a bit. I have to traverse this tree in multiple overlapping passes (one for inodes, one for data blocks). This is because to avoid seeking, the logic goes through each block in the filesystem and figures out (in order) what goes there, so the inodes have to know where the data blocks will be ahead of time, and then when it gets around to writing the data blocks the "what goes here" query has to agree with the earlier calculation. Add in the fact that there are several different contexts (writing out group overhead, inodes, indirect blocks, directory blocks, or file data blocks) and it gets... well silly, really.
I have dti and dtb pointers to the current position in the tree. There's no guarantee one is earlier than the other, inodes start out ahead of data blocks with small files but can fall behind if you have lots of small files.
Unfortunately, earlier assumption I mentioned not being true, isn't true because neither the root directory inode nor the journal inode are actually present in the directory tree I collected. The reason they're not present is they're not dynamically allocated. The first 10 inodes are reserved, and of that the root directory is inode 2, and the journal is inode 8. I have to write out these inodes manually, at fixed locations, but then writing out their data has to be part of normal traversal to allocate indirect blocks and such.
So for the inode traversal, they're not part of the tree. For data block traversal, they are. Possibly I can mark up the tree with an extra top level of nodes and start the block traversal at the root and the inode traversal with the first child node.
The next problem is "do I traverse the tree depth first or breadth first"? When writing out an individual directory I have to go breadth first, but this is really a special case of writing out a file, and the only information I need is inode numbers, not block numbers, so the preallocation problem's only half as bad as it could be. To deal with hardlinks I have to set the inode numbers to their final values in check_treelinks() anyway.
Breadth first inode allocation has the advantage of grouping the directory inodes which gives better directory performance in the resulting filesystem, although it's all cached into dentries after the first access, and lots of the traversals are depth first anyway, so that's not as big a concern as it might be. But depth first is the only sane way to find subdirectories and descend into them in one pass, which is why check_treelinks() is currently allocating inodes depth first. (It's already n^2, I'm trying to make that suck as little as possible.)
In terms of determining indirect block locations in the inode, the first 12 are sequential, #13 is also sequential (start of the indirect bloks), the 14th is at position 13+(BLOCK_SIZE/4) (after the singly indirect blocks), and the 15th is 13+(BLOCK_SIZE/4)+(BLOCK_SIZE/4)^2 (after the doubly indirect blocks). Except these are start + that many _available_ data blocks, not absolute data blocks. When a file crosses group boundaries, it has to skip past the blocks taken up by group accounting info.
Should this be part of file_blocks_used()? The block count that returns includes the index blocks and it's a pain to filter 'em back out, but if I recalculate that from the length I'm duplicating a 64 bit division with rounding, which is a bit painful. Having it in file_blocks_used() implies I should fill this out in check_treesize where that's being called, except that's called before I know the size of the root directory, and thus before I know the starting block.
Sigh. If I was in Austin, working on problems like this, I'd go for long walks while I worked out the details in my head. Unfortunately, I'm stuck in Pittsburgh until August. I am unhappy about this. (Even at the Te Cafe, the people at the next table are discussing why Britney Spears shaved her head. At great length. They actually _care_ about this. Welcome to Pittsburgh, where this qualifies as deep intellectual conversation. In case you were curious, the leading theory is to bypass drug testing on the way into rehab. After the first thirty seconds, you'd think they'd run out of elaboration on that sentence, but no.)
I suppose I can put the root directory at the _end_ of the filesystem, which seems a bit silly but again, dentry cacheing should cope with this after bootup. Less useful with the journal but I'll worry about that later. Unfortunately, this doesn't help skip the gaps caused by block group overhead, for which it needs to know the group count (so it can determine how many blocks the inode tables take up), and check_treesize is called before that gets determined in the filesystem auto-sizing case. So I can't assign final block locations that early.
I hate Pittsburgh.
I can assign block numbers starting from 0 for each file, and then adjust them later. This is probably the best approach.
The mke2fs/gene2fs code is coming up on 500 lines. Closing in on a working implementation, although the ext2 file format continues to be weird.
Ok, in the group descriptor table, entry used_dirs_count: why does it need to know? Just a sanity thing for fsck, or is this actually used to mount the filesystem? Seems kinda oddball to track that field there. And what's the deal with fragments? Linus vetoed fully implementing fragments back in 1995, but the fragment fields are still used. In the superblock s_log_frag_size and s_frags_per_group are copies of the appropriate "blocks" fields, but in the inode structure i_faddr, l_i_frag, and l_i_fsize are also fragment stuff, at at least the last two seem to be initialized to zero and then never used. Possibly i_faddr is used when appending, but if it has to work down through the index blocks that's not much help.
It's sad how much I have to recalculate. The superblock needs to know alloation totals. The group index needs to know allocation totals. The inode table needs to know allocation totals, and then the index blocks, and then the data blocks have to go in the right places. And each one advances at a different rate, so has to be calculated independently. Wheee...
David Mandala pointed me at the LSB compliance test suite, which would probably be a good thing to try FWL under once I get it to that point. Oh, nevermind. I extracted the test suite tarball and it contained 10 RPMs. (How is this wrong, let me count the ways: a tarball full of RPM files is roughly as clueless as a tarball full of zip files, any standard that requires RPM was brain damaged even _before_ RPM became unmaintained for several years, and any standard that won't even let you run the test suite without RPM is too stupid to live.) New policy: let the Linux Standard Base fade into well-earned irrelevance.
Snowed again, covered the car overnight. As far as I can tell, it's snowed every day for more than a week now. Dug it out anyway, as I'm losing my taste for trudging a mile through the snow in flip-flops and the finally-dug-out sneakers have no arch support (walking in them hurts after the first few hundred yards, but walking barefoot doesn't. Dunno what's up with that).Today I'm at Starbucks again (I.E. without internet access, but sometimes it's good to avoid the endless distraction that is the internet, and the music here's better here, currently Peter Gabrel's "big time") digging up an old script I did to turn my old busybox patch list into a mercurial repository. The busybox developers (ok, Bernhard) were really unhappy about switching the repository format so even though I got it working I didn't use it. But now I want a busybox repository on my laptop, which I can do things like "hg annotate" on without internet access.
The reason is I spent 4 years contributing code to BusyBox, and a lot of it's my code that I wrote and thus fair game for me to put into toybox. But most of it's also tangled up with other people's IP claims and I need to untangle it before I feel comfortable grabbing it. (I refuse to infect toybox with Bruce Disease. And nobody is using any part of toybox under GPLv3, ever.)
On a related note, I've developed an unexpected sympathy for Wil Wheaton, with regards to "William Fucking Shatner". I _SO_ want to refer to Bruce the same way, even though I don't swear so it feels weird to do so. Although in Wil's case WFS actually did some good work and kept at it for something like 30 years, and I went to the effort to _prove_ that BFP's contributions were damn trivial, a sideline to something else, that he abandoned the thing after the first few months, and that none of his "contributions" remain in the current project.
Bruce's BusyBox was a quick and dirty hack for the Debian install disk, just like Red Hat's "nash". Bruce glued together a dozen utilities mostly written by other people and then abandoned the result, because he had no vision beyond that and no CLUE what it could become. Erik Andersen then picked up the abandoned project two years later and turned it into something real; Erik could just as easily have based his work on nash instead of busybox and it would make absolutely no difference today. _Erik_ founded BusyBox, not Bruce. For Bruce to come back ten years later, out of the blue, and _dictate_terms_ was inexecusable.
Yes, I still hold a grudge.
So in order for me to grab my old BusyBox implementations of mdev, switch_root, bunzip, sed, sort, seq, mount, umount, nc, dmesg... It's not enough to just grab the most recent busybox version at the time I stopped maintaining 'em. I'm going through the revision history and picking out _just_my_code_. Which is a silly amount of work, but with BusyBox specifically I want a huge firewall between me and it.
The first thing I'm picking out is, of course, the new test suite infrastructure I wrote. Because I should get the test suite in place before toybox gets too much bigger.
Going back and filling in a hole in the archive, because I didn't even turn on my laptop saturday.
Slept late, thawed out the car (memo to self: "stuck in snow" takes about half an hour less to undo if you take the parking brake off), went to the grocery store, cooked two meals, watched Grosse Pointe Blank, read a bit, went to bed. Exciting.
Pondered a bit about the nature of Pittsburgh, how one notices that one's neighbor has a large dog by what it leaves in the snow in the tiny little patch of "dirt with snow on it claiming to be lawn" about 6 inches from the sidewalk, how over the course of the day this gets tracked into said sidewalk, and how this doesn't noticeably increase the amount of general background dirt that is life in Pittsburgh.
Each ext2 block group has a certain number of overhead blocks, so in order to figure out how many total blocks there are, I need to figure out how many groups there are. In order to figure out the per-group overhead, I need to know how many blocks the inode table takes up. This starts from the total number of inodes the filesystem needs (which I've got) and is divided by the number of block groups, rounded up to fill blocks. Meaning in order to figure out how many block groups there are, I need to know how many block groups there are. Sigh.
The approach I'm taking is to figure out how many blocks I can fit in the first block group, and then as long as I'm over budget try again with one more block group, and loop until it all fits.
Big meeting! Unfortunately, the only time the new build system was mentioned was when I brought it up, and I didn't learn anything new about management's committment to it (or lack thereof). Sigh. Couldn't quite squeeze in an individual meeting with Joe because his arrival into town was delayed 2 days by snow and as a result his time was insanely overscheduled.
Ok, pending issues for gene2fs. What should the user interface look like ("mke2fs -D dirname file.img", or "gene2fs dirname file.img"?) Keep in mind the need to specify nodes in a file (ala scripts/gen_initramfs_list.sh from the kernel) so device nodes and such can be created without needing root access.
For the generate empty file case, need to calculate block and inode totals, extrapolate a group count, etc. This is mostly a sequencing issue. Find blocks needed by all files (including directory entries and indirect blocks). Add blocks needed by first group overhead. While (too many blocks for groups) add group (and add block overhead for that group). Ok, I can make that work.
Initializing the superblock needs to know blocksize, but I need to know file length to init block size, but in the gene2fs the files fed in determine the file length, but the number of indirect blocks needed to store the files depends on the block size! Grr. Circular. Either I should hardwire a block size for the gene2fs when they don't specify it (4k is the modern standard), I need some kind of fudge factor, or I have to run the size calculations _twice_, which would suck.
So my attempt to install x86-64 kubuntu has been stalled at 4% of "Running 32gnome_power_manager" for 3 days now. I think it's time to call that one "hung". (It's an "add insult to injury" sort of hang since I chose Kubuntu rather than Ubuntu so it wouldn't install bits of gnome, but there it is.) Time to try the Very Special Installer.
[hours later]... It's at 6% of "Select and install software", saying "Please wait..." with no further details. Been there an hour. It installed rather a lot before this (this isn't the first entire progress bar to go by), and I have no idea why it prompted me for what resolution to run X11 in _after_ a couple hours of installing packages. Or what the current stalled progress bar is doing. Or how to do the equivalent of alt-F2... Ah, just do a cursor grab and then KDE won't pop up some window but the application will instead get it. Nice.
According to alt-F4 it's doing a lot of "selecting previously deselected package blah" and "unpacking blah". And now the progress bar's unstuck. I dunno if poking it unstuck it or if it was just a strange coincidence. Huh.
Upgraded the Firmware Linux build to gcc 4.1.2, and updated the uClibc snapshot, and backed out the overly compliated build.sh, and fixed a couple bugs in the sparc and mips kernel config files, and other stuff. The snow here was putting me to sleep earlier, but there's an unlimited supply of Diet Mountain Dew which doesn't _quite_ compensate, but certainly helps.
I'm slowly restoring my stash of empty Pepsi One and Coke Zero cans after _somebody_ threw them out, but at the moment I have far more Pepsi One cans than Coke Zero. We seem to be out of zero...
If the musical Oklahoma had been set in Pittsburgh, "Slurry with a fringe on top" would definitely be the song about how dirty the snow gets downtown. This was running through my head all the way to the bus. Annual weather conditions should not constitute a biohazard.
Fade's been setting up her new MacBook all morning, trying to get it in a state she can do work with it. Hasn't been easy. (She's been using Konqueror's fish:// mode to get a GUI way of doing scp to and from the server. Safari apprently doesn't support that. Installing firefox involved it popping up a window containing some icons and no text, which isn't a nice thing to do to someone new to MacOS X. And she's setting up her email client for the second time.)
Yesterday I googled for reactions to the 64-bit paper Eric and I wrote, finding both insightful commentary and deep willful stupidity. (And somewhere in between: Why is Richard Stallman's response to our paper more widely linked to than our original paper, when he makes it clear he didn't even bother to READ the darn thing?)
This comes to mind watching Fade try to set up a mac, almost as annoyed at the thing as she was with Linux. Macintoshes are the definition of user friendly, the problem is simply that it's not the machine she's used to, which she's spent years learning. She doesn't really want to leave that machine behind (and won't unless forced). I asked her why she bought a mac:
I suspect that the bad reviews of Vista are getting conflated with the hilariously bad reviews of the Zune. I don't think either of us have actually _seen_ it yet...
Added a short term todo list to the top of this page. The main problem with my todo lists is they go stale, so as long as I'm in the habit of editing this page fairly regularly (and after 3 years I think I can make this claim) daily, I might as well try keeping the todo list here and see how that goes.
Checking the Penguicon schedule I see they've scheduled Fade's and my wedding opposite an Elizabeth Bear reading, which makes Fade very unhappy. She suggested switching that reading with Charlie Stross's reading, except that Steve Jackson just disovered Charlie Stross's stuff and his reaction to getting to meet him was, and I quote, "woot". (Did I mention Steve's conducting the ceremony?)
Let's tally up the geek points here. I met the woman I'm marrying when she was working the Steve Jackson Games booth at ConDFW, and she now telecommutes for SJG was assistant webmistress. (She spent this morning squeeing over her new Mac laptop.) We're having our wedding at a combination science fiction convention and Linux expo that I co-founded. There's a very real possibility we'll move the wedding so it isn't opposite an Elizabeth Bear reading. Steve Jackson is officiating, and Eric Raymond is best man. There is a very real possibility that Steve will perform the ceremony in Full Pirate Regalia, although we aren't explaining to Fade's parents what this would signify. We're having special t-shirts made up to give to all the wedding attendees.
The fact that I write this while cross-compiling a Linux system in another window (tracking down strange bugs in the current uClibc svn) is sheer noise at this point. We've officially buried the needle of geekdom. I could wear battery powered deely-boppers during the ceremony and this would probably actually _reduce_ the geek quotient, or more likely simply pass unnoticed.
Updated my web page's main page today, and got a start on indexing my writing page. Much, muh, muh more to do there. And I want my C key working without having to hit it 3 times, darn it. The perils of a 2-year-old laptop.
Mark sent me a login to his Mac Mini box. I now have a darwin test environment, and can figure out why toybox is refusing to build there.
Last night we had another TimeSys Alumni reunion at the Church Pub Brew Works Thingy. (Apparently www.churchbrew.com.) On the ex-employee front, Garrett was there, and Tony and Sandra made it. On the current employee front, I was there, and Thomas and Jan and Al. Much beer with names like "pipe organ" was consumed by those who were into that sort of thing. I had a fairly expensive steak that was excellent, and some fairly expensive ice cream that wasn't.
Back at the Te Cafe for the first time in a week, relaxing and spending the gift certificates Fade got me. I might actually get gene2fs done today.
Fun notes about ext2: directory entries are padded to the next 4-byte multiple. I should write up some documentation for this... Heh. Discovered the Linux kernel's Documentation/filesystems/ext2.txt while writing this. Yes, I've gone this long without noticing it. (Well it didn't come up in a Google search!) So _that's_ where the top 32 bits of long file lengths go, in the dir_acl field. I had wondered, and was going to loopback mount some images and poke a bit to get it to generate some of this, when I got around to it...
I'm adding a "truncate" command to toybox, to call the truncate() syscall on a file. This is an easy way to create sparse files (the most obvious alternative is playing games with dd, which is awkward and doesn't create an entirely sparse file). Trying to figure out if I should call it setlength or something. (Asked on #cross-lfs and they seem ok naming it after the syscall.)
So "usage: truncate filename [size]", possibly with a -f to create a file that didn't previously exist (so you don't have to touch file; truncate file 8M). And yeah, it should understand the k/m/g extensions. I should make a function for this. No, I should add it to get_optflags(). Except this isn't an option flag. Hmmm...
Ah, I can add a -l option to touch and do this that way. That makes sense. A bit more work to implement all the SUSv3 touch options just now (and upgrade get_optflags()), but hey...
So last night we had a big TimeSys Alumni gathering (Piggy, Greg, Garrett, Christian, and Sam were there), and the current crowd was there too. Joe the CEO finally made it, and Thomas and Jan were in from Germany and Ron from California. I was there, as were Jeremiah and Phil. We had to move to a bigger set of tables. It was great. We were there for hours. Joe started taking notes halfway through.
As fallout from this, Christian wrote a script to turn my blog into an rss feed. I've added it to my rsync script.I think I may finally be getting some help on the new build system. Not quite immediately, but the week of meetings is over and Thomas says we've got management buy-in. We proposed buying a couple new quad x86-64 systems with 4 gigs of ram each (good thing to have on general principles and not a major capitol outlay as such things go), and staffing four new positions. Management is considering our request.
Jeremiah's already helping a bit, but we need to free up his time to work on IDE/GUI/installer stuff. We need a full-time QEMU expert (to be hired from outside the company, somebody who's already doing this at a fire-breathing geek level). We need a part-time gcc internals person (who we can probably get through Thomas's Linutronix; he's got somebody already, we just have to work out the details to pay him to spend time on our stuff). We need a Gentoo expert (if I can finagle it, I'm going to throw Mark at them as a part-time telecommuter until he graduates, although this position will eventually have to be full-time). And we'll need Q/A and sysadmin stuff for the new cluster, and so on.
Of course I haven't gotten any actual work done on the build system this week (I was in the office every day, no telecommuting while we have guests), but hey. That's what weekends are for. :)
My laptop's "c" key is still unhappy with me.
Still installing x86-64 ubuntu. I figured out why it was going nuts earlier: I made the 8 gig image file as root, ran qemu and had to reboot because it tried to go into full-screen mode and failed resulting in a garbage display I couldn't break out of (as it always does when you run it as root, perhaps it's intentional in a "don't do that!" sort of way), and on the reboot I was of course running as a normal user. But the image file still belonged to root, and was thus read-only. QEMU does not warn about this, but it can't write to the drive...
Darn it, I took my keyboard apart in Tuesday's meeting and cleaned huge amounts of cat hair out of the keyboard, but the C key is still intermittent. Deeply annoying. I wonder if Beth has any compressed air cans lying around?
Catching up on the QEMU mailing list, now that 0.9.0 is out. Among other things, trying to figure out what changes are in the new release. Lots of changes to support Darwin, which isn't surprising. Fade bought a Mac laptop to replace her old Dell.
Still installing x86-64 in an image file on the desktop machine at work. Got interrupted for various reasons and restarted more than once already...
Turned the alarm clock off rather than hitting the snooze alarm to avoid making the downstairs neighbors violent (yeah, same people with the marijuana smoke and the "take the baby" drama), and fell right back to sleep. I'm on the bus at 10:30 am, although it's about halfway through the bus trip now that the thing's drained enough to finally let me sit down. (Insert $RANT_ABOUT_BUS, a standard member of $I_HATE_PITTSBURGH.)
Thomas ... Ok, laptop battery died. It's not even lasting half a bus ride anymore. Sad.
Thomas is theoretically meeting with Joe this morning, so I'm trying to catch up on email and attack gene2fs some more. I've added another file mirror to engineering.timesys.com, to complement the one on landley.net. Both are rsynced from my laptop's sources/packages directory for the Firmware Linux builds I do, but the difference between them is the landley.net one is just the current stuff, while engineering.timesys.com is an rsync without --delete, so it should keep old versions around. (Admittedly only going back to now. I should add in the stuff for the 0.0.1, 0.1.0, and 0.1.1 releases. In theory it's just a matter of grabbing the appropriate download.sh, running it, and running my rsync script. In practice, some of the things (like the uClibc snapshot versions) have cycled off the upstream websites and I need to recreate them somehow. Fun...
There is a new qemu 0.9.0 release. I have downloaded and built it, and am in the process of installing the x86-64 version of kubuntu on it. Wow, it's slow emulating a 64 bit processor on a 32 bit laptop. I'd install the ppc version but qemu's Open Firmware clone is still refusing to give the ppc boot cd's kernel the appropriate device enumeration info. Not quite sure what to do about this. (If I have to become an open firmware expert, I may cry.)
On the #firmware irc channel on freenode, paul_c asked (while I was out at dinner with the engineering department), "what's a typical install size of an ix86 build ?". Running du on the mini-native directory gives 39186k (including rounding each file up to 4k), and the tar.bz2 is 13 megs. The cross compiler tarball is just over 11 megs, so most of this is the toolchain in mini-native.
Another day spent talking about the new build system (based on Firmware Linux). Essentially trying to convince my company to commit to it, and to deprecate the old build system.
I think it was a success. Engineering was unanimous, but the most management could get from management in the short term was approval to continue what we're doing. Getting the other engineers assigned to work on it, getting a couple of targeted new hires (we really need a QEMU guy, it would be nice to have dedicated Gentoo and GCC expertise as well, and a full time ARM guy wouldn't be bad either). And a committment to End-Of-Life the old build system will require further discussion, although they seem to be aware that this counts as "facing reality" at this point. We'll see what upper management comes back with.
Went out to dinner with Thomas and Jan again afterwards. This time Mahnoney's was open. He's probably sick of my company at this point. :)
Haven't gotten a darn thing done on toybox or the new build system. Maybe I'll get a bit in tonight, but I'm off to the bus bright and early again tomorrow. (This morning I clearly heard the downstairs neighbors screaming about coming upstairs and "beating [my] ass" over my alarm clock waking them up. Didn't know they could hear it, but I can clearly hear them shouting through the floor, so I guess they can. Ah, Pittsburgh. I hate it.)
It was cold enough I actually wore shoes today. The wind chill apparently made it down to -17 farenheit yesterday. With flip-flops, this made my toes hurt. I wound up having to borrow socks from Fade, because mine are in storage somewhere.
Thomas Gleixner has arrived, who is cooler than the rest of the engineering department combined (including me). And he brought Toblerone. Three different flavors.
Spent the day in the 5th floor conference room at work, with Thomas and Jan and Jeremiah and Phil, talking about the new build system. Went out to dinner with Thomas and Jan afterwards. Manhoney's was closed and I really don't know Pittsburgh that well, but we found food in the university area.
The deadline avoidance effect continues apace: spent about 3 hours this morning cleaning the apartment. I can see all of the couch and the kitchen table now. There was sweeping. I used a lint brush. (I tried using the lint brush directly on a couple of the cats, cutting out the middleman as it were. Didn't accomplish much, but both cats seemed to enjoy it, which is probably the important thing.)
Finally headed out to visit the tea shop (can't get much work done at home with 5 cats demanding attention), but it was just too cold so I grabbed the car and drove to Starbuck's again. (Starbuck's has parking for 50 cents per hour.) So I'm still offline, but that's probably a good thing. Focus on ext2. (And figure out why the "c" key isn't responding to normal pressure... Ok, that was frightening. It's possile there's more hair in my keyboard than on my head. Need better tools. But not right _now_.)
Ok, I made a function to collect the tree (struct stat plus the filename). In the mke2fs case this is just one node for "lost+found". I need to use this to fill out the free blocks and free inodes fields in the superblock (ignoring journaling and htree hash for now, that's all that's left).
I can actually abuse the stat.st_blocks field to precalculate how many blocks each file takes up (which isn't quite trivial because longer files have indirect index blocks), and get a running total from that. And inode total is just a count of nodes in the tree (I think).
For reference, here are the largest file sizes each level of indirect block indexes can hold:
blocksize direct indirect double indirect triple indirect 1024 12,288 274,432 67,383,296 17,247,252,480 4096 49,152 4,243,456 4,299,210,752 4,402,345,721,856
Mmmm. Evil molasses cookie. I see why starbucks spreads so much. (Well I spend days at the Te Cafe for work. It's a bit odd spending weekends there too.)
I also need to calculate the link count for each file (what I get from the filesystem is useless because there could be hardlinks outside the tree I captured), and the trivial way to do that is O(n^2) but I'm not implementing a hash table for this because I want simple code. There's probably some trivial way to do this, but I'm not seeing it right now. (Hibernating. $HATE_FOR_PITTSBURGH.) I'll worry about hard links later.
Need a way to iterate through the tree linearly; not all algorithms are quite happy with this recursion thing. The easy way is probably to add a "parent" pointer.
Found an off-by-one error in a gcc error message. Minor cosmetic issue (the member name of a "has no member" message is one short). I remember the first compiler bug I found (in the Borland C++ 3.0 optimizer, an increment was optimizing out when it shouldn't, but it came back when I stuck in a printf). That was a huge thing for me, blow to the worldview and a definite "no fair!" moment. But finding a bug in gcc? It's been _weeks_ since I've found a new one of those, I was due for one...
Ok, 8000 miles isn't _too_ bad for an oil change...
Headed to Starbucks for a change of pace. (I'm not into coffee, but the hot chocolate's nice, and I got a cookie.) No internet access here (the variant of no access is "T-mobile hot-spot". I don't spring for pay toilets either.) But I've got a laptop full of data and it's actually less distraction from working on gene2fs.
So naturally, I spent the past hour writing The commoditization argument for open source. Reminds me of why I found graduate school (and deadlines in general) so motivating: there's no limit to the amount of effort I'll put in as long as I have something to avoid.
Right, gene2fs...
Fade got me Te Cafe gift certificates for my birthday. Woot! (And Cathy got me a book on the history of the potato, which is something everybody should have.)
Dug out my old Red Hat 9 image to regression test the toybox build, and fixed toybox so it would build under Red Hat 9. I note with mild amusement that "make defconfig" and "make allnoconfig" produce almost the same size executable, because gcc 2.96 doesn't do --gc-sections. I should teach the makefile to skip .c files that don't have anything the current .config needs, but I refuse to go into a huge amount of detail for this. I'll probably continue to build "lib/*.c" and use the CONFIG_APPNAME stuff to determine which files to build out of "toys/*.c". Of course this means that gene2fs has to be a sub-config-thingy of mke2fs (CONFIG_MKE2FS_GEN), but that's not so bad.
Today's question is "what information do I need to capture for gene2fs"?
In order to create an ext2 filesystem as a streaming archive (output you can pipe, with no seeks), I have to do some serious readahead. The superblock needs to know how many free blocks are left in the filesystem, so I have to read everything in order to write out the very first block.
However, I don't actually need the file data yet, just the directory information, so the problem isn't insurmountable. Just annoying. (About like rsync's "creating file list" step before it actually does anything.)
If the filesystem it's packing up changes between the time it does the directory scan and the time it goes back to read the file contents, I don't care. Don't do that. I'll pad the file with zeroes and emit a warning, but fundamentally this is not my problem.
The hard question this raises is how much directory information should the first pass capture? The tree of files can get very large, and I want to be as memory-efficient as possible. All I really _need_ is filename and size in bytes, I can write out superblock and group descriptors based on this information. On the other hand, I could also fill out a complete inode structure for each one on the first pass, which means I only iterate over the directories once, which is faster and less easily confused by files disappearing out from under it. The second time I'd just open the file and read its' contents, and if it's missing or short: zero pad.
It looks like I'll have to define a structure to capture this data. I need a cross between ext2_dentry and ext2_inode, because neither one stores all the fields I need, and both together would be kind of bloated for this purpose (block[15], padding, fragments, acl fields, generation, deletion time...). I'm a little unhappy about writing code to copy the data from one structure to another, but oh well.
I need to figure out how sparse files are stored, with holes in 'em. Possibly zero index blocks?
I'm starting to sympathize with the ubuntu guys for removing bash and replacing it with dash. Bash 3.x is JUST SO BROKEN. Bash 2.05b was evil, but it was consistent and usable about it. Bash 3.x is spawning a subshell for { } (I.E. it's confusing curly brackets with parentheses), and then when you hit ctrl-c it kills the parent shell but not the subshell. (So you get your prompt back but the output is overwritten by the continuing background process. Exactly what ctrl-c is supposed to do, sure...)
So there's mirror of Firmware Linux now. Wheee.
The mips build breaking turned out to be due to adding toybox to the build. The Firmware Linux build was always my most comprehensive stress test for BusyBox (and the BusyBox work that spun out if it is how I become BusyBox maintainer), and now that I've started over from scratch I've always meant to hook toybox up to FWL. So when I added it to the build, I built all the commands and put them in the path for the rest of the build.
Echo broke. How does echo break the build? Because the mips build (and not the armv4l or i686 builds) was doing "echo ln -fb blah blah", and echo was going "unrecognized option -fb". Yes, echo option parsing should stop at the first non-option argument, and yes I implemented support for that in lib/args.c, and yes once I added the + to the start of the echo option list string in toylist.h to indicate I wanted to _use_ this functionality, it started working right.
Tangents. Firmware Linux has _always_ been about tangents. The initramfs documentation I wrote up? FWL tangent. The User Mode Linux howto and tty layer fix? Firmware Linux tangents. The gcc wrapper script, miniconfig, everything I've learned about Squashfs and QEMU and tcc and mercurial...
It's a marvelous hobby. :)
And now, alas, it looks like I need to learn how to use gdb with qemu. Grumble.
$DAILY_RANT_ABOUT_HATING_PITTSBURGH
So the gcc thing looks like an mmap() issue and might actually be solvable by taking a larger hammer to it at configuration time. I'll play with that tomorrow.
Today, I just got my new mke2fs writing an almost-valid empty ext2 filesystem (according to dumpe2fs, anyway; haven't run fsck on it yet). It's missing two superblock fields and the root directory. Now to add the contents precalculation infrastructure, and probably get it to detect when it's writing out a block of zeroes so it can lseek instead (to keep sparse files as sparse as possible when formatting them). And I should add a journal at some point, and figure out what to do about directory hashing, but those are both optional.
Adding toybox to the firmware build pointed out to me that I haven't had a release in a while (downloading a random mercurial URL is weird because there's no consistent filename in the URL), and there's no "make install" target. (Infrastructure, yes. Target, no...)
Slight design problem with toybox install: running the toybox command gives a list of symlink names to make back to the toybox executable, but when you're cross-compiling you can't run it. Oops.
In response to some email on the firmware list, I fixed a couple things with the build scripts. The big one is that each build stage now deletes the appropriate build subdirectory so it doesn't get confused by what's already there. (I'm not working out extensive dependencies, instead I'm just making each build stage reasonably short and self-contained, and letting them be called individually.)
Also, I moved host-tools to have its' own build directory, and taught the top build.sh to do multiple targets at once (currently iterating through them sequentially, although they could be backgrounded in parallel fairly easily). It also writes log files without needing that on the command line. Possibly there should be command line options for log files and backgrounding.
Plus I updated to a more recent uClibc snapshot. The mips build broke and I don't know why, maybe this will fix it.
Slept 12 hours (it was cold with the windows open), woke up, the bathroom was still full of marijuana smoke (I think the vents in there connect directly with downstairs). Checked my email and got Fade's email from yesterday, warning me about the marijuana smoke while I was still at the coffee shop.
I hate Pittsburgh. Not quite enough to break the lease yet, but that only lasts until August...
Jeremiah booted up the native build environment under qemu (using his new config which I checked in yesterday, and telling qemu-system-arm it had a versatile board with a scsi hard drive). It worked pretty well, except that gcc barfed, saying it had exhausted virtual memory. (What?)
We tried again feeding twice as much memory to qemu (-m 256), which made no difference. The problem could be uClibc, but if so I'd expect bash and busybox to be affected and they don't seem to be. The problem could be the kernel, but ditto. The problem isn't even the gcc wrapper or the main gcc executable, it's cc1.
I just guessed in jeremiah's direction that maybe /dev/shm needs a tmpfs on it. Nope, that didn't fix it.
So shortly after getting home from work I asked myself "ok, what happens if the end of the filesystem occurs in the middle of a block group's metadata?" And I started playing around with dd to make various multiples of 1024 bytes to see what mke2fs would do with 'em. My first guess was it would split the blocks across two groups, but apparently this is not the case.
I already knew that the skipped 1024 bytes at the beginning (to preserve the partition table and boot sector if you format /dev/hda) counts as a block with 1024k blocks, but for all other block sizes it's just an idiosyncrasy of the first block. (This is why there's a "first data block" field of the superblock, one of the many fields that doesn't actually need to be recorded and could just be calculated instead.)
My first new learning experience was that the number of blocks in the filesystem is rounded down to page size (4k), so with 1024 byte blocks an 8195 byte filesystem is made as 8192 blocks. (I vaguely knew this from my years on the linux-kernel mailing list, but had forgotten. It's a kernel internal implementation thingy, and no I'm not humoring Alpha's 8k page size, because Compaq killed that processor.)
But the _fun_ part is that an 8196 block filesystem is 8193 blocks, and so is everything up to 8380 blocks when it finally clicks up to the new size. With 1024 byte blocks, that's 8192 bit bitmaps, yielding 8192 blocks per group, plus the one strange skipped block (that counts as a block with 1k blocks). The second block group has a backup of the superblock and group descriptors, and if there isn't enough space for the group's metadata it just trims the filesystem down to the end of the previous group. (Of course the first free block in the new group is 8328, so the existing mke2fs won't use the new group unless it has not just the metadata but 52 free blocks in it as well, which is 187 blocks total. I don't know why.)
I'm not sure this behavior makes much sense. The new metadata is only really four blocks in this case (the first two of which are the backup superblock and group descriptors, the other two are the block and inode bitmaps). What's taking up the majority of the second group's metadata is the second inode table (blocks 8197-8327, or 131 blocks), but that's not new space. There were just as many inodes in the 8193 block filesystem with only one group, in that case the first group's inode has a 262 block long inode table. So ignoring the "round down to page size" bit for a moment, I could see trimming a filesystem of 8194-8197 blocks down to 8193 (since the 4 block overhead of adding a second group eats any gains), but after that you start gaining data blocks...
Of course this assumes you can control where the new group starts. The superblock has lots of fields that can theoretically be calculated and have no reason to be stored, and blocks_per_group is one of them. If the e2fs driver actually reads that and uses it, then I might be able to do more intelligent group distribution...
Ok, crunch time.
The way I'm doing gene2fs and mke2fs, they're essentially the same program. It's just that mke2fs is hardwired to create one empty directory (lost+found) and gene2fs takes a source directory instead. (Which means you might not have a lost+found, but then I delete that from my filesystems anyway. Yes, I know why it's there and I know doing so inconveniences fsck, I just don't care much. If you're filesystem's horked enough for it to make a difference (despite journaling), you _really_ should have it backed up. Oh well.)
Is _just_ the superblock backed up in the groups that have superblock backups, or are the group descriptors also backed up? Hmmm...
Ok, first_ino is 11 because the first 10 inodes are reserved for various things (such as the root directory). Why does the superblock have def_resuid and def_resgid when the root inode has owner and group fields?
Yet another recruiter emailing me out of the blue (this time on behalf of Akamai), and as usual they'd want me to move to California. (I don't want to move to california. It's crowded and insanely expensive.)
(Note: got back home around 7pm and the apartment was completely FULL of pot smoke leaking up from downstairs. Opened all the windows despite the snow, and half an hour later it still hadn't cleared. I had to go outside a few times just so I could breathe. Wound up going to bed around 7:30 and sleeping until morning. I wouldn't have made disparaging remarks about California afterwards: it's GOT to be better than Pittsburgh. I hate this city.)
Very much hibernating today. Woke up after noon, ate too much, played some Disgaea, didn't set foot outside of the house, accomplished nothing.
I'm still hibernating. I hate being up north during the winter. Caffeine helps, but not enough.
It would be nice if the ext2 group descriptor table always fit nicely into one ext2 block. Heck, if it always fit in 4k it would make programming easier because I could just treat (struct ext2_group)toybuf as an array. Unfortunately, the math doesn't work out right, because sizeof(ext2_group) is 32 bytes and at 4k block size that's 128 groups, and a 4k block allocation bitmap (which limits group size) is 32868 blocks, and 32768*128 is 4,194,304, which is 1/1024th of the largest possible block count in ext2 (1<<32).
How the 48 bit block count of ext4 works into all this, I'm have yet to look at. I'm not even making a journal for ext3 yet.
So now I'm wondering, do I make my code small simple and stupid and write out each group structure as I construct it (32 bytes at a time), or do I group them into blocks (making the code bigger but both faster and "less disgusting")...
I'd have finished all this a week ago if I was _awake_. I miss Austin.
And the ext2 file format continues to make me go "whaaa...?" The superblock's inodes_per_group field is uint32_t, but the free_inodes_count field in the group structure is uint16_t. Add in a potential endianness conversion and I think I've got to call SWAP_LE32 on it _twice_ on big endian platforms to assign one to the other. Wheee...
27 megabytes of spam to download this morning. Well, it's down from 67 megabytes on monday (although that _was_ coming back from a weekend without net access at ConFusion).
So Thomas Gleixner gets into town (along with his sidekick, Jan) on Sunday February 4, and heads back on the 10th. I'd like to have the new build system in a state where we can make a go/no-go decision on it while they're in town. This means I have a deadline.
At David Mandala's suggestion, I'm trying nolisting to cut down on spam. My spam filter's now taking half an hour to sort through 10,000 spam messages each morning, which is a bit silly.
Tracking down a bug in toybox's get_optflags() (parsing mke2fs -N 1234). As I use more complex option parsing, I squeeze out more bugs. I expected this: I tend to program by tangent at times. Oh, _duh_. I apparently never implemented the integer argument case. Yeah, that'd do it...
Ok, the reason I left this unimplemented is I never worked out a good syntax for optionally specifying integer ranges. Hmmm...
Apparently Ulrich Drepper, the maintainer of glibc, has been at war with RMS and the FSF since 2001. (See the part at the end, starting "And now for some not so nice things.").
I suppose I should put together a big master list of "why the FSF is bad for Free Software". But today, I'm fiddling with mke2fs.
And got nothing done. :( (But I _did_ track down an archive of old WWIV .mod files, which I've mirrored.)
Pittsburgh in January apparently means "hills with snow on them make concrete sidwalks slippery". (Why don't I get fortune cookies that say anything useful like that? None of the fortune cookies I've gotten in the past few years even _try_ to predict anything. The ones I've gotten in pittsburgh are just "advice cookies", with no predictive aspect at all.) Slipped a lot, hurt my hand slightly, but now I'm having some very nice tea at the tea shop. (Not my usual Assam Borengajulie, but instead something called Ilam.)
After a day at said tea shop, I've now got most of the ext2 superblock being filled out correctly, and I understand _why_. Now I just have to write out the rest of the block groups, each of which may have a superblock backup (at least if it's a power of 3, 5, or 7), group descriptors (whatever those are), block bitmaps, inode bitmaps, inode tables, and data blocks. (Although the last two only come in for gene2fs, I think. Still, that's the point of this exercise: to get a gene2fs I can use for Firmware Linux.) Pretty decent progress, and it's only dinner time.
I've also got fresh reasons to boggle at the kmail developers. (When I right click on a message I do NOT want to go to it and open it up, I'm trying to select "delete" and I don't want my heavily loaded system to load a kanji font for spam.) Similarly, when I right click the "trash" folder I don't want to go into it and open it up. For some reason, doing an "empty trash" when it's open causes it to re-index the thing (or something equally expensive) after each delete, and I can get 10,000 spams in a day winding up in there. It's been emptying it for half an hour minutes, CPU pegged, and won't even repaint the window until it's done. Sigh. (Yes, I should remember to use the pulldown menu and empty trash from that to avoid this bug. Obviously trying to empty trash through the trash folder icon is a dumb thing to do.)
So what happens when you specify a number of inodes that doesn't divide evenly across blocks in block groups? It rounds up.
Sat down with hexdump and looked at an ext2 superblock today. Reading through source code is a long and roundabout way to get anything done when you're dealing with something that's "evolved" as much as mke2fs. (And yes I'm thinking "antibiotic resistance" when I say that.) Reading through the actual data structures is much more interesting.
The general theory is to start with dd if=/dev/zero, then run mke2fs on the image, and anything nonzero was presumably set that way for a reason.
Some of it's obvious, such as skipping the first 1k (which may have the boot sector and the partition table in it), then writing the superblock, then padding the superblock so it's also an even kilobyte.
I don't know why the size of the ext2 inode structure (in bytes) is written into a field of the ext2 superblock, and then treated as a constant elsewhere. I don't know why reserved blocks would have a default uid/gid. I don't know what fragments are in this context, or why "blocks per group" is 8 times larger than the total number of blocks in the filesystem. I don't know why mke2fs initialized htree_hash_seed and hash_version without setting the hash bit in the feature flags. And I have no idea why htree_hash_seed would be set to a uuid (unless they need a big essentially random number for some reason).
But that's not the serious boggling stuff. The serious boggling comes from the difference in padding at the end of the structure. Here's the end of the superblock structure from the kernel's version of include/linux/ext3_fs.h from linux 2.6.19.2:
__le32 s_first_meta_bg; /* First metablock block group */
__u32 s_reserved[190]; /* Padding to the end of the block */
(I don't know what a metablock group is either, it's set to 0 so I can probably ignore it. It's ust the last field before the "reserved" padding at the end to make the superblock end at an even 4k.)
Except that s_reserved[0] has a _timestamp_ in it. (The unix time of when mke2fs was run.) What the...? Leaking data in uninitialized memory? Nope. Here's the same snippet of the end of the superblock e2fsprogs version:
__u32 s_first_meta_bg; /* First metablock group */
__u32 s_mkfs_time; /* When the filesystem was created */
__u32 s_jnl_blocks[17]; /* Backup of the journal inode */
__u32 s_reserved[172]; /* Padding to the end of the block */
So a year old version of mke2fs is defining fields in the ext2 superblock that the current linux kernel still doesn't know about. Is it for e2fsck? Some strange debugging purpose? Dunno.
I've also googled for "ext2 layout" and gotten a half-dozen web pages I need to read. Printed out four of them. This could take a while...
Most of today was driving back from ConFusion. Sleepy. Went to bed early saturday (before midnight even, and that was after six diet mountain dews and two non-diet) because I was just too tired for room parties. Slept late, napped in the car home, and now going to bed early.
I hate pittsburgh, and I hate being up north during the winter. I'm going to be lethargic until at least _march_. Sigh.
I've been spoiled by Penguicon. This one seems to be a "relaxicon", which is ok I suppose. Oh well, this is the convention Tracy chaired before Penguicon, so I've been meaning to come here for 5 years now. Finally did.
Added echo support to toybox, with -n and -e. Support for parsing escapes required character by character output to do cleanly, and that meant ansi FILE * output instead of writing to filehandles (to buffer output instead of doing one syscall per character), so I wound up doing xprintf() and xputc() to catch and properly error out when output fills up. (Yes, try "echo blah > /dev/full" and notice the error message.)
Also added "yes". Dithered for a bit over whether it should share code with echo, decided against it for the moment.
Poking at NOFORK again. The current NOFORK means "you _cannot_ fork", because this must run in the same process context. It's not a performance thing, it's because things like "cd", "exit", and "export" have to occur in the shell's process context to do their thing.
For performance reasons, I'd like to figure out how to make forking optional in toysh. When ctrl-z happens, we need a separate process, but _until_ ctrl-z happens we don't. This means the obvious way to defer work until it's needed (and potentially avoid it entirely if it isn't) is fork in the signal handler. (All together now: "eeeeeeeewww".) Unfortunately, SIGSTOP is one of the two signals you can't intercept (the other is SIGKILL), so I can't do that (and it wouldn't work with vfork() anyway, and I'm not sure you actually can fork in a signal handler).
What I really need a second process context for is so the job can suspend and give me the shell prompt back. Shell has to be a separate process from what gets suspended, therefore I need a process context. Where I'd like to _avoid_ spawning a new process context is running something simple like "echo" within a shell script. Fork and exec are very heavyweight there. Exec potentially runs the dynamic linker, and vfork() can't run anything interesting without exec(), and it's nice to use vfork() for nommu systems.
In theory, this is a performance hack I only care about when running a shell script: Interactive use from an actual prompt aren't going to notice fork overhead. Add to this the fact that I already need process contexts for the second and later processes in any job (any command line that has a pipe in it is going to fork), and this becomes a fairly special-purpose optimization. I already fork a process context to run a shell script, and when we're _not_ interactive we transfer control to whoever ran us when we get suspended. So the optional fork avoidance becomes something triggered by !interactive.
Heading to ConFusion with Garrett. Eric and Cathy Raymond are there (we're meeting them for dinner if we can ever get through this traffic).
Re-acquainting myself with the guts of the bunzip2 code I wrote 4 years ago. Have I mentioned I love mercurial? Checkins from my laptop on the road. Good thing.
Fade headed off to the airport this morning to visit Seattle for a week.
The Firmware packaging for QEMU is being a pain. My current theory is that when you feed QEMU the "-kernel" and an unpartitioned ext2 image as "-hda", the bootloader overlay it puts at the start of the hda image (so the fake bios can boot from that and load the kernel) overwrites the first ext2 superblock. (It's recognizing the drive, but just won't mount the sucker. I should make an initramfs so I can examine the start of hda to see what the heck's actually there.) But I've been distracted by other things and haven't nailed it down yet.
I hadn't known about Myrvhold's Laws before reading about Andrew Tanenbaum's talk at LCA. "Software is a gas which expands to fill its container" is a marvelous quote, and exactly why I'm doing toybox and Firmware Linux. It's easy for small software to become general purpose but much harder to trim down large programs. (Too bad the rest of Tanenbaum's talk degenerates into the same old prejudices he had in 1990, but I'm not surprised. The PC is not reliable hardware, touching devices wrong can lock up the whole bus so it doesn't help to have your driver is in userspace. And doing away with memory mapped I/O sounds remarkably like calling DMA a "performance hack" 15 years ago...)
Got a very simple bzcat checked into toybox, and I'm working my way through the busybox change history from my original bunzip drop to the present to see what changes are interesting. Handling several of them in a different way (toybox has error_exit(), don't need longjmp() for error handling).
Also replacing reread() and rewrite() (which handle -EINTR), which shouldn't be needed thanks to sigaction(SA_RESTART). (Here's what Linus had to say about that.) What they don't say is how to set SA_RESTART globally. Do Linux's SIG_DFL signal handlers have SA_RESTART? I've found hints that it's on by default, but haven't been able to confirm this. Google is being unhelpful, and it's an obscure enough question to ask that I'm mostly getting head scratching and blank looks.
Wandered into McDonald's this morning, planning to get one of those McGristle things. I've had a bit of a weakness for them ever since Howard Tayler described the two-pancake-with-maple-syrup-sandwich nature of the things, although I haven't had one in months because they stop serving them at 10:30 am and I'm seldom awake enough to be hungry before then.
McDonald's had a sign out front advertising "2 for $2 Sausage McGriddles" and that sounded like a good deal, so I asked how much to upgrade that to Sausage McGriddle with Egg (which is what I normally get). Answer: you can't. One of those is $2.39, two would be $4.78.
I didn't particularly need two of them, and I'm not deeply bothered by an extra $1.39 for one, but it was enough to talk me out of a fairly unhealthy sandwich since I had a slim-fast in my backpack anyway. I'm not going to give money to a company that's being actively stupid to my face. (Note to McDonalds: your advertising just _cost_ you a sale I'd already come in to the store intending to make. Congratulations. Probably saved me about 200 calories, and $1.39.)
Yesterday it rained in Pittsburgh, all day, turning the accumulated spit on the streets into a thin slime that clung to everything and smelled terrible.
Today: Sore throat, stuffed up head, and it's snowing. Wheee...
Banging on the mke2fs implementation a bit more, since that's low-brain at this point...
And apparently my web page and email were down because Eric upgraded named to a version that no longer recognizes "#" as a comment character. So nice of them to break a previously working config file, wasn't it? It doesn't even warn about it (instead just goes "servfail" on resolves), and when you do run it with -g to run it in the foreground and see the log to stderr, it says "bad owner name" which is a truly _horrible_error message for this issue...
Sigh. Eric has a tendency to use tools that were a good idea 15 years ago, but not today. (He's still using sendmail, too.) I need to set up my own server...
I am definitely overusing the elipsis today...
Ooh!
+static ssize_t return_EIO_ssize(void)
+{
+ return -EIO;
+}
+
+
+static ssize_t bad_file_write(struct file *filp, const char __user *buf,
+ size_t siz, loff_t *ppos)
+ __attribute__((alias("return_EIO_ssize")));
I hadn't known about attribute alias. I'm sure I can find a use for that. :)
On the toybox front, xread() probably doesn't work the way I thought it would, which his actually something of a relief. I had reread() checking for -EINTR and reissuing the system call, and according to the Linux Programming Guide I can just feed SA_RESTART to sigaction() and the system should do it for me. (Long ago I thought it was the case that the system could restart interrupted system calls for me, but busybox behaved otherwise and I couldn't find code to handle this in uClibc so I assumed busybox was right. My bad. Busybox was probably being paranoid to humor Slowaris or Tru64 or some such system I actively don't care about. Not my problem.)
This leaves me with xread(), readall(), and xreadall() (and corresponding write functions), which I can live with.
Got Firmware Linux 0.1.1 out today. Not a major thing, there's a news entry on the FWL news page about it. Mostly it means there's prebuilt tarballs for mips and sparc now. Still working on ext2 packaging, getting distracted by all sorts of things...
And I finally got around to benchmarking the qemu vs native performance, and got 28 minutes to build binutils under Red Hat 9, vs 3 1/2 minutes to build natively. The downside is that RH9 uses gcc 2.96 and Ubuntu is using gcc 4, so the difference is probably signifcantly larger. Sigh. Still, ballpark of a factor of 10, which is about what I expected.
Trying to install ubuntu 6.10 into an image via qemu is turning out more frustrating than I expected, due to ubuntu 6.10 being broken. It no longer has a text install, so I have to do graphics mode. It defaults to 1024x768 meaning the window is too big to fit on my desktop (the bottom hides behind my kde taskbar, so I can't access the taskbar in the qemu window. When I move my host desktop's taskbar out of the way and then put it back again, kde wants to reposition all the windows and moves the top of the qemu window off the top of the screen).
So I restart the gentoo install and tell it "800x600", which is a nice size that fits on my desktop. And it does indeed boot in 800x600, until X11 comes up at which point it switches back to 1024x768 again. Because obviously what I told it I _want_ doesn't matter, the hardware CAN do more and that's all that's important.
Sigh. Oh, and booting with 128 megs of ram means it swaps its guts out and the OOM killer aborts the install process. I can't get far enough in the install to partition and format the image to give it a swap file, because it doesn't have enough memory to go that far. Because it wants to boot the full desktop before it's actually installed, and had no option to do otherwise. This is immensely stupid.
Bertl (whose name turns out to be Herbert Poetzl, which sounds familiar) just sent me a mipsel config file. Considering I've had this half-done for weeks with no attention to spare for it, I'm happy to throw out the one I was working on (for "on my todo list" values of working) and check in his instead. :)
This spawned a discussion about .config files in here documents being somewhat disgusting (which it is), so I got off my ass and converted at least the kernel .config files into miniconfig files, using my horrible converter script (which I also checked in).
I seem to have broken the mips build. Weird. It's the qemu application emulation sanity check, which is refusing to load it. I can't tell if it's really a static executable: Even when I build it static, readelf -a gives me 251 symbols... Ah, that's the debug section, strip debug info and it's happy again. Ok...
Possibly an endianness issue? Yes, it is. It's using $KARCH, so it's getting "mips", but it needs both qemu-mipsel and qemu-system-mipsel, which brings up the question "is that the right $KARCH tuple"? Trying it the other way...
Also, the sparc kernel giving me a bunch of "section mismatch" warnings while linking doesn't exactly inspire confidence either. And busybox has lots of warnings about getting int where it expects long int in printfs. Wheee...
Stuff to deal with in the morning, I suspect. Bedtime.
So Thomas Gleixner forwarded my gcc bug to one of his linutronix guys who messes with the guts of gcc, and the reply he got back was that gcc 4.1 is unsalvageable (not just due to this bug but due to others even on i686) and I should just revert back to 3.x or 4.0, or use a cvs snapshot. (I'm awaiting to hear which snapshot would be good to try.)
And people wonder why I mess with tcc? It may be unfinished and not nearly as ambitious as gcc about supporting lots of target platforms or optimizing, but it's not fundamentally broken by design and saturated with FSF megalomania ("all-linux-tuples-end-in-gnu-dammit"), either. It has the _potential_ to be a far better compiler, someday...
Sigh. The tinycc-devel@nongnu.org mailing list has decided to spam-block my email. ("reason: 550 We don't want your stock spam". It's deeply sad that Fabrice Bellard still thinks that anything to do with the Free Software Foundation, the Gnu project, or Savannah is remotely worthwhile. This hasn't been true for a decade.)
Oh well. If I decide to put effort into my fork I can start up a new mailing list. Until then, my policy on spam blocking is that anybody who blocks my email isn't somebody I want to talk to anyway. (I've _never_ fought an incompetent spam block, it's not worth my time. Your spam filter is not my problem.)
In other news, there's now a #firmware IRC channel on freenode, via which Bertl sent me a sparc config file (but still hasn't given me a name to credit the checkin to).
Snow today. Hibernating. Zzzzz...
Upgraded firmware to the 2.6.19.2 kernel, which should make the filesystem corruption bug (when you combine mmap and a file size change) go away. Not that I've hit it, but it makes me feel better.
Busy couple of days.
There's a new release of genext2fs, and I'm poking at it in passing. Got caught up in some work related stuff today having to do with expanding the engineering department (which I said I wouldn't talk about here and I won't, although Joe's finally talking to David and I think this is a good thing.)
On the relocatable toolchain front, Peter Mazinger pointed me at lib/interp.c which is a generated file, created during the uClibc build for the sole purpose of overriding the normal library loader path.
Since when should shared libraries care about this sort of stuff? You need a working shared library loader to load the suckers in the first place, and it has to be able to load more than one library, so why can't it just work recursively here? You can't specify two shared library loaders for a single executable (so all the shared libraries you load _directly_ have to be able to use the same library loader). So what possible use is there to have libraries loaded _indirectly_ specify a different library loader? What's the POINT?
Grumble
I've been grumpy the past few days. Even with the flourescent lights, the whole "sun going down at 4 pm on overcast days" thing is getting to me. I don't mind the cold so much (not that we've actually had much this year, we had the heater off and the windows open today), but I mind the short days.
Michael-Luke Jones pointed me at e2fsimage, as a less-abandoned alternative to genext2fs. Unfortunately, it wants the e2fsprogs header files, which I'm not thrilled about.
I'm already a decent chunk of the way into adding mke2fs to toybox (without needing e2fsprogs). Been distracted by lots of other things, but it's not that hard of a problem...
The help system's for toybox is being a bit of a problem. I've got to specify help text entries in Config.in for menuconfig anyway, and I've been trying to shoehorn these help entries into something approaching man page format so I can generate man pages (and html) from them. (BusyBox didn't have this problem because it just redundantly specified several different help entries for each applet: in usage.h there was the one line version, the --help version, and the man page version, and none of them had anything to do with the help text in Config.in.)
What I'm currently doing is having the first line of some Config.in help text entries start with "usage:" to signify it's in man page format. Then I can string together the various usage strings for the same command, and even have a "synonyms:" line when a command has more than one name. (Note, I haven't written the help text generator for this yet. It'll need to make a usage.h file to #include to get --help text, a big html page for online help, and individual man pages for each command.)
Unfortunately the logic gets convoluted and conditional. For example, I've got a number of minor variants of the same functinality: mke2fs, mke3fs, mkfs.ext2, mkfs.ext3, gene2fs, and gene3fs. The mke?fs and mkfs.ext? versions are simple synonyms. Any version with a 3 in it sets -j by default, and those only show up when you enable MKE2FS_JOURNAL. And gene?fs versions take two arguments instead of one (in which case the first argument is still the name of the filesystem to create but it will be created and auto-sized if it doesn't exist, and the second argument is a directory to populate the new filesystem from). Those only show up when you enable MKE2FS_GEN. But what about gene3fs? That only shows up when you have both JOURNAL and GEN enabled for MKE2FS. Several of the individual -X options also need to be guarded by more than one symbol.
I've made this kind of nested guarding easy to do in C via the USE() macros, but the config hiearchy (X depends on Y) has to be expressed in Config.in, and that needs help text. I want to generate the man pages generated from the Config.in help text to avoid maintaining duplicate information, but I also want the Config.in help text to be human readable which means I can't insert conditional markup into it (unless I want to teach kconfig about it, and that whole directory's mostly unmodified upstream Linux kernel code at present. That code doesn't wind up in the toybox binary and I'm not out to write a configuration system, I'd _really_ rather not make any changes to it I can't push back upstream.)
Perhaps I can generate everything (Config.in, usage.h, toybox.1, and toybox.html) from some master file in yet another format, but _what_ format? It has to specify help text and configuration hierarchy, and right now Config.in does that (sort of badly, but I can work with it).
Ironically, the actual mke2fs functionality has been taking up _far_ less of my time than wrestling with the help system design. It's a simpler problem, and one that's been solved before.
Firmware Linux was mentioned by Linux Weekly News.
Boggle.
In other news, Phil at timesys responded to my observation that the gnu implementation of "cat" being 833 lines of C code was what drove me to look at busybox in the first place. He pointed me at the gnu implementation of Hello World. Allow me to quote from the IRC channel:
[16:30]landley: GNU hello ... 332117 bytes in tar.bz2 ... 4362 lines total in 11 .c and 13. h files ... 199 files total (lots of i18n stuff) [16:38] landley: it's up to version 2.2 now
While I realize that the purpose of that program is probably to provide examples of things like i18n and demonstrate coding standards, I find the second link is still almost as boggle-worthy as the first link. Just not nearly as unexpected.
Heading into the office for an all hands meeting. No idea what definition of "on vacation" this falls under.
And so, I tackle mke2fs and gene2fs in toybox. The conventional one actually seems to be alled "genext2fs", but the sourceforge page for that points to debian for the old release (warning that it's old) and says to get a newer version just grab a cvs snapshot (giving a dead link, although I could probably track it down through sourceforge). I started writing my own some time ago as a reaction against the horrible mke2s that went into busybox, and I might as well dig that up and brush it off now that I find myself in need of one.
Why do I need a gene2fs? Because loopback mounting an ext2 image requires root access, and my build script shouldn't. So how do I package up the mini-native root filesystem in order for qemu to mount it?
It's slow going because I've got my bunzip2 and mdev code queued up to go into toybox, and I'm going through the history to make sure I'm just taking _my_ code so I don't have to ask anybody about stripping the stupid "or later" notice out of it (not that I think this is a _real_ requirement, but I'm still mad). I really should finish that before tackling something new, and I keep getting sidetracked by that. Oh well.
In theory I'm still on vacation. In practice since I spent the past week working on the build system and I'm likely to continue doing so this week, and that's work related, I've logged into the company's IRC and I'm calling it telecommuting half days. (This way I don't have worry whether I'll have enough vacation time to go to ConFusion later this month.)
Playing with an Ubuntu 5.10 build environment under qemu last night, I noticed that qemu's --configure barfs if you haven't got SDL installed (so it can fake up a graphics card), and tells you to rerun the configure with "--disable-gfx-check". Now what _I_ want to know is what is the _point_ of a configure that forces you to create a command line specifying the environment? I can see the point of --disable-gcc-check since that's warning you the sucker won't build without a patch, but qemu builds without SDL support just fine, you just don't get an emulated graphics card. That kind of thing is what ./configure is _for_, isn't it?
Grrr. Teaching the build to humor qemu, and while I'm at it teach the cross compiler to only build qemu for the platform it actually needs (I.E. --target-list), and this is a cleaner way to handle that SKIP_QEMU_TEST thing anyway.
Happy new year!
I'm running the build on a Red Hat 9 image under qemu. Once again I'm suffused with sheer joy that hardware that doesn't even actually exist is _running_my_code_. And there was much rejoicing.
...Up until the point it died attempting to do "make headers_install" in the 2.6.19.1 kernel, which apparently doesn't work under Red Hat 9, "No rule to make target `|', needed by `asm-generic'." It's getting caught up on the differing bugs in the older version of gnu make; it expects the _current_ bugs, not the old ones.
Hmmm. I can install Ubuntu 5.10, can I still install it? And is it old enough to be a regression testing environment?