Mercurial > hg > aboriginal
comparison design.html @ 2:be48c60f9edb
Update it a bit. Talk about environmental dependencies, etc.
author | landley@driftwood |
---|---|
date | Sun, 13 Aug 2006 22:30:17 -0400 |
parents | 9b6afefcc082 |
children |
comparison
equal
deleted
inserted
replaced
1:9add2b1ccdfa | 2:be48c60f9edb |
---|---|
41 </ul> | 41 </ul> |
42 | 42 |
43 <p>Unfortunately, that doesn't work yet.</p> | 43 <p>Unfortunately, that doesn't work yet.</p> |
44 | 44 |
45 <h2>Some differences between theory and reality.</h2> | 45 <h2>Some differences between theory and reality.</h2> |
46 | |
47 <h3>Environmental dependencies.</h2> | |
48 | |
49 <p>Environmental dependencies are things that need to be installed before you | |
50 can build or run a given package. Lots of packages depend on things like zlib, | |
51 SDL, texinfo, and all sorts of other strange things. (The GnuCash project | |
52 stalled years ago after it released a version with so many environmental | |
53 dependencies it was impossible to build or install. Environmental dependencies | |
54 have a complexity cost, and are thus something to be minimized.)</p> | |
55 | |
56 <p>A good build system will scan its environment to figure out what it has | |
57 available, and disable functionality that depends on stuff that isn't | |
58 available. (This is generally done with autoconf, which is disgusting but | |
59 suffers from a lack of alternatives.) That way, the complexity cost is | |
60 optional: you can build a minimal version of the package if that's all you | |
61 need.</p> | |
62 | |
63 <p>A really good build system can be told that the environment | |
64 it's building in and the environment the result will run in are different, | |
65 so just because it finds zlib on the build system doesn't mean that the | |
66 target system will have zlib installed on it. (And even if it does, it may not | |
67 be the same version. This is one of the big things that makes cross-compiling | |
68 such a pain. One big reason for statically linking programs is to eliminate | |
69 this kind of environmental dependency.)</p> | |
70 | |
71 <p>The Firmware Linux build process is structured the way it is to eliminate | |
72 environmental dependencies. Some are unavoidable (such as C libraries needing | |
73 kernel headers or gcc needing binutils), but the intermediate system is | |
74 the minimal fully functional Linux development environment I currently know | |
75 how to build, and then we chroot into that and work our way back up from there | |
76 by building more packages in the new environment.</p> | |
77 | |
78 <h3>Resolving environmental dependencies.</h2> | |
46 | 79 |
47 <p><b>To build uClibc you need kernel headers</b> identifying the syscalls and | 80 <p><b>To build uClibc you need kernel headers</b> identifying the syscalls and |
48 such it can make to the OS. Way back when you could use the kernel headers | 81 such it can make to the OS. Way back when you could use the kernel headers |
49 straight out of the Linux kernel 2.4 tarball and they'd work fine, but sometime | 82 straight out of the Linux kernel 2.4 tarball and they'd work fine, but sometime |
50 during 2.5 the kernel developers decided that exporting a sane API to userspace | 83 during 2.5 the kernel developers decided that exporting a sane API to userspace |
89 <p>This means that to compile a minimal build environment, you need seven | 122 <p>This means that to compile a minimal build environment, you need seven |
90 packages, and to actually run the result we use an eighth package (QEMU).</p> | 123 packages, and to actually run the result we use an eighth package (QEMU).</p> |
91 | 124 |
92 <p>This can actually be made to work. The next question is how?</p> | 125 <p>This can actually be made to work. The next question is how?</p> |
93 | 126 |
94 <h2>Additional complications: cross-compiling and avoiding root access</h2> | 127 <h2>Additional complications</h2> |
128 | |
129 <h3>Cross-compiling and avoiding root access</h2> | |
95 | 130 |
96 <p>The first problem is that we're cross-compiling. We can't help it. | 131 <p>The first problem is that we're cross-compiling. We can't help it. |
97 You're cross-compiling any time you create target binaries that won't run on | 132 You're cross-compiling any time you create target binaries that won't run on |
98 the host system. Even when both the host and target are on the same processor, | 133 the host system. Even when both the host and target are on the same processor, |
99 if they're sufficiently different that one can't run the other's binaries, then | 134 if they're sufficiently different that one can't run the other's binaries, then |
141 should always work on a newer kernel than the one it was built to expect, but | 176 should always work on a newer kernel than the one it was built to expect, but |
142 not necessarily an older one.)</p> | 177 not necessarily an older one.)</p> |
143 | 178 |
144 <h2>Ways to make it all work</h2> | 179 <h2>Ways to make it all work</h2> |
145 | 180 |
181 <h3>Cross compiling vs native compiling under emulation</h3> | |
182 | |
146 <p>Cross compiling is a pain. There are a lot of ways to get it to sort of | 183 <p>Cross compiling is a pain. There are a lot of ways to get it to sort of |
147 kinda work for certain versions of certain packages built on certain versions | 184 kinda work for certain versions of certain packages built on certain versions |
148 of certain distributions. But making it reliable or generally applicable is | 185 of certain distributions. But making it reliable or generally applicable is |
149 hard to do.</p> | 186 hard to do.</p> |
150 | 187 |
154 and why you might want to do it. Keep in mind that I wrote that for a company | 191 and why you might want to do it. Keep in mind that I wrote that for a company |
155 that specializes in cross-compiling. Personally, I consider cross-compiling | 192 that specializes in cross-compiling. Personally, I consider cross-compiling |
156 a necessary evil to be minimized, and that's how Firmware Linux is designed. | 193 a necessary evil to be minimized, and that's how Firmware Linux is designed. |
157 We cross-compile just enough stuff to get a working native compiler for the | 194 We cross-compile just enough stuff to get a working native compiler for the |
158 new platform, which we then run under emulation.</p> | 195 new platform, which we then run under emulation.</p> |
196 | |
197 <h3>Which emulator?</h3> | |
159 | 198 |
160 <p>The emulator Firmware Linux 0.8x used was User Mode Linux (here's a | 199 <p>The emulator Firmware Linux 0.8x used was User Mode Linux (here's a |
161 <a href=http://www.landley.net/code/UML.html>UML mini-howto</a> I wrote | 200 <a href=http://www.landley.net/code/UML.html>UML mini-howto</a> I wrote |
162 while getting this to work). Since we already need the linux-kernel source | 201 while getting this to work). Since we already need the linux-kernel source |
163 tarball anyway, building User Mode Linux from it was convenient and minimized | 202 tarball anyway, building User Mode Linux from it was convenient and minimized |
184 <p>The main difference with QEMU is a sharper dividing line between the host | 223 <p>The main difference with QEMU is a sharper dividing line between the host |
185 system and the emulated target. Under UML we could switch to the emulated | 224 system and the emulated target. Under UML we could switch to the emulated |
186 system early and still run host binaries (via the hostfs mount). This meant | 225 system early and still run host binaries (via the hostfs mount). This meant |
187 we could be much more relaxed about cross compiling, because we had one | 226 we could be much more relaxed about cross compiling, because we had one |
188 environment that ran both types of binaries. But this doesn't work if we're | 227 environment that ran both types of binaries. But this doesn't work if we're |
189 building an ARM or PPC system on an x86 host.</p> | 228 building an ARM, PPC, or x86-64 system on an x86 host.</p> |
190 | 229 |
191 <p>Instead, we sequence more carefully. We cross-compile a minimal | 230 <p>Instead, we need to sequence more carefully. We build a cross-compiler, |
192 intermediate system from the seven packages listed earlier, and build a kernel | 231 use that to cross-compile a minimal intermediate system from the seven packages |
193 and QEMU. We run the kernel under QEMU with the new intermediate system, and | 232 listed earlier, and build a kernel and QEMU. Then we run the kernel under QEMU |
194 have it build the rest.</p> | 233 with the new intermediate system, and have it build the rest natively.</p> |
195 | 234 |
196 <h2>Alternatives to emulation</h2> | 235 <p>It's possible to use other emulators instead of QEMU, and I have a todo |
236 item to look at armulator. (I looked at another nommu system simulator at | |
237 Ottawa Linux Symposium, but after resolving the third unnecessary environmental | |
238 dependency and still not being able to get it to finish compiling yet, I | |
239 gave up. Armulator may be a patch against an obsolete version of gdb, but I | |
240 could at least get it to build.)</p> | |
241 | |
242 <h3>Alternatives to emulation</h3> | |
197 | 243 |
198 <p>The main downsides of emulation are that is it's slow, can use a lot of | 244 <p>The main downsides of emulation are that is it's slow, can use a lot of |
199 memory, and can be tricky to debug if something goes wrong in the emulated | 245 memory, and can be tricky to debug if something goes wrong in the emulated |
200 environment. Cross compiling is sufficiently harder than native compiling that | 246 environment. Cross compiling is sufficiently harder than native compiling that |
201 I consider it a good trade-off, but there are alternatives.</p> | 247 I consider it a good trade-off, but there are alternatives.</p> |
237 | 283 |
238 <p>(Note: QEMU can export a host directory to the target through the emulated | 284 <p>(Note: QEMU can export a host directory to the target through the emulated |
239 network card as an smb filesystem, but you don't want to run your root | 285 network card as an smb filesystem, but you don't want to run your root |
240 filesystem on smb.)</p> | 286 filesystem on smb.)</p> |
241 | 287 |
242 <h2>Filesystem:</h2> | 288 <h2>Filesystem Layout</h2> |
243 | 289 |
244 <pre> | 290 <p>Firmware Linux's directory hierarchy is a bit idiosyncratic: some redundant |
245 Our directory hierarchy is a bit idiosyncratic: some redundant directories have | 291 directories have been merged, with symlinks from the standard positions |
246 been merged, with symlinks from the standard positions pointing to their new | 292 pointing to their new positions. On the bright side, this makes it easy to |
247 positions. | 293 make the root partition read-only.</p> |
248 | 294 |
249 The set "bin->usr/bin, sbin->usr/sbin, lib->usr/lib" all serve to consolidate | 295 <h3>Simplifying the $PATH.</h3> |
296 | |
297 <p>The set "bin->usr/bin, sbin->usr/sbin, lib->usr/lib" all serve to consolidate | |
250 all the executables under /usr. This has a bunch of nice effects: making a | 298 all the executables under /usr. This has a bunch of nice effects: making a |
251 a read-only run from CD filesystem easier to do, allowing du /usr to show | 299 a read-only run-from-CD filesystem easier to do, allowing du /usr to show |
252 the whole system size, allowing everything outside of there to be mounted | 300 the whole system size, allowing everything outside of there to be mounted |
253 noexec, and of course having just one place to look for everything. (Normal | 301 noexec, and of course having just one place to look for everything. (Normal |
254 executables are in /usr/bin. Root only executables are in /usr/sbin. | 302 executables are in /usr/bin. Root only executables are in /usr/sbin. |
255 Libraries are in /usr/lib.) | 303 Libraries are in /usr/lib.)</p> |
256 | 304 |
257 For those of you wondering why /bin and /usr/sbin were split in the first | 305 <p>For those of you wondering why /bin and /usr/sbin were split in the first |
258 place, the answer is it's because Ken Thompson and Dennis Ritchie ran out | 306 place, the answer is it's because Ken Thompson and Dennis Ritchie ran out |
259 of space on the original 2.5 megabyte RK-05 disk pack their root partition | 307 of space on the original 2.5 megabyte RK-05 disk pack their root partition |
260 lived on in 1971, and leaked the OS into their second RK-05 disk pack where | 308 lived on in 1971, and leaked the OS into their second RK-05 disk pack where |
261 the user home directories lived. (/usr was what /home is today.) | 309 the user home directories lived. When they got more disk space, they created |
262 | 310 a new direct (/home) and moved all the user home directories there.</p> |
263 The real reason we kept it is tradition. The execuse is that the root | 311 |
312 <p>The real reason we kept it is tradition. The execuse is that the root | |
264 partition contains early boot stuff and /usr may get mounted later, but these | 313 partition contains early boot stuff and /usr may get mounted later, but these |
265 days we use initial ramdisks (initrd and initramfs) to handle that sort of | 314 days we use initial ramdisks (initrd and initramfs) to handle that sort of |
266 thing. The version skew issues of actually trying to mix and match different | 315 thing. The version skew issues of actually trying to mix and match different |
267 versions of /lib/libc.so.* living on a local hard drive with a /usr/bin/* | 316 versions of /lib/libc.so.* living on a local hard drive with a /usr/bin/* |
268 from the network mount are not pretty. | 317 from the network mount are not pretty.</p> |
269 | 318 |
270 I.E. The seperation is just a historical relic, and I've consolidated it in | 319 <p>I.E. The seperation is just a historical relic, and I've consolidated it in |
271 the name of simplicity. | 320 the name of simplicity.</p> |
272 | 321 |
273 The one bit where this can cause a problem is merging /lib with /usr/lib, | 322 <p>The one bit where this can cause a problem is merging /lib with /usr/lib, |
274 which means that the same library can show up in the search path twice, and | 323 which means that the same library can show up in the search path twice, and |
275 when that happens binutils gets confused and bloats the resulting executables. | 324 when that happens binutils gets confused and bloats the resulting executables. |
276 (They become as big as statically linked, but still refuse to run without | 325 (They become as big as statically linked, but still refuse to run without |
277 opening the shared libraries.) This is really a bug in either binutils or | 326 opening the shared libraries.) This is really a bug in either binutils or |
278 collect2, and has probably been fixed since I first onticed it. In any case, | 327 collect2, and has probably been fixed since I first noticed it. In any case, |
279 the proper fix is to take /lib out of the binutils search path, which we do. | 328 the proper fix is to take /lib out of the binutils search path, which we do. |
280 The symlink is left there in case somebody's using dlopen, and for "standards | 329 The symlink is left there in case somebody's using dlopen, and for "standards |
281 compliance". | 330 compliance".</p> |
282 | 331 |
283 Similarly, all the editable stuff has been moved under "var", including | 332 <p>On a related note, there's no reason for "/opt". After the original Unix |
284 tmp->var/tmp, and etc->var/etc. (Whether /etc really needs to be editable is | 333 leaked into /usr, Unix shipped out into the world in semi-standardized forms |
285 an issue to be revisited later...) Remember to put root's home directory | 334 (Version 7, System III, the Berkeley Software Distribution...) and sites that |
286 somewhere writeable (I.E. /root should move to either /var/root or | 335 installed these wanted places to add their own packages to the system without |
287 /home/root), and life is good. | 336 mixing their additions in with the base system. So they created "/usr/local" |
288 | 337 and created a third instance of bin/sbin/lib and so on under there. Then |
289 Other detail: /tmp is much less useful these days than it used to be. Long | 338 Linux distributors wanted a place to install optional packages, and they had |
290 ago in the days of little hard drive space and even less ram, people made | 339 /bin, /usr/bin, and /usr/local/bin to choose from, but the problem with each |
340 of those is that they were already in use and thus might be cluttered by who | |
341 knows what. So a new directory was created, /opt, for "optional" packages | |
342 like firefox or open office.</p> | |
343 | |
344 <p>It's only a matter of time before somebody suggests /opt/local, and I'm | |
345 not humoring this. Executables for everybody go in /usr/bin, ones usable | |
346 only by root go in /usr/sbin. There's no /usr/local or /opt. /bin and | |
347 /sbin are symlinks to the corresponding /usr directories, but there's no | |
348 reason to put them in the $PATH.</p> | |
349 | |
350 <h3>Consolidating writeable directories.</h3> | |
351 | |
352 <p>All the editable stuff has been moved under "var", starting with symlinking | |
353 tmp->var/tmp. Although /tmp is much less useful these days than it used to | |
354 be, some things (like X) still love to stick things like named pipes in there. | |
355 Long ago in the days of little hard drive space and even less ram, people made | |
291 extensive use of temporary files and they threw them in /tmp because ~home | 356 extensive use of temporary files and they threw them in /tmp because ~home |
292 had an ironclad quota. These days, putting anything in /tmp with a predictable | 357 had an ironclad quota. These days, putting anything in /tmp with a predictable |
293 filename is a security issue (symlink attacks, you can be made to overwrite | 358 filename is a security issue (symlink attacks, you can be made to overwrite |
294 any arbitrary file you have access to). Most temporary files for things | 359 any arbitrary file you have access to). Most temporary files for things |
295 like the printer or email migrated to /var/spool, where there are persistent | 360 like the printer or email migrated to /var/spool (where there are |
296 subdirectories with known ownership and permissions. | 361 persistent subdirectories with known ownership and permissions) or in the |
297 | 362 user's home directory under something like "~/.kde".</p> |
298 The result of all this is that a running system can have / be mounted read only | 363 |
299 (with /usr living under that), /var can be ramfs/tmpfs with a tarball extracted | 364 <p>The theoretical difference between /tmp and /var/tmp is that the contents |
300 into it, /dev can be ramfs/tmpfs managed by udev (with /dev/pts as devpts under | 365 of /var/tmp should definitely be deleted by the system init scripts on every |
301 that: note that /dev/shm naturally inherits /dev's tmpfs), /proc can be procfs, | 366 reboot, but the contents of /tmp may be preserved across reboots. Except |
302 /sys can bs sysfs. Optionally, /home can be be an actual writeable filesystem | 367 deleting everyting out of /tmp during a reboot is a good idea anyway, and any |
303 on a hard drive or the network. | 368 program that actually depends on the contents of /tmp being preserved across |
304 </pre> | 369 a reboot is obviously broken, so there's no reason not to symlink them |
370 together.</p> | |
371 | |
372 <p>(I case it hasn't become apparent yet, there's 30 years of accumulated cruft | |
373 in the standards, convering a lot of cases that don't apply outside of | |
374 supercomputing centers where 500 people share accounts on a mainframe that | |
375 has a dedicated support staff. They serve no purpose on a laptop, let alone | |
376 an embedded system.)</p> | |
377 | |
378 <p>The corner case is /etc, which can be writeable (we symlink it to | |
379 var/etc) or a read-only part of the / partition. It's really a question of | |
380 whether you want to update configuration information and user accounts in a | |
381 running system, or whether that stuff should be fixed before deploying. | |
382 We're doing some cleanup, but leaving /etc writeable (as a symlink to | |
383 /var/etc). Firmware Linux symlinks /etc/mtab->/proc/mounts, which | |
384 is required by modern stuff like shared subtrees. If you want a read-only | |
385 /etc, use "find /etc -type f | xargs ls -lt" to see what gets updated on the | |
386 live system. Some specific cases are that /etc/adjtime was moved to /var | |
387 by LSB and /etc/resolv.conf should be a symlink somewhere writeable.</p> | |
388 | |
389 <h3>The resulting mount points</h3> | |
390 | |
391 <p>The result of all this is that a running system can have / be mounted read | |
392 only (with /usr living under that), /var can be ramfs or tmpfs with a tarball | |
393 extracted to initialize it on boot, /dev can be ramfs/tmpfs managed by udev or | |
394 mdev (with /dev/pts as devpts under that: note that /dev/shm naturally inherits | |
395 /dev's tmpfs and some things like User Mode Linux get upset if /dev/shm is | |
396 mounted noexec), /proc can be procfs, /sys can bs sysfs. Optionally, /home | |
397 can be be an actual writeable filesystem on a hard drive or the network.</p> | |
398 | |
399 <p>Remember to | |
400 put root's home directory somewhere writeable (I.E. /root should move to | |
401 either /var/root or /home/root, change the passwd entry to do this), and life | |
402 is good.</p> | |
403 | |
404 </p> |