comparison design.html @ 2:be48c60f9edb

Update it a bit. Talk about environmental dependencies, etc.
author landley@driftwood
date Sun, 13 Aug 2006 22:30:17 -0400
parents 9b6afefcc082
children
comparison
equal deleted inserted replaced
1:9add2b1ccdfa 2:be48c60f9edb
41 </ul> 41 </ul>
42 42
43 <p>Unfortunately, that doesn't work yet.</p> 43 <p>Unfortunately, that doesn't work yet.</p>
44 44
45 <h2>Some differences between theory and reality.</h2> 45 <h2>Some differences between theory and reality.</h2>
46
47 <h3>Environmental dependencies.</h2>
48
49 <p>Environmental dependencies are things that need to be installed before you
50 can build or run a given package. Lots of packages depend on things like zlib,
51 SDL, texinfo, and all sorts of other strange things. (The GnuCash project
52 stalled years ago after it released a version with so many environmental
53 dependencies it was impossible to build or install. Environmental dependencies
54 have a complexity cost, and are thus something to be minimized.)</p>
55
56 <p>A good build system will scan its environment to figure out what it has
57 available, and disable functionality that depends on stuff that isn't
58 available. (This is generally done with autoconf, which is disgusting but
59 suffers from a lack of alternatives.) That way, the complexity cost is
60 optional: you can build a minimal version of the package if that's all you
61 need.</p>
62
63 <p>A really good build system can be told that the environment
64 it's building in and the environment the result will run in are different,
65 so just because it finds zlib on the build system doesn't mean that the
66 target system will have zlib installed on it. (And even if it does, it may not
67 be the same version. This is one of the big things that makes cross-compiling
68 such a pain. One big reason for statically linking programs is to eliminate
69 this kind of environmental dependency.)</p>
70
71 <p>The Firmware Linux build process is structured the way it is to eliminate
72 environmental dependencies. Some are unavoidable (such as C libraries needing
73 kernel headers or gcc needing binutils), but the intermediate system is
74 the minimal fully functional Linux development environment I currently know
75 how to build, and then we chroot into that and work our way back up from there
76 by building more packages in the new environment.</p>
77
78 <h3>Resolving environmental dependencies.</h2>
46 79
47 <p><b>To build uClibc you need kernel headers</b> identifying the syscalls and 80 <p><b>To build uClibc you need kernel headers</b> identifying the syscalls and
48 such it can make to the OS. Way back when you could use the kernel headers 81 such it can make to the OS. Way back when you could use the kernel headers
49 straight out of the Linux kernel 2.4 tarball and they'd work fine, but sometime 82 straight out of the Linux kernel 2.4 tarball and they'd work fine, but sometime
50 during 2.5 the kernel developers decided that exporting a sane API to userspace 83 during 2.5 the kernel developers decided that exporting a sane API to userspace
89 <p>This means that to compile a minimal build environment, you need seven 122 <p>This means that to compile a minimal build environment, you need seven
90 packages, and to actually run the result we use an eighth package (QEMU).</p> 123 packages, and to actually run the result we use an eighth package (QEMU).</p>
91 124
92 <p>This can actually be made to work. The next question is how?</p> 125 <p>This can actually be made to work. The next question is how?</p>
93 126
94 <h2>Additional complications: cross-compiling and avoiding root access</h2> 127 <h2>Additional complications</h2>
128
129 <h3>Cross-compiling and avoiding root access</h2>
95 130
96 <p>The first problem is that we're cross-compiling. We can't help it. 131 <p>The first problem is that we're cross-compiling. We can't help it.
97 You're cross-compiling any time you create target binaries that won't run on 132 You're cross-compiling any time you create target binaries that won't run on
98 the host system. Even when both the host and target are on the same processor, 133 the host system. Even when both the host and target are on the same processor,
99 if they're sufficiently different that one can't run the other's binaries, then 134 if they're sufficiently different that one can't run the other's binaries, then
141 should always work on a newer kernel than the one it was built to expect, but 176 should always work on a newer kernel than the one it was built to expect, but
142 not necessarily an older one.)</p> 177 not necessarily an older one.)</p>
143 178
144 <h2>Ways to make it all work</h2> 179 <h2>Ways to make it all work</h2>
145 180
181 <h3>Cross compiling vs native compiling under emulation</h3>
182
146 <p>Cross compiling is a pain. There are a lot of ways to get it to sort of 183 <p>Cross compiling is a pain. There are a lot of ways to get it to sort of
147 kinda work for certain versions of certain packages built on certain versions 184 kinda work for certain versions of certain packages built on certain versions
148 of certain distributions. But making it reliable or generally applicable is 185 of certain distributions. But making it reliable or generally applicable is
149 hard to do.</p> 186 hard to do.</p>
150 187
154 and why you might want to do it. Keep in mind that I wrote that for a company 191 and why you might want to do it. Keep in mind that I wrote that for a company
155 that specializes in cross-compiling. Personally, I consider cross-compiling 192 that specializes in cross-compiling. Personally, I consider cross-compiling
156 a necessary evil to be minimized, and that's how Firmware Linux is designed. 193 a necessary evil to be minimized, and that's how Firmware Linux is designed.
157 We cross-compile just enough stuff to get a working native compiler for the 194 We cross-compile just enough stuff to get a working native compiler for the
158 new platform, which we then run under emulation.</p> 195 new platform, which we then run under emulation.</p>
196
197 <h3>Which emulator?</h3>
159 198
160 <p>The emulator Firmware Linux 0.8x used was User Mode Linux (here's a 199 <p>The emulator Firmware Linux 0.8x used was User Mode Linux (here's a
161 <a href=http://www.landley.net/code/UML.html>UML mini-howto</a> I wrote 200 <a href=http://www.landley.net/code/UML.html>UML mini-howto</a> I wrote
162 while getting this to work). Since we already need the linux-kernel source 201 while getting this to work). Since we already need the linux-kernel source
163 tarball anyway, building User Mode Linux from it was convenient and minimized 202 tarball anyway, building User Mode Linux from it was convenient and minimized
184 <p>The main difference with QEMU is a sharper dividing line between the host 223 <p>The main difference with QEMU is a sharper dividing line between the host
185 system and the emulated target. Under UML we could switch to the emulated 224 system and the emulated target. Under UML we could switch to the emulated
186 system early and still run host binaries (via the hostfs mount). This meant 225 system early and still run host binaries (via the hostfs mount). This meant
187 we could be much more relaxed about cross compiling, because we had one 226 we could be much more relaxed about cross compiling, because we had one
188 environment that ran both types of binaries. But this doesn't work if we're 227 environment that ran both types of binaries. But this doesn't work if we're
189 building an ARM or PPC system on an x86 host.</p> 228 building an ARM, PPC, or x86-64 system on an x86 host.</p>
190 229
191 <p>Instead, we sequence more carefully. We cross-compile a minimal 230 <p>Instead, we need to sequence more carefully. We build a cross-compiler,
192 intermediate system from the seven packages listed earlier, and build a kernel 231 use that to cross-compile a minimal intermediate system from the seven packages
193 and QEMU. We run the kernel under QEMU with the new intermediate system, and 232 listed earlier, and build a kernel and QEMU. Then we run the kernel under QEMU
194 have it build the rest.</p> 233 with the new intermediate system, and have it build the rest natively.</p>
195 234
196 <h2>Alternatives to emulation</h2> 235 <p>It's possible to use other emulators instead of QEMU, and I have a todo
236 item to look at armulator. (I looked at another nommu system simulator at
237 Ottawa Linux Symposium, but after resolving the third unnecessary environmental
238 dependency and still not being able to get it to finish compiling yet, I
239 gave up. Armulator may be a patch against an obsolete version of gdb, but I
240 could at least get it to build.)</p>
241
242 <h3>Alternatives to emulation</h3>
197 243
198 <p>The main downsides of emulation are that is it's slow, can use a lot of 244 <p>The main downsides of emulation are that is it's slow, can use a lot of
199 memory, and can be tricky to debug if something goes wrong in the emulated 245 memory, and can be tricky to debug if something goes wrong in the emulated
200 environment. Cross compiling is sufficiently harder than native compiling that 246 environment. Cross compiling is sufficiently harder than native compiling that
201 I consider it a good trade-off, but there are alternatives.</p> 247 I consider it a good trade-off, but there are alternatives.</p>
237 283
238 <p>(Note: QEMU can export a host directory to the target through the emulated 284 <p>(Note: QEMU can export a host directory to the target through the emulated
239 network card as an smb filesystem, but you don't want to run your root 285 network card as an smb filesystem, but you don't want to run your root
240 filesystem on smb.)</p> 286 filesystem on smb.)</p>
241 287
242 <h2>Filesystem:</h2> 288 <h2>Filesystem Layout</h2>
243 289
244 <pre> 290 <p>Firmware Linux's directory hierarchy is a bit idiosyncratic: some redundant
245 Our directory hierarchy is a bit idiosyncratic: some redundant directories have 291 directories have been merged, with symlinks from the standard positions
246 been merged, with symlinks from the standard positions pointing to their new 292 pointing to their new positions. On the bright side, this makes it easy to
247 positions. 293 make the root partition read-only.</p>
248 294
249 The set "bin->usr/bin, sbin->usr/sbin, lib->usr/lib" all serve to consolidate 295 <h3>Simplifying the $PATH.</h3>
296
297 <p>The set "bin->usr/bin, sbin->usr/sbin, lib->usr/lib" all serve to consolidate
250 all the executables under /usr. This has a bunch of nice effects: making a 298 all the executables under /usr. This has a bunch of nice effects: making a
251 a read-only run from CD filesystem easier to do, allowing du /usr to show 299 a read-only run-from-CD filesystem easier to do, allowing du /usr to show
252 the whole system size, allowing everything outside of there to be mounted 300 the whole system size, allowing everything outside of there to be mounted
253 noexec, and of course having just one place to look for everything. (Normal 301 noexec, and of course having just one place to look for everything. (Normal
254 executables are in /usr/bin. Root only executables are in /usr/sbin. 302 executables are in /usr/bin. Root only executables are in /usr/sbin.
255 Libraries are in /usr/lib.) 303 Libraries are in /usr/lib.)</p>
256 304
257 For those of you wondering why /bin and /usr/sbin were split in the first 305 <p>For those of you wondering why /bin and /usr/sbin were split in the first
258 place, the answer is it's because Ken Thompson and Dennis Ritchie ran out 306 place, the answer is it's because Ken Thompson and Dennis Ritchie ran out
259 of space on the original 2.5 megabyte RK-05 disk pack their root partition 307 of space on the original 2.5 megabyte RK-05 disk pack their root partition
260 lived on in 1971, and leaked the OS into their second RK-05 disk pack where 308 lived on in 1971, and leaked the OS into their second RK-05 disk pack where
261 the user home directories lived. (/usr was what /home is today.) 309 the user home directories lived. When they got more disk space, they created
262 310 a new direct (/home) and moved all the user home directories there.</p>
263 The real reason we kept it is tradition. The execuse is that the root 311
312 <p>The real reason we kept it is tradition. The execuse is that the root
264 partition contains early boot stuff and /usr may get mounted later, but these 313 partition contains early boot stuff and /usr may get mounted later, but these
265 days we use initial ramdisks (initrd and initramfs) to handle that sort of 314 days we use initial ramdisks (initrd and initramfs) to handle that sort of
266 thing. The version skew issues of actually trying to mix and match different 315 thing. The version skew issues of actually trying to mix and match different
267 versions of /lib/libc.so.* living on a local hard drive with a /usr/bin/* 316 versions of /lib/libc.so.* living on a local hard drive with a /usr/bin/*
268 from the network mount are not pretty. 317 from the network mount are not pretty.</p>
269 318
270 I.E. The seperation is just a historical relic, and I've consolidated it in 319 <p>I.E. The seperation is just a historical relic, and I've consolidated it in
271 the name of simplicity. 320 the name of simplicity.</p>
272 321
273 The one bit where this can cause a problem is merging /lib with /usr/lib, 322 <p>The one bit where this can cause a problem is merging /lib with /usr/lib,
274 which means that the same library can show up in the search path twice, and 323 which means that the same library can show up in the search path twice, and
275 when that happens binutils gets confused and bloats the resulting executables. 324 when that happens binutils gets confused and bloats the resulting executables.
276 (They become as big as statically linked, but still refuse to run without 325 (They become as big as statically linked, but still refuse to run without
277 opening the shared libraries.) This is really a bug in either binutils or 326 opening the shared libraries.) This is really a bug in either binutils or
278 collect2, and has probably been fixed since I first onticed it. In any case, 327 collect2, and has probably been fixed since I first noticed it. In any case,
279 the proper fix is to take /lib out of the binutils search path, which we do. 328 the proper fix is to take /lib out of the binutils search path, which we do.
280 The symlink is left there in case somebody's using dlopen, and for "standards 329 The symlink is left there in case somebody's using dlopen, and for "standards
281 compliance". 330 compliance".</p>
282 331
283 Similarly, all the editable stuff has been moved under "var", including 332 <p>On a related note, there's no reason for "/opt". After the original Unix
284 tmp->var/tmp, and etc->var/etc. (Whether /etc really needs to be editable is 333 leaked into /usr, Unix shipped out into the world in semi-standardized forms
285 an issue to be revisited later...) Remember to put root's home directory 334 (Version 7, System III, the Berkeley Software Distribution...) and sites that
286 somewhere writeable (I.E. /root should move to either /var/root or 335 installed these wanted places to add their own packages to the system without
287 /home/root), and life is good. 336 mixing their additions in with the base system. So they created "/usr/local"
288 337 and created a third instance of bin/sbin/lib and so on under there. Then
289 Other detail: /tmp is much less useful these days than it used to be. Long 338 Linux distributors wanted a place to install optional packages, and they had
290 ago in the days of little hard drive space and even less ram, people made 339 /bin, /usr/bin, and /usr/local/bin to choose from, but the problem with each
340 of those is that they were already in use and thus might be cluttered by who
341 knows what. So a new directory was created, /opt, for "optional" packages
342 like firefox or open office.</p>
343
344 <p>It's only a matter of time before somebody suggests /opt/local, and I'm
345 not humoring this. Executables for everybody go in /usr/bin, ones usable
346 only by root go in /usr/sbin. There's no /usr/local or /opt. /bin and
347 /sbin are symlinks to the corresponding /usr directories, but there's no
348 reason to put them in the $PATH.</p>
349
350 <h3>Consolidating writeable directories.</h3>
351
352 <p>All the editable stuff has been moved under "var", starting with symlinking
353 tmp->var/tmp. Although /tmp is much less useful these days than it used to
354 be, some things (like X) still love to stick things like named pipes in there.
355 Long ago in the days of little hard drive space and even less ram, people made
291 extensive use of temporary files and they threw them in /tmp because ~home 356 extensive use of temporary files and they threw them in /tmp because ~home
292 had an ironclad quota. These days, putting anything in /tmp with a predictable 357 had an ironclad quota. These days, putting anything in /tmp with a predictable
293 filename is a security issue (symlink attacks, you can be made to overwrite 358 filename is a security issue (symlink attacks, you can be made to overwrite
294 any arbitrary file you have access to). Most temporary files for things 359 any arbitrary file you have access to). Most temporary files for things
295 like the printer or email migrated to /var/spool, where there are persistent 360 like the printer or email migrated to /var/spool (where there are
296 subdirectories with known ownership and permissions. 361 persistent subdirectories with known ownership and permissions) or in the
297 362 user's home directory under something like "~/.kde".</p>
298 The result of all this is that a running system can have / be mounted read only 363
299 (with /usr living under that), /var can be ramfs/tmpfs with a tarball extracted 364 <p>The theoretical difference between /tmp and /var/tmp is that the contents
300 into it, /dev can be ramfs/tmpfs managed by udev (with /dev/pts as devpts under 365 of /var/tmp should definitely be deleted by the system init scripts on every
301 that: note that /dev/shm naturally inherits /dev's tmpfs), /proc can be procfs, 366 reboot, but the contents of /tmp may be preserved across reboots. Except
302 /sys can bs sysfs. Optionally, /home can be be an actual writeable filesystem 367 deleting everyting out of /tmp during a reboot is a good idea anyway, and any
303 on a hard drive or the network. 368 program that actually depends on the contents of /tmp being preserved across
304 </pre> 369 a reboot is obviously broken, so there's no reason not to symlink them
370 together.</p>
371
372 <p>(I case it hasn't become apparent yet, there's 30 years of accumulated cruft
373 in the standards, convering a lot of cases that don't apply outside of
374 supercomputing centers where 500 people share accounts on a mainframe that
375 has a dedicated support staff. They serve no purpose on a laptop, let alone
376 an embedded system.)</p>
377
378 <p>The corner case is /etc, which can be writeable (we symlink it to
379 var/etc) or a read-only part of the / partition. It's really a question of
380 whether you want to update configuration information and user accounts in a
381 running system, or whether that stuff should be fixed before deploying.
382 We're doing some cleanup, but leaving /etc writeable (as a symlink to
383 /var/etc). Firmware Linux symlinks /etc/mtab->/proc/mounts, which
384 is required by modern stuff like shared subtrees. If you want a read-only
385 /etc, use "find /etc -type f | xargs ls -lt" to see what gets updated on the
386 live system. Some specific cases are that /etc/adjtime was moved to /var
387 by LSB and /etc/resolv.conf should be a symlink somewhere writeable.</p>
388
389 <h3>The resulting mount points</h3>
390
391 <p>The result of all this is that a running system can have / be mounted read
392 only (with /usr living under that), /var can be ramfs or tmpfs with a tarball
393 extracted to initialize it on boot, /dev can be ramfs/tmpfs managed by udev or
394 mdev (with /dev/pts as devpts under that: note that /dev/shm naturally inherits
395 /dev's tmpfs and some things like User Mode Linux get upset if /dev/shm is
396 mounted noexec), /proc can be procfs, /sys can bs sysfs. Optionally, /home
397 can be be an actual writeable filesystem on a hard drive or the network.</p>
398
399 <p>Remember to
400 put root's home directory somewhere writeable (I.E. /root should move to
401 either /var/root or /home/root, change the passwd entry to do this), and life
402 is good.</p>
403
404 </p>