view www/build-stages.html @ 1826:1ba206595781 draft

Update build-stages.html documentation.
author Rob Landley <rob@landley.net>
date Wed, 09 Dec 2015 15:28:51 -0600
parents 3a66b5554d1e
children
line wrap: on
line source

<html>
<title>Aboriginal Linux - Build Stages</title>
<body>
<!--#include file="header.html" -->

<p>The Aboriginal Linux build scripts are the source code for the Aboriginal
Linux project. If you would like to build your own cross compiler or target
system image from source, use these build scripts. They're written in bash
and should be fairly easy to read.</p>

<h2>Quick start</h2>

<p>Run <b>build.sh</b> with no arguments to see a list of targets. Select
a target, and run <b>build.sh $TARGET</b> with the target name in place of
$TARGET. When it finishes, run <b>more/dev-environment.sh $TARGET</b> to
boot the resulting system image under QEMU, configured for use as a
development environment. Type <b>exit</b> to shut down the emulator.</p>

<h2>Overview</h2>

<p><b>build.sh</b> runs the following stages, in order:</p>

<ul>
<li><p><b>download.sh</b> - Download source packages used by the rest of the build.</p></li>
<li><p><b>host-tools.sh</b> - Build prerequisites host needs to run remaining stages.</p></li>
<li><p><b>simple-cross-compiler.sh</b> - Build cross compiler for selected target architecture.</p></li>
<li><p><b>[</b>cross-compiler.sh<b>]</b> - optionally produce a more portable
cross compiler which can be <a href=bin/>packaged as a tarball</a> for use
elsewhere. This stage is skipped unless CROSS_COMPILER_HOST is set (usually
to i686 or x86_64).</p></li>
<li><p><b>root-filesystem.sh</b> - Build simple initramfs filesystem, just enough to boot to a shell prompt.</p></li>
<li><p><b>native-compiler.sh</b> - Build native compiler to install/run on target.</p></li>
<li><p><b>system-image.sh</b> - Build bootable linux kernel and package together filesystem image and kernel with scripts to launch them under an emulator.</p></li>
</ul>

<p>The top level wrapper script <b>build.sh</b> runs the above stages in order,
but each stage script can also be run individually. Each of the above
build scripts (except download.sh and host-tools.sh) take a single argument:
the name of the target architecture to build code for. Run build.sh with no
arguments to see a list of available targets.</p>

<p>Each build stage (except download.sh and host-tools.sh) produces its output
in the "build" directory under a subdirectory named after the script plus the
target. It also produces a tarball of that directory if the build stage
completed successfully. (The download.sh script populates the "packages"
directory instead, and host-tools.sh produces its output the directory
"build/host" with no tarball version since those programs are intended to
run locally.)</p>

<p>All downloaded files wind up in the "packages" directory. Output from
compiles is generated in the "build" directory. These are the only two
directories the build writes to, and both directories may be deleted (to be
recreated by the build scripts). The equivalent of "distclean" is
<b>rm -rf build packages</b> from the top level directory.</p>

<p>None of these scripts need to be run as root -- an explicit design goal of
Aboriginal Linux is that root access on the host is never required.</p>

<h2>Files</h2>

<p>The files in the top level directory of the Aboriginal Linux source
are:</p>

<ul>
<li><b>configure</b>
<blockquote>
<p>This is a configuration file rather than a build script. It contains
several variables that can be set to control the build's behavior,
with descriptions of each. Each variables may be set in this file, or
exported as environment variables.</p>

<p>A useful shell syntax to export environment variables for just a single
command, without persistently altering the environment, is to list the
assignments before the command on the same line. For example:</p>

<blockquote><pre><b>CROSS_COMPILER_HOST=i686 SYSIMAGE_TYPE=ext2 ./build.sh armv5l</b></pre></blockquote>

<p>Configuration variables can also be persistently set on a per-target basis
in the appropriate sources/targets file.</p>
</li>

<li><b>build.sh $TARGET</b>
<blockquote>
<p>Top level wrapper script which builds a system image for a target,
by calling most of the other scripts listed here in the appropriate
order. When run without arguments, build.sh lists available architectures.
Run with one argument, it builds that target. Run with two arguments, the
second is the name of a build stage to restart the build at.</p>

<p>This script is just a wrapper, it contains no actual build logic (except
checking some of the configuration variables).</p>
</li>

<li><b>download.sh</b>

<blockquote>
<p>Uses wget to download the source code required by the later build stages,
saving it in the "packages" directory. It compares the sha1 checksum
of any existing tarballs to an expected value, only downloading new
source tarballs when it needs to.</p>

<p>If a package's primary site is down, it checks a series of fallback mirrors.
The environment variable PREFERRED_MIRROR can insert a new mirror at
the start of the list, which is checked before even the official website.</p>

<p>This script is not target-specific, and only needs to be called once
even when building multiple architectures.</p>
</blockquote>
</li>

<li><b>host-tools.sh</b>

<blockquote>
<p>Sanitizes the host environment by building known versions of needed tools
from source code, then restricting the $PATH to just those tools. This is
technically an optional step which can be skipped, but without it the build
process is extremely brittle (sensitive to changes in the host
distro/environment).</p>

<p>This "airlock" step serves a similar purpose to the temporary system
(/tools) built by <a href=http://linuxfromscratch.org/lfs/view/stable/chapter05/introduction.html>Linux From Scratch's chapter 5</a>, isolating the new system
from variations in the host. It also acts as an early check that the resulting
system images offer a sufficient development environment to rebuild themselves
from source, because the host tool versions used to build them in the first
place are the same ones the scripts install into the target root filesystem.</p>

<p>This script populates the "build/host" directory, which is automatically
used by later stages if it exists. It is not target specific, and only
needs to be run once when building multiple architectures.</p>
</blockquote>
</li>

<li><b>simple-cross-compiler.sh $TARGET</b>

<blockquote>
<p>Creates a cross compiler for the selected target architecture, built from
gcc, binutils, musl, and the Linux kernel headers. This compiler runs
on the host and produces programs that run on the target.</p>

<p>This compiler is sufficient to build a system image for the target, but
isn't as powerful as the compilers created by cross-compiler.sh or
native-compiler.sh. (It doesn't include thread support, uClibc++, or the
shared version of libgcc. The binaries aren't statically linked, and they
may leak host path details and thus not find their data files if moved to
another directory location.)</p>
</blockquote>
</li>

<li><b>cross-compiler.sh $TARGET</b>

<blockquote>
<p>This optional step creates a more full-featured cross compiler, with
thread support, uClibc++, and the shared version of libgcc. This is
not required to build a system image, but the prebuilt binary compilers
shipped in the downloads/binaries directory are built this way.</p>

<p>The build.sh wrapper script only calls this stage if the config variable
CROSS_COMPILER_HOST is set, indicating which host architecture to build for.
(For PC hardware, i686 is a good choice, since most 64 bit PCs can run static
32 bit code. If you run "./cross-comiler.sh $TARGET" manually without setting
CROSS_COMPILER_HOST, it defaults to i686.)</p>

<p>This compiler is statically linked against musl-libc, for maximum
portability. (You can set BUILD_STATIC=none to dynamically link instead,
but then have to install musl's shared libraries on the host.)</p>
</blockquote>
</li>

<li><b>root-filesystem.sh $TARGET</b>

<blockquote>
<p>Creates a root filesystem (with uCLibc, BusyBox, and an init script)
which contains just enough infrastructure to boot up to a shell prompt.
By default this is packaged as an initramfs, see SYSIMAGE_TYPE in config
to see other available filesystem types.</p>

<p>This creates empty directories, copies the skeleton files from
sources/root-filesystem, builds toybox, and finally adds the contents
of the directory $MY_OVERLAY points to (if any). On appropriate hardware
(or with an appropriate emulator), you should be able to chroot into this.</p>
</blockquote>
</li>

<li><b>native-compiler.sh $TARGET</b>

<blockquote>
<p>This step creates a compiler for the selected target, using one
or more of the existing simple cross compilers. The compiler it produces
runs on the target and produces programs that also run on the target.</p>

<p>By default this compiler is statically linked so you can add it to an
existing target root filesystem. Use BUILD_STATIC=none to disable this.</p>

<p>This compiler includes binutils, gcc, musl, make, bash, and distcc. Because
it's a native compiler, the executable names do not have prefixes the
way the cross compilers do. (I.E. just "ld" instead of "$TARGET-ld".)</p>
</blockquote>
</li>

<li><b>system-image.sh $TARGET</b>
<blockquote>
<p>This does three things:</p>
<ul>
<li><p>Packages up the root-filesystem (the default is cpio.gz for initramfs,
see SYSIMAGE_TYPE in config for alternatives) and the native-compiler
(as a squashfs).</p></li>
<li><p>Adds emulator launch scripts (usually for QEMU), see the
<a href=<about.html#sysimage_use>About page</a> for details.</p></li>
<li><p>Builds a linux kernel, generally configured for use with QEMU).</p></li>

<p>The <b>mini.config</b> file is the kernel configuration in
<a href=http://landley.net/aboriginal/FAQ.html#miniconfig>miniconfig</a>
format (I.E. start from 'allnoconfig' and list the symbols you'd need to
switch on in menuconfig, allowing that to resolve dependencies as it goes).
It's created by combining the sources/baseconfig-linux settings
(which are the same for each $TARGET) with the target-specific LINUX_CONFIG
entries from sources/targets/$TARGET. This gives you a starting point
to build your own kernel and package up root-filoesystem in your own way
if you want to bypass the system-image.sh stage.</p>
</blockquote>
</li>
</ul>

<p>The <b>sources/</b> directory contains infrastructure, defining variables
and shell functions used by the rest of the build. Most prominently, the
the shell functions "<b>setupfor</b>" prepares a temporary copy of
the source (extract and patch the relevant source tarballs and cd into the
directory), and the shell function "<b>cleanup</b>" deletes that temporary copy
when finished. (The actual implementation has
<a href=FAQ.html#debug_source>some optimizations</a> you can usually ignore.)
The function "<b>build_section</b>" does both and calls a build script from
sources/sections in between (see sources/sections/README for details).</p>

<p>The <b>sources/targets/</b> directory contains all target-specific
information. Each target has a single file defining all target-specific
information, and adding a target just means adding a file to this directory.</p>

<p>The <b>more/</b> directory contains all the additional scripts the user may
want to run directly, but which aren't build stages.</p>

<p>The <b>more/native-build-from-build.sh</b> script (which calls the
native-build.sh script in a system-image) uses <a href=control-images>build
control images</a>, externally supplied filesystem images (usually squashfs)
which the system image's init script automatically mount on /mnt. Then if the
file /mnt/init exists (I.E. an executable "init" script at the top of the
build control image), the system image init script will run that file instead
of dropping to a shell prompt. This allows arbitrary automated behavior
out of the newly booted image, operating on supplied data.</p>

<!--#include file="footer.html" -->