Mercurial > hg > aboriginal

annotate more/buildall.sh @ 1737:1acbe7e78d9d draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Switch from bz2 to gz tarball output. bz2 is slowly being replaced by lz variants, but gzip isn't going anywhere. (It retains a niche as a streaming protocol, it's the 80/20 solution optimizing for speed, low memory requirements, and simple compressor implementation.)
author Rob Landley <rob@landley.net>
date Fri, 20 Feb 2015 13:12:45 -0600
parents f0315ac24c40
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
610
a47844b73616 New simple buildall.sh that's a wrapper around build.sh. (Chop out half of build-all-targets, need to make a separate script to build static toolchains.)
Rob Landley <rob@landley.net>
parents:
diff changeset
1 #!/bin/bash
a47844b73616 New simple buildall.sh that's a wrapper around build.sh. (Chop out half of build-all-targets, need to make a separate script to build static toolchains.)
Rob Landley <rob@landley.net>
parents:
diff changeset
2
1182
c5ebf1e353c7 Apparently this never quite got updated for commit 1116.
Rob Landley <rob@landley.net>
parents: 1157
diff changeset
3 # Build every target architecture, saving log files to build/log.
610
a47844b73616 New simple buildall.sh that's a wrapper around build.sh. (Chop out half of build-all-targets, need to make a separate script to build static toolchains.)
Rob Landley <rob@landley.net>
parents:
diff changeset
4 # If $FORK is set, build them in parallel.
a47844b73616 New simple buildall.sh that's a wrapper around build.sh. (Chop out half of build-all-targets, need to make a separate script to build static toolchains.)
Rob Landley <rob@landley.net>
parents:
diff changeset
5
1273
dadbb8873a36 functions.sh no longer includes utility-functions.sh, so adjust includes.
Rob Landley <rob@landley.net>
parents: 1192
diff changeset
6 . sources/utility_functions.sh || exit 1
610
a47844b73616 New simple buildall.sh that's a wrapper around build.sh. (Chop out half of build-all-targets, need to make a separate script to build static toolchains.)
Rob Landley <rob@landley.net>
parents:
diff changeset
7
1350
7eabfa815c90 Split up system-image.sh into root-image.sh, linux-kernel.sh, and system-image.sh. Rename CROSS_HOST_ARCH to CROSS_COMPILER_HOST.
Rob Landley <rob@landley.net>
parents: 1344
diff changeset
8 [ -z "$CROSS_COMPILER_HOST" ] && export CROSS_COMPILER_HOST=i686
778
e373d2b6d0b8 Add killtree and make stuff use it. Update smoketest-all.sh to use doforklog, allowing FORK=1 to run stuff in parallel.
Rob Landley <rob@landley.net>
parents: 776
diff changeset
9
e373d2b6d0b8 Add killtree and make stuff use it. Update smoketest-all.sh to use doforklog, allowing FORK=1 to run stuff in parallel.
Rob Landley <rob@landley.net>
parents: 776
diff changeset
10 trap "killtree $$" EXIT
e373d2b6d0b8 Add killtree and make stuff use it. Update smoketest-all.sh to use doforklog, allowing FORK=1 to run stuff in parallel.
Rob Landley <rob@landley.net>
parents: 776
diff changeset
11
787
23bff7aa79eb Teach buildall.sh to use new build.sh infrastructure, and replace doforklog with orthogonal maybe_fork and maybe_quiet functions.
Rob Landley <rob@landley.net>
parents: 778
diff changeset
12 # Build the host architecture. This has to be built first so the other
23bff7aa79eb Teach buildall.sh to use new build.sh infrastructure, and replace doforklog with orthogonal maybe_fork and maybe_quiet functions.
Rob Landley <rob@landley.net>
parents: 778
diff changeset
13 # architectures can canadian cross static compilers to run on the host using
23bff7aa79eb Teach buildall.sh to use new build.sh infrastructure, and replace doforklog with orthogonal maybe_fork and maybe_quiet functions.
Rob Landley <rob@landley.net>
parents: 778
diff changeset
14 # this toolchain to link against a host version of uClibc.
23bff7aa79eb Teach buildall.sh to use new build.sh infrastructure, and replace doforklog with orthogonal maybe_fork and maybe_quiet functions.
Rob Landley <rob@landley.net>
parents: 778
diff changeset
15
23bff7aa79eb Teach buildall.sh to use new build.sh infrastructure, and replace doforklog with orthogonal maybe_fork and maybe_quiet functions.
Rob Landley <rob@landley.net>
parents: 778
diff changeset
16 # This also performs the download.sh and host-tools.sh steps, which don't
23bff7aa79eb Teach buildall.sh to use new build.sh infrastructure, and replace doforklog with orthogonal maybe_fork and maybe_quiet functions.
Rob Landley <rob@landley.net>
parents: 778
diff changeset
17 # parallelize well if many build.sh instances try to call them at once.
23bff7aa79eb Teach buildall.sh to use new build.sh infrastructure, and replace doforklog with orthogonal maybe_fork and maybe_quiet functions.
Rob Landley <rob@landley.net>
parents: 778
diff changeset
18
23bff7aa79eb Teach buildall.sh to use new build.sh infrastructure, and replace doforklog with orthogonal maybe_fork and maybe_quiet functions.
Rob Landley <rob@landley.net>
parents: 778
diff changeset
19 # If this fails, don't bother trying to build the other targets.
610
a47844b73616 New simple buildall.sh that's a wrapper around build.sh. (Chop out half of build-all-targets, need to make a separate script to build static toolchains.)
Rob Landley <rob@landley.net>
parents:
diff changeset
20
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
21 [ -z "$NO_CLEAN" ] && blank_tempdir build
1344
6a04a89a0d19 Add BUILD_NATIVE_ONLY and NO_CLEAN controls to buildall.sh. Also break out the hdb.img creation and do that single threaded for all targets, then reuse the scratch drives. (Avoids triggering timeout.sh due to disk thrashing on large FORK=1 builds.)
Rob Landley <rob@landley.net>
parents: 1273
diff changeset
22
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
23 mkdir -p build/logs &&
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
24 (EXTRACT_ALL=1 ./download.sh 2>&1 &&
1559
f0315ac24c40 Call record-commands.sh from buildall.sh.
Rob Landley <rob@landley.net>
parents: 1494
diff changeset
25 more/record-commands.sh &&
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
26 ./host-tools.sh 2>&1 &&
1559
f0315ac24c40 Call record-commands.sh from buildall.sh.
Rob Landley <rob@landley.net>
parents: 1494
diff changeset
27 more/record-commands.sh &&
f0315ac24c40 Call record-commands.sh from buildall.sh.
Rob Landley <rob@landley.net>
parents: 1494
diff changeset
28 ./simple-cross-compiler.sh "$CROSS_COMPILER_HOST" 2>&1 ||
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
29 dienow) | tee build/logs/build-host-cc.txt | maybe_quiet
1192
8d5a672cd293 build-control-images.sh got broken out a while ago, remove redundant code from buildall.
Rob Landley <rob@landley.net>
parents: 1182
diff changeset
30
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
31 cp packages/MANIFEST build || dienow
1114
be94bf38648b Break up buildall.sh so it's easier to run individual portions of it.
Rob Landley <rob@landley.net>
parents: 1103
diff changeset
32
1441
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
33 # Adjust $CPUS so as not to overload the machine, max 2 build processes
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
34 # per gigabyte of RAM
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
35
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
36 if [ ! -z "$FORK" ] && [ -z "$CPUS" ]
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
37 then
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
38 MEGS=$(($(awk '/MemTotal:/{print $2}' /proc/meminfo)/1024))
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
39 TARGET_COUNT=$(find sources/targets -maxdepth 1 -type f | wc -l)
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
40 export CPUS=$(($MEGS/($TARGET_COUNT*512)))
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
41 [ "$CPUS" -lt 1 ] && CPUS=1
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
42 fi
4923bba45afe Try to auto-balance FORK=1 and CPUS so as not to overload build server.
Rob Landley <rob@landley.net>
parents: 1417
diff changeset
43
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
44 # Build all non-hw targets, possibly in parallel
1192
8d5a672cd293 build-control-images.sh got broken out a while ago, remove redundant code from buildall.
Rob Landley <rob@landley.net>
parents: 1182
diff changeset
45
1157
300e6d919d86 Move "sources/more/" to just "more/", add a README explaining why those scripts are there, and adjust calls to them.
Rob Landley <rob@landley.net>
parents: 1124
diff changeset
46 more/for-each-target.sh \
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
47 './build.sh $TARGET 2>&1 | tee build/logs/build-${TARGET}.txt'
1192
8d5a672cd293 build-control-images.sh got broken out a while ago, remove redundant code from buildall.
Rob Landley <rob@landley.net>
parents: 1182
diff changeset
48
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
49 # Run smoketest.sh for each non-hw target.
1192
8d5a672cd293 build-control-images.sh got broken out a while ago, remove redundant code from buildall.
Rob Landley <rob@landley.net>
parents: 1182
diff changeset
50
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
51 more/for-each-target.sh \
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
52 'more/smoketest.sh $TARGET 2>&1 | tee build/logs/smoketest-$TARGET.txt'
974
51a8a0076c9d Oops. Wait for native build to finish before exiting.
Rob Landley <rob@landley.net>
parents: 972
diff changeset
53
1417
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
54 # If we have a control image, build natively
aabc07905de3 Now that the control-images have their own repository, remove them from this one.
Rob Landley <rob@landley.net>
parents: 1409
diff changeset
55
1494
0921d99c1b71 Allow buildall to do multiple native control-images, and put control image name in native-*.txt log.
Rob Landley <rob@landley.net>
parents: 1441
diff changeset
56 for i in "$@"
0921d99c1b71 Allow buildall to do multiple native control-images, and put control image name in native-*.txt log.
Rob Landley <rob@landley.net>
parents: 1441
diff changeset
57 do
0921d99c1b71 Allow buildall to do multiple native control-images, and put control image name in native-*.txt log.
Rob Landley <rob@landley.net>
parents: 1441
diff changeset
58 more/buildall-native.sh "$1"
0921d99c1b71 Allow buildall to do multiple native control-images, and put control image name in native-*.txt log.
Rob Landley <rob@landley.net>
parents: 1441
diff changeset
59 done
852
6ecee01bffa6 Update paths and comments in cronjob.sh, rename "none" to "stable", minor cleanup to logging.
Rob Landley <rob@landley.net>
parents: 850
diff changeset
60
1409
8d0567d46cd7 Create smoketest-report.sh producing CSV with more info.
Rob Landley <rob@landley.net>
parents: 1350
diff changeset
61 more/smoketest-report.sh | tee build/logs/status.txt