CircleCI hardocodes a maximum build time of 5 hours

[tlaurion]

Houston, we have a problem:
https://app.circleci.com/pipelines/github/tlaurion/heads/694/workflows/9d351748-166a-4fa6-b0e4-7a08510cedcc/jobs/765

Build fails because:

Build timed out after 5h0m0s

context deadline exceeded

Documented here:
https://discuss.circleci.com/t/job-times-out-after-5-hours/32220/4

Possible solutions:

• Reduce the number of boards supported under CI
• Reduce the number of coreboot versions built (move all boards to latest coreboot version so that coreboot 4.11 and latest version are built (coreboot musl-cross for each coreboot version is taking a lot of time to build) )
• Build on top to a docker image including the musl-cross-make tollchain we use and revert to having coreboot build on top (bad idea)
• Test ccache ?
• Check impacts of bumping CPUS=4 to something higher
• Test if memory issues are still happening when removing CPUS= statement altogether.

[tlaurion]

Some stats obtained by grepping in the first board being built by CircleCI. Stats obtained by grep "2021-01-31" on the log downloaded locally.

• The first make BOARD= build (2h53) needs to build musl-cross-make if not in cache.

2021-01-31 18:52:52+00:00 Wrong gawk detected: 
--2021-01-31 18:52:53--  http://gnu.mirror.constant.com/gawk/gawk-4.2.1.tar.xz
2021-01-31 18:52:53 (33.1 MB/s) - '/root/project/packages/gawk-4.2.1.tar.xz.tmp' saved [2985412/2985412]
2021-01-31 18:54:26+00:00 WGET https://www.coreboot.org/releases/coreboot-blobs-4.11.tar.xz
--2021-01-31 18:54:26--  https://www.coreboot.org/releases/coreboot-blobs-4.11.tar.xz
2021-01-31 18:57:13 (152 KB/s) - '/root/project/packages/coreboot-blobs-4.11.tar.xz.tmp' saved [25766200/25766200]
2021-01-31 18:57:16+00:00 CONFIG coreboot-blobs
2021-01-31 18:57:16+00:00 MAKE coreboot-blobs
2021-01-31 18:57:16+00:00 DONE coreboot-blobs
2021-01-31 18:57:16+00:00 WGET https://www.coreboot.org/releases/coreboot-4.11.tar.xz
--2021-01-31 18:57:16--  https://www.coreboot.org/releases/coreboot-4.11.tar.xz
2021-01-31 19:02:16 (133 KB/s) - '/root/project/packages/coreboot-4.11.tar.xz.tmp' saved [40730352/40730352]
2021-01-31 19:24:14+00:00 CONFIG coreboot

Here, we get by the difference in the two last timestamps that building coreboot buildstack took ~22 minutes.

2021-01-31 19:24:15+00:00 WGET https://github.com/richfelker/musl-cross-make/archive/38e52db8358c043ae82b346a2e6e66bc86a53bc1.tar.gz
--2021-01-31 19:24:15--  https://github.com/richfelker/musl-cross-make/archive/38e52db8358c043ae82b346a2e6e66bc86a53bc1.tar.gz
--2021-01-31 19:24:16--  https://codeload.github.com/richfelker/musl-cross-make/tar.gz/38e52db8358c043ae82b346a2e6e66bc86a53bc1
2021-01-31 19:24:16 (7.76 MB/s) - '/root/project/packages/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1.tar.gz.tmp' saved [142143]
2021-01-31 19:24:16+00:00 CONFIG musl-cross
2021-01-31 19:24:16+00:00 WGET https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.14.62.tar.xz
--2021-01-31 19:24:16--  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.14.62.tar.xz
2021-01-31 19:24:17 (242 MB/s) - '/root/project/packages/linux-4.14.62.tar.xz.tmp' saved [100957512/100957512]
2021-01-31 19:25:00+00:00 MAKE musl-cross
--2021-01-31 19:25:00--  https://ftp.gnu.org/pub/gnu/gcc/gcc-8.3.0/gcc-8.3.0.tar.xz
2021-01-31 19:25:02 (49.3 MB/s) - 'gcc-8.3.0.tar.xz' saved [63694700/63694700]
--2021-01-31 19:25:02--  http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=3d5db9ebe860
2021-01-31 19:25:02 (2.53 MB/s) - 'config.sub' saved [36159]
--2021-01-31 19:25:25--  https://ftp.gnu.org/pub/gnu/binutils/binutils-2.32.tar.xz
2021-01-31 19:25:25 (39.0 MB/s) - 'binutils-2.32.tar.xz' saved [20774880/20774880]
--2021-01-31 19:25:33--  https://www.musl-libc.org/releases/musl-1.1.24.tar.gz
2021-01-31 19:25:33 (10.7 MB/s) - 'musl-1.1.24.tar.gz' saved [1024988/1024988]
--2021-01-31 19:25:33--  https://ftp.gnu.org/pub/gnu/gmp/gmp-6.1.2.tar.bz2
2021-01-31 19:25:34 (18.6 MB/s) - 'gmp-6.1.2.tar.bz2' saved [2386766/2386766]
--2021-01-31 19:25:36--  https://ftp.gnu.org/pub/gnu/mpc/mpc-1.1.0.tar.gz
2021-01-31 19:25:36 (6.51 MB/s) - 'mpc-1.1.0.tar.gz' saved [701263/701263]
--2021-01-31 19:25:36--  https://ftp.gnu.org/pub/gnu/mpfr/mpfr-4.0.2.tar.bz2
2021-01-31 19:25:36 (9.80 MB/s) - 'mpfr-4.0.2.tar.bz2' saved [1652074/1652074]
--2021-01-31 19:25:38--  http://ftp.barfooze.de/pub/sabotage/tarballs//linux-headers-4.19.88.tar.xz
2021-01-31 19:25:44 (164 KB/s) - 'linux-headers-4.19.88.tar.xz' saved [1052880/1052880]
2021-01-31 21:03:50+00:00 DONE musl-cross

The last two lines informs us that after having downloaded the packages, building musl-cross (musl-cross-make) took 1h38.

• After having built musl-cross-make, building a shared linux version across boards take ~30 minutes:

2021-01-31 21:03:55+00:00 MAKE linux
2021-01-31 21:31:20+00:00 DONE linux

• Then, all other modules are built, normally shared across boards.
• As of right now, we have variable versions for: coreboot, linux, and cryptsetup.
• For the above board, building the rom took 2h53, on which coreboot buildstack took 22 minutes, musl-cross-make took 1h38, where linux took around 30 minutes, while the rest of the modules took the 23 minutes left.

Next board.

• After that, the next following board, depending on the same coreboot and linux version takes 1m26.

[Tonux599]

I don't know a lot about CircleCI but this states:

Jobs have a maximum runtime of 5 hours. If your jobs are timing out, consider running some of them concurrently using workflows.

So could we not just split up tasks into jobs?

Edit: i.e. could have have each board be a job, in which case the 5hr limit would be the time it takes to build each board

[Tonux599]

Actually following on from above, it looks like we could really speed up building by splitting jobs and making them parallel. A workflow could go something like:

Initial prep -> Build a single board of coreboot 4.8.1, 4.11 and 4.13 in parallel -> build all the rest of the boards in parallel.

[Tonux599]

@tlaurion messy as hell PoC here

So looks like we can persist data between jobs with persist_to_workspace and attach_workspace but that's not too friendly with root (/) so we will have to re-run apt install for each job unless we setup a docker image with those dependencies installed.

Edit: slightly better here, still not in parallel though.

Edit: even better with parallel here

[tlaurion]

@Tonux599 : https://app.circleci.com/pipelines/github/Tonux599/heads/67/workflows/7d0131b5-f6e3-44d9-8b35-2154c57f5b79
Is awesome!!!!

My suggestion would be to split it up my coreboot version. Looking in your config

[tlaurion]

Suggestions:

• Add apt install call as a variable and call it instead of duplicating.

I think if that last build succeeds, we can do another rounds of tweaks after! Awesome @Tonux599! You are now the CircleCI master of heads :)

[Tonux599]

It does look promising, I'll open a PR with cleaner commits so we can go forward with its current state. One thing to note though, I'm not convinced the cache will be saved correctly as the job save_cache inherits from the last job built workspace (I think). So it may not contain coreboot 4.11 or 4.13 musl-cross.

I think a cleaner way of doing it might be:

Build a CB 4.8.1 board -> build a CB 4.11 board -> build a CB 4.13 board -> save cache -> build everything else in parallel.

Saying that though, persist_to_workspace and attach_workspace allow picking certain subdirectories so we may be able to still have save_cache last with defined directory's from its parents.

More testing and playing around with required!

[tlaurion]

Wrapping my head around this.

Trying to understand why the builds are taking so long still in between the dependencies.

If I get this dependency right, dependencies linked to parallelization should be smaller steps to save time between dpendencies of builds (like in this case, all other board builds are waiting for qemu-coreboot to finish before starting)

• make musl-cross
• make all other boards

Watching the process of the builds, I have intuition that a make statemnt is currently missing in coreboot to do what we want. We have coreboot, coreboot-blobs, coreboot.menuconfig, coreboot.saveconfig and their clean counterparts, where coreboot.crossgcc (building crossgcc-i386 and iasl) would speed up all other boards. We could then have the following minimal dependencies shared

1. make musl-cross CPUS=8 V=1
2. make BOARD=$(candidate of each coreboot versions) coreboot.crossgcc CPUS=8 V=1 (dependency on 1)
3. make BOARD= (all other boards candidates) (dependency on 2)

What do you think from your experiences of today @Tonux599 ?

The reasoning out of this is

• qemu-coreboot builds musl-cross, coreboot.crossgcc, linux, and modules, then builds its coreboot stages and warps everything in. (1h39 if without musl-cross cache nor coreboot nor linux cache)
• librem_mini builds its coreboot.crossgcc for aother version, its different linux version and modules, then finally coreboot its coreboot stages and warps everything in.
  • librem_mini_v2 does the same but depends on finished time of qemu-coreboot and librem_mini coreboot. That could be hypothetically parallelized

Catch my draft ? IS it a silly idea? The total build time of

[tlaurion]

It does look promising, I'll open a PR with cleaner commits so we can go forward with its current state. One thing to note though, I'm not convinced the cache will be saved correctly as the job save_cache inherits from the last job built workspace (I think). So it may not contain coreboot 4.11 or 4.13 musl-cross.

I think a cleaner way of doing it might be:

Build a CB 4.8.1 board -> build a CB 4.11 board -> build a CB 4.13 board -> save cache -> build everything else in parallel.

Saying that though, persist_to_workspace and attach_workspace allow picking certain subdirectories so we may be able to still have save_cache last with defined directory's from its parents.

More testing and playing around with required!

@Tonux599 yeah... I'll sleep on it, but I really think
1- musl-cross only for qemu-coreboot board
2- then parallelize coreboot.crossgcc only for diffferent coreboot versions
3- then parallelize boards on top of coreboot.crossgcc cache, built on top of musl-cross (in a parallelized way?)

From my analysis of build time #970 (comment)
We know that building modules is minimal as opposed to building musl-cross, coreboot crossgcc and linux.
While linux are varying a lot recently across boards, caching it is not really interesting as of now (while the real solutio nwould be to upgrade it for all board config, and migrating old boards to latest coreboot will also limit the crosggc building time for different boards).

We now add a caching time of 15 minutes per board building of parent + extraction time from the child. (Which could be reduced by selective caching like you said).

[Tonux599]

Firstly I need to amend my previous comment, by the looks of it all the workspace layers are applied so the cache should be saved correctly.

1. make musl-cross CPUS=8 V=1

2. make BOARD=$(candidate of each coreboot versions) coreboot.crossgcc CPUS=8 V=1 (dependency on 1)

3. make BOARD= (all other boards candidates) (dependency on 2)

This is what I would be aiming for. However the only problem with that is all the small "things" (gpg, busybox, cryptsetup) would be needlessly be being build side-by-side at point 2. I think ideally we need:

1. make musl-cross CPUS=8 V=1
2. make (all the small stuff, gpg, busybox, cryptsetup)
3. make BOARD=$(candidate of each coreboot versions) coreboot.crossgcc CPUS=8 V=1
4. make BOARD= (all other boards candidates)

There is definitely more optimisation to be done, I think its just a case of sifting though it and making adjustments where we can.

[tlaurion]

@Tonux599 Can you trigger a rebuild please? (it will create another build, just so we reuse your cache and see)

[tlaurion]

Nervermind the cache is not ready yet. 30 minutes damn. Normal. I could cancel the workflow. I guess i'll be able to trigger a rebuild.

[Tonux599]

Nervermind the cache is not ready yet. 30 minutes damn. Normal.

Yeah we probably don't actually need all those workspace layers being applied, just one from each variant board of coreboot.

[tlaurion]

@Tonux599 A quickfix here though would be to replace the qemu-coreboot by one of the boards that has it all already. Like the x230-hotp-maximized board.