ci: make dcou partitioned steps not-skewed and cached #4452
Conversation
ryoqun
force-pushed
the
even-dcou-builds-wip
branch
4 times, most recently
from
8c95636 to
c0978e9
Compare
| @@ -6,7 +6,7 @@ ARG \ | |||
| RUST_NIGHTLY_VERSION= \ | |||
| GOLANG_VERSION=1.21.3 \ | |||
| NODE_MAJOR=18 \ | |||
| SCCACHE_VERSION=v0.8.1 \ | |||
| SCCACHE_VERSION=v0.9.1 \ | |||
There was a problem hiding this comment.
this newer version starts to show cache hit rates by --show-stats, which is handy to some extent.
| mkdir -p "$HOME/.cache/sccache-for-docker" | ||
| CONTAINER_HOME="/" | ||
| ARGS+=( | ||
| --volume "$HOME/.cache/sccache-for-docker:$CONTAINER_HOME/.cache/sccache" |
There was a problem hiding this comment.
fyi, /var/lib/buildkite/... didn't work well due to permission issue...
ci/test-dev-context-only-utils.sh
Outdated
| @@ -1,6 +1,29 @@ | |||
| #!/usr/bin/env bash | |||
|
|
|||
| set -eo pipefail | |||
| source ./ci/_ | |||
|
|
|||
| (unset RUSTC_WRAPPER; cargo install --force --git https://github.com/ryoqun/cargo-hack.git --branch interleaved-partition cargo-hack) | |||
There was a problem hiding this comment.
Ideally, should be moved to Dockerfile temporarily after anza's official cargo-hack fork is created...
I'm planning to upstream this pr to cargo-hack once after landing this pr to our master to see improvements. Currently, it's hard to see the effect of this cargo hack change by itself due to general inefficiency of caching...
ryoqun
changed the title
ryoqun
force-pushed
the
even-dcou-builds-wip
branch
3 times, most recently
from
19f777e to
11ab9be
Compare
|
@yihau hey, this pr is ready for code-review. could you update our rust docker image with these changes if things looks acceptable to merge into master branch as an experiment? After that, i'll update the pr. Note that if the time reduce isn't so promising, i want to try local centralized redis server. Worse, I'm fine to revert this pr altogether after landing in a week or two if things went worse. |
ryoqun
force-pushed
the
even-dcou-builds-wip
branch
from
5709051 to
0202e1b
Compare
| # extremely unrealistic for such diverting compilation behaviors to be desired | ||
| # as a sane use-case. So, just unset CI_COMMIT unconditionally to increase | ||
| # cache efficiency. | ||
| unset CI_COMMIT |
There was a problem hiding this comment.
fyi, I put this unset CI_COMMIT here rather than ./ci/test-dev-context-only-utils.sh, because this is beneficial, regardless use of sccache.
Problem
dcou is taking too much time
Summary of Changes
patch upstream cargo hack a bit (todo: create an upstream pr...)
extracted from here: https://github.com/anza-xyz/agave/compare/master...ryoqun:dcou-partition-debug-wip?expand=1
Results
before:
5-15 mins and dcou 3/3 is always taking longer:
sample: https://buildkite.com/anza/agave/builds/17598:
after:
dcou jobs are now 5-10 mins (already i primed the machine-specific caches by repeated manual ci runs, lol), no longer dcou 3/3 isn't slowest. all jobs usually takes same time depending on cache hit rate:
sample: https://buildkite.com/anza/agave/builds/17833 (note that dcou 3/3 ran on machine with no prior dcou run by coincidence):