fix(ci): improve ccache persistence across builds #128

forbes · 2026-02-09T22:05:39Z

forbes commented

2026-02-09 22:05:39 +00:00

Problem

Builds take ~65 minutes with 0% ccache hit rate despite restoring a cached ccache directory. CI logs confirm:

Cache restored from key: ccache-build-main-20260207
...
Hits:               0 / 2579 ( 0.00%)
Misses:          2579 / 2579 (100.0%)
...
skipping post step for 'Save ccache'; main step was skipped

Root Cause 1: `compiler_check = mtime` (critical)

The default ccache compiler_check mode hashes the compiler binary's modification time. Since pixi installs a fresh conda environment with fresh compiler binaries on every CI run, the compiler mtime changes every time, invalidating the entire cache.

Root Cause 2: Cache never re-saved after first save

The save step used if: cache-hit != 'true', so once a cache key was restored, subsequent builds never saved their new ccache objects. Every compilation after the first build was thrown away.

Changes

All workflows (`build.yml`, `release.yml`, `build.sh`)

CCACHE_COMPILERCHECK=content — Hash compiler binary content instead of mtime. Same compiler version = same content = cache hits, regardless of when the binary was installed.

Both CI workflows

Save key: Use github.run_id instead of date — every build saves its cache
Save condition: always() instead of cache-hit != 'true'
Removed: Compute cache date key step (no longer needed)

Release workflow

Added ccache-build-main- to restore-keys fallback chain

Expected Impact

With these fixes, incremental builds should see high ccache hit rates (~90%+ for small changes), reducing build times from ~65 minutes to potentially ~10-15 minutes.

## Problem Builds take ~65 minutes with **0% ccache hit rate** despite restoring a cached ccache directory. CI logs confirm: ``` Cache restored from key: ccache-build-main-20260207 ... Hits: 0 / 2579 ( 0.00%) Misses: 2579 / 2579 (100.0%) ... skipping post step for 'Save ccache'; main step was skipped ``` ### Root Cause 1: `compiler_check = mtime` (critical) The default ccache `compiler_check` mode hashes the compiler binary's **modification time**. Since pixi installs a fresh conda environment with fresh compiler binaries on every CI run, the compiler mtime changes every time, **invalidating the entire cache**. ### Root Cause 2: Cache never re-saved after first save The save step used `if: cache-hit != 'true'`, so once a cache key was restored, subsequent builds never saved their new ccache objects. Every compilation after the first build was thrown away. ## Changes ### All workflows (`build.yml`, `release.yml`, `build.sh`) - **`CCACHE_COMPILERCHECK=content`** — Hash compiler binary content instead of mtime. Same compiler version = same content = cache hits, regardless of when the binary was installed. ### Both CI workflows - **Save key**: Use `github.run_id` instead of date — every build saves its cache - **Save condition**: `always()` instead of `cache-hit != 'true'` - **Removed**: `Compute cache date key` step (no longer needed) ### Release workflow - Added `ccache-build-main-` to restore-keys fallback chain ## Expected Impact With these fixes, incremental builds should see high ccache hit rates (~90%+ for small changes), reducing build times from ~65 minutes to potentially ~10-15 minutes.

forbes added 1 commit 2026-02-09 22:05:40 +00:00

fix(ci): improve ccache persistence across builds

Build and Test / build (pull_request) Has been cancelled

Details

7fca71318d

Use github.run_id for cache save keys so every build saves its
accumulated ccache objects, instead of the previous date-based keys
which froze the cache after the first save each day (cache-hit=true
prevented re-saving).

Restore keys still use branch/main prefixes for warm fallback.

Also add ccache-build-main- fallback to release workflow so release
builds can bootstrap from the latest main build cache.

forbes force-pushed fix/ci-cache-strategy from 7fca71318d to dfc32cabd6

2026-02-10 00:04:25 +00:00

Compare

forbes added 1 commit 2026-02-10 12:58:13 +00:00

Merge branch 'main' into fix/ci-cache-strategy

Build and Test / build (pull_request) Successful in 40m34s

Details

f7f44414ef

forbes merged commit 185460077e into main

2026-02-10 12:58:21 +00:00

forbes referenced this issue from a commit

2026-02-10 12:58:22 +00:00