Build Caching, Local and Remote

Remote caching is the highest-leverage change available to a large build, and the difference between a well-configured one and a badly-configured one is the difference between eighty percent hits and eight.

The idea

Local cache: results of previous runs on this machine, reused when inputs match.

Remote cache: the same, shared. Someone builds it once — usually continuous integration — and everyone else downloads the outputs instead of computing them.

The hit rate is everything. A cache that misses is a cache that costs you a network round trip and gives nothing.

What breaks the hit rate

Diagnosing this is most of the work, and the causes are a short list.

Absolute paths baked into outputs. The most common single cause. A build run in /home/alice/project produces different bytes from one in /home/bob/project, so the hashes differ and nothing is shared.

Timestamps embedded in artefacts.

Environment differences: compiler version, locale, JDK or toolchain version, operating system. Any undeclared difference means a different hash.

Non-deterministic ordering, where a task iterates a directory or a set in an unstable order.

Overdeclared inputs, which is the incremental build problem showing up again: if a task depends on more than it uses, it invalidates more often and shares less.

Different flags between local and CI, which is very common and means the two never share anything.

Making it hit

Normalise paths. Relative paths in inputs and outputs, and path mapping where the tool supports it.

Pin the toolchain. Same compiler version everywhere, declared as a cache input rather than assumed.

Remove timestamps and build metadata from artefacts, or exclude them from the hash.

Sort anything that iterates.

Run the same tasks with the same flags locally and in CI, or accept that they will never share.

Populate from CI. Developers should mostly read from the cache; the writers should be trusted machines. Letting every laptop write invites poisoning and inconsistency.

Correctness before speed

A cache that returns a wrong result is worse than no cache, and the failure is hard to diagnose.

Only allow caching for tasks whose inputs are fully declared.

Do not cache tasks with side effects outside their declared outputs.

Verify periodically: run a build with the cache disabled and compare outputs. Differences are undeclared inputs.

Separate the write and read permissions. Untrusted contexts — forked pull requests, developer machines — should read only.

Measuring it

Hit rate, overall and by task. A low overall rate is usually a small number of tasks missing constantly.

Bytes transferred, which tells you whether the network is now the bottleneck.

Time saved, as the difference between cached and uncached duration for the same work.

Hit rate by machine type. A rate that is high in CI and low locally is the path or environment difference, every time.

Where it does not help

Small codebases, where the network round trip exceeds the computation.

Tasks that are already fast.

Genuinely novel work. The first build of a new change cannot be cached, which means caching helps the ninety percent of runs that repeat work and does nothing for the ten percent that matter most to the person waiting.

Which is why caching complements incremental builds rather than replacing them.

Diagnosing a low hit rate

A sequence that finds the cause in an afternoon.

Compare cache keys for the same task between two machines. The differing component is the cause.

Check for absolute paths in the key inputs and in the outputs.

Check toolchain versions on both machines, including patch versions.

Check locale and time zone.

Check the flags each context passes; local and CI frequently differ.

Fix the top cause and re-measure. Hit rates in this field are usually near zero or near eighty percent, with little in between, because one systematic difference defeats everything.

Who may write

A permission decision with correctness and security consequences.

Trusted CI machines write.

Developers read.

Forked pull requests read only, always, or an untrusted change can poison what everyone else restores.

A poisoned entry propagates silently and is extremely hard to diagnose, because the build is correct on the machine that produced it.

Set this at the start. Widening write access later is easy; discovering why everyone's build is wrong is not.