Incremental Builds

The fastest build is the one that does not run. Incremental builds are how that happens, and most codebases have them configured in a way that quietly defeats them.

How it works

The build tool knows the inputs to each task — source files, dependencies, configuration, environment.

It hashes them. If the hash matches a previous run whose outputs it still has, it reuses the outputs instead of running the task.

The unit is the task, not the project. A well-modelled build skips most tasks on most runs.

Everything depends on the inputs being declared correctly, which is where it goes wrong.

Why it stops working

Almost every case is one of these.

Undeclared inputs. A task reads a file the build system does not know about. The hash does not change, the task is skipped, and the output is stale — which is worse than slow.

Overdeclared inputs. A task declares a whole directory when it uses one file. Any change in the directory invalidates it. This is the more common failure and it is silent: the build is simply slower than it should be.

Absolute paths in inputs or outputs, which differ between machines and defeat sharing.

Timestamps and non-determinism. A build that embeds the current time in its output never matches a previous run.

Environment leakage. A task that reads an environment variable not declared as an input produces wrong results when it changes, or misses changes when it should not.

Tasks that are not idempotent, so reuse is unsafe and the tool disables it.

Finding the problem

Run the build twice with no changes. The second run should do almost nothing. If it does substantial work, tasks are being invalidated by something that did not change.

Ask the build tool why. Most modern tools can explain which input changed and caused a task to re-run. This is the single most useful diagnostic in the field and most teams have never used it.

Look for the task that always runs. There is usually one, it usually has an undeclared or overdeclared input, and fixing it frequently transforms the whole build.

Check for absolute paths in the build output, which is the usual reason cache sharing fails between machines.

Granularity

A trade-off worth making deliberately.

Coarse tasks are simpler to declare and invalidate more work when anything changes.

Fine tasks skip more but add scheduling overhead and more places to get the declarations wrong.

Module boundaries matter more than task granularity. A codebase in one enormous module can only ever rebuild everything; splitting it is what makes incremental work possible at all.

Which is why build performance and codebase structure are the same problem, and why a build cannot be made fast without touching how the code is organised.

What to measure

Second-run duration with no changes. Should be near zero.

Proportion of tasks skipped on a typical incremental build.

The most frequently re-run task, which is your work queue.

Duration distribution, where a bimodal shape means some runs hit the incremental path and some do not, and the reason for the second mode is the finding.

Asking the build tool why

The most useful diagnostic available and the one most teams have never run.

Modern build tools can report why a task re-ran — which input changed, and what its previous and current values were.

Run it after a no-op build. Any task that executed had an input change, and the tool will name it.

The answer is usually a timestamp, an absolute path, or an overdeclared directory.

Fix the top one and re-run. Build performance work is frequently a short sequence of these, each taking minutes and each removing a class of invalidation.

The stale output problem

The failure mode that is worse than slowness and harder to diagnose.

An undeclared input means a task is skipped when it should have run.

The build is green and the output is wrong, which surfaces later somewhere confusing.

Symptoms: a clean build behaves differently from an incremental one; a problem disappears after clearing caches.

When someone says "try a clean build", that is this, and it should be treated as a defect rather than as a workaround.

Find it by comparing outputs from a clean and an incremental build of the same commit.