Reproducible Builds

A build is reproducible when the same inputs produce byte-identical outputs regardless of where and when it runs. Most builds are not, and the consequences are wider than most people expect.

Why it matters beyond tidiness

Caching depends on it. If two machines produce different bytes from the same source, they cannot share results, and the cache hit rate collapses.

Debugging depends on it. "Works on my machine" is a reproducibility failure with a friendly name.

Verification depends on it. You cannot confirm that a released artefact came from a particular commit if rebuilding it produces something different.

Supply chain assurance depends on it, which is increasingly a procurement question rather than an engineering preference.

What breaks it

The list is short and every codebase has some of it.

Timestamps embedded in artefacts — build time, file modification times in archives.

Absolute paths in compiled output, debug symbols or generated code.

Non-deterministic ordering: directory iteration, hash map iteration, parallel output interleaving.

Environment variables read but not declared.

Locale and time zone, which affect string sorting and date formatting.

Unpinned dependencies, where "latest" resolves differently on different days.

Random values: generated identifiers, temporary file names embedded in output.

Network access during the build, which makes the result depend on the state of the world.

Finding the differences

Build twice on the same machine and compare. Any difference here is non-determinism within one environment, which is the easiest class to fix.

Build on two different machines and compare. Differences here are environment leakage.

Use a binary diff tool that can look inside archives, because most differences are inside a jar, wheel or container layer rather than at the top level.

Fix in that order. Same-machine determinism first, then cross-machine.

Making it hold

Pin everything. Toolchain versions, dependency versions, base images — declared, not resolved at build time.

Set a fixed timestamp for artefact metadata rather than using the current time.

Normalise paths to be relative, or map them explicitly.

Sort every collection that reaches an output.

Fix locale and time zone in the build environment.

Forbid network access during the build proper. Fetch dependencies in a separate, cached step.

Then verify it in CI: build twice, compare, fail on difference. Without that check it decays within a quarter.

The realistic target

Full byte-for-byte reproducibility across every artefact is a substantial project and is not always warranted.

The practical version: reproducible enough that the cache hits reliably and that two machines produce equivalent results.

Prioritise by cache impact. The tasks whose outputs are large and frequently reused are the ones worth making deterministic first.

Exclude what you cannot fix, explicitly, from the hash rather than pretending it is stable.

Record the exceptions with a reason, because an undocumented exclusion is how a wrong result gets cached and shared.

The two-build check

A check that costs one pipeline stage and defends the property permanently.

Build twice in the same pipeline run.

Compare the outputs byte for byte.

Fail on difference, with the differing paths reported.

Add it once reproducibility is achieved, or it fails constantly and gets disabled.

Exclude the artefacts you have accepted as non-deterministic, by name, with the reason recorded.

Without this check the property decays within a quarter, because every new task is a chance to reintroduce a timestamp.

Accepting exceptions properly

Full reproducibility is rarely worth pursuing to the last artefact, and the exceptions need recording rather than ignoring.

Name each artefact that is not deterministic.

Say why, specifically.

Exclude it from the hash rather than pretending it is stable, which is what produces wrongly-shared cache entries.

Set a review date.

Report the count, which should shrink. An unreviewed exception list grows until reproducibility is nominal.