Artefacts and Dependency Fetching

On a fresh runner, a substantial share of build time can be spent fetching things from the internet. It is rarely measured and usually straightforward to reduce.

Where the time goes

Dependency resolution, where the tool queries a registry to work out versions. Slow, network-bound, and avoidable with a lock file.

Downloading, which scales with the size of the dependency set.

Extracting archives, which is disk-bound.

Verifying checksums, usually cheap.

Measure each, because the mix differs by ecosystem and the fix differs by component.

The fixes

Commit a lock file and use it. Resolution then becomes a lookup rather than a search.

Run a local mirror or proxy of the public registry. Faster, and it removes a dependency on a service you do not control.

Cache the dependency directory in CI, keyed on the lock file hash.

Prime runner images with the common dependency set, so the cold case is warmer.

Fetch in parallel, where the tool supports it.

Prune what you do not use. Dependency sets accumulate, and unused entries are downloaded on every cold build forever.

The registry as a dependency

Worth treating as an availability concern rather than an assumption.

A public registry outage stops all builds if you fetch directly.

A proxy with a cache keeps you working through a short outage.

Pinned versions plus a cache means a removed or yanked package does not break historic builds.

Vendoring — committing dependencies into the repository — is the strongest form and carries a real cost in repository size and review noise. Defensible for a small critical set, unwieldy as a general policy.

Supply chain, briefly

Not the subject of this collection, and the overlap is worth naming.

Pinning by digest rather than by tag helps reproducibility and integrity simultaneously.

Checksum verification should be on. It is cheap.

A proxy gives you a record of what was fetched, which is useful when a compromised package is announced.

These are the same practices that make builds fast and cacheable, which makes the case easier to fund than a pure security argument would be.

Artefact publishing

The other direction, and it has its own costs.

Publishing large artefacts on every build consumes storage and time. Publish what is needed.

Retention policies, or storage grows without bound and someone deletes it urgently at the worst moment.

Deduplication, where the tooling supports it.

Do not publish from the blocking path if the artefact is only needed later.

What to measure

Fetch time as a proportion of cold build duration.

Cache hit rate on the dependency cache, separately from the build cache.

Registry response times, which drift and are invisible until someone looks.

Dependency count and total size, over time. Both grow monotonically without attention, and the growth is a slow tax on every cold build.

Pruning the dependency set

Dependency lists grow monotonically and are almost never reduced.

List what is declared.

List what is actually imported, which most ecosystems have tooling for.

The difference is downloaded on every cold build, forever.

Remove it, in one change, and measure the fetch time before and after.

Repeat annually, because the gap reopens with every abandoned experiment that left its dependency behind.

The registry as a single point of failure

Worth treating as an availability dependency rather than as background infrastructure.

A public registry outage stops every cold build.

A caching proxy keeps you working through a short one, and gives you a record of what was fetched.

Pinned versions plus a cache mean a yanked package does not break historic builds.

Test it: block the public registry in a controlled run and see what still works.

Most organisations discover the answer during an outage, which is the expensive way to find out.