Test Data and Fixtures

In many suites, most of the time is spent getting to the point where the test can start. It is rarely measured separately from the test itself.

Measuring setup separately

Most frameworks can report setup and teardown time distinctly from test execution.

Do it once. The proportion is usually startling — setup dominating execution by a wide margin is normal.

Aggregate by fixture, which identifies the expensive ones.

That list is the work queue, and it is short.

Where the time goes

Database schema creation per test or per class.

Loading large fixture files where a handful of rows would do.

Full application context startup for a test exercising one component.

Container startup per test rather than shared.

Network calls to real services during setup.

Generating data that could be created once and reused.

Reducing it

Share what is immutable. A schema, a container, a loaded configuration — created once per suite, not per test.

Isolate only what is mutable. Transactions rolled back after each test are far cheaper than recreating a database.

Build fixtures in code, not from files, where the file is large and only a fraction is used.

Use builders with defaults, so a test that cares about one field does not construct a hundred.

Lazy-load. Do not create what the test does not touch.

Snapshot and restore an expensive prepared state rather than rebuilding it.

The realism trade-off

The genuine tension, and it should be a decision rather than a drift.

A real database catches schema and query problems that an in-memory substitute does not, and it is slower.

An in-memory substitute is fast and behaves differently — different SQL dialect, different constraint handling, different transaction semantics. Tests that pass against it and fail against the real thing are worse than no tests.

A reasonable arrangement: the real engine, shared across the suite, with transaction rollback for isolation. Most of the realism, most of the speed.

Avoid the substitute that is nearly the same, which is where the expensive surprises come from.

Fixture maintenance

Fixtures accumulate. A file added for one test grows as others need one more field, until it is a large object nobody understands.

Tests become coupled to fixture details they do not care about, so an unrelated change breaks fifty tests.

Prefer per-test construction with defaults over shared fixture files, which decouples tests from each other.

Delete unused fixtures, which requires knowing which are used — a coverage question applied to test data.

What to measure

Setup time as a proportion of suite duration.

The ten most expensive fixtures.

Suite duration against test count, which shows whether setup cost is growing faster than the suite.

Time to run a single test in isolation, which is the number developers experience most and which fixed per-test setup dominates.

Transaction rollback for isolation

The arrangement that gets most of the realism at most of the speed.

One real database for the suite, started once.

Each test runs inside a transaction that is rolled back afterwards.

Isolation without recreation, which is orders of magnitude faster than a fresh schema per test.

Watch for tests that commit, which escape the rollback and pollute the next test.

Watch for code that manages its own transactions, which is the usual reason this arrangement does not work and is worth knowing before adopting it.

Building fixtures in code

The change that decouples tests from each other and from a growing shared file.

A builder with sensible defaults, where a test overrides only what it cares about.

No shared fixture file that every test loads and nobody understands.

A test that cares about one field constructs one field, which makes it readable and makes an unrelated change unable to break it.

Migrate incrementally, as tests are touched.

The signal that you need this is a change to a fixture breaking fifty unrelated tests.