Reproducible test data for CI/CD: what to freeze and verify

Editorial review: 2026-10-11 · Sources and scope below

What exactly must be reproducible?

Scenario equality means the required cases still exist. Value equality means the same records and multiplicities exist. Byte equality also depends on order, serialization and formatting. Two JSON files can contain the same values but differ in whitespace; two identically wrong files can share a hash. Keep business assertions separate from the replay check.

Which inputs belong in the run record?

Record the generator and dependency versions, model commit, referenced files, explicit seed, reference date, locale and time zone. For mutable data, record the selected snapshot and stable read order. Record execution topology and exporter configuration if they can affect the requested result. Store this beside the test result so a later engineer can reconstruct the run.

Which controls does the generator actually expose?

Faker documents seeded results for the same calls and version. GenRocket documents seeding for supported, configured Generators; bind the seed rather than assuming every Generator uses it. SDV reset_sampling applies to sampling a fitted model. MOSTLY AI distinguishes conditioning rows from random_state. Define the required property and confirm it for the exact route. These controls do not establish identical output across changed software or operating systems.

How do you validate a pipeline before scaling it?

First generate a small fixture and assert keys, relationships and the exact business result. Run twice with frozen inputs and compare the promised property. Next, change one input deliberately and inspect the difference. Finally, interrupt a run and rerun it in an isolated target. Count duplicates, partial writes and leftover data. Measure setup and recovery work separately from generation time.

What does a replay result not prove?

A successful replay for one runtime and exporter is not evidence for every connector, worker count or operating system. It also says nothing by itself about privacy, realistic distributions or business coverage. Learned generation needs a separately defined quality evaluation. Keep untested properties explicitly unverified and extend the proof only when the required path has been exercised.

What about statistical repeatability?

For learned data, the useful contract may be stable task performance or bounded differences in distributions rather than identical rows. Define a holdout, metrics and tolerances before training, and repeat the complete experiment. Identical samples can still be poor or disclose source information. Evaluate business utility and disclosure risk separately from a sampling seed.

Executed customer-label fixture

The local verification harness ran Faker, DATAMIMIC CE and SDV on 11 October 2026 in Linux with Python 3.12. Each product supplies four customer labels. The common Python adapter assigns IDs and the six fixed payment rows. SQLite checks this assembled fixture. The products do not build its relationships.

Faker uses seed_instance(42); CE person/v1 uses seed 42 and a fixed clock. SDV fits a GaussianCopulaSynthesizer on 256 synthetic-only label/score rows. Only its label column enters the fixture. There is no production data, privacy score or distribution-quality evaluation. Cleanup and rollback are checked by the neutral SQLite script.

MOSTLY AI SDK 6.1.5 with LOCAL dependencies was installed without an account. Its actual training attempt was blocked when the environment denied AF_UNIX socket creation (Operation not permitted). No generated output or failed business result is attributed to the product.

This does not test product-managed relationships, Enterprise editions, database connectors, subsetting, privacy, throughput or portability. The exact tested versions are not described as the latest releases.

Download scripts, dependency lock and results (.zip) · Read the run record (.json)

python run_fixture.py
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements-poc.lock
.venv/bin/python compare_generators.py --model-dir .fixture-model --output results-local.json

Illustrative scenarios and editorial acceptance criteria; not measured product benchmarks.

References and further reading

• Faker: seeded replay and version boundaries
• DATAMIMIC CE: local model execution
• DATAMIMIC Platform: scheduled execution and shared projects
• DATAMIMIC Platform: task logs and output artifacts
• GenRocket: configured Generator seeds
• SDV: reset_sampling of a fitted model
• MOSTLY AI: random_state and conditioning seed rows

Choose your next step

• Test data tool PoC: a reusable acceptance checklist
• Open-source test data tools: Faker, DATAMIMIC CE and alternatives
• Define requirements and a proof of concept
• Compare CE and Platform responsibilities