When should you use explicit rules?
Use deliberate rules when a rare or invalid state must exist: a missing address, an expired token or a duplicate reversal. Write the expected outcome next to the scenario. Rules can make coverage intentional, but someone must maintain them as the application changes. Evaluate reusable models and lightweight fixtures separately from a full enterprise execution platform.
When do learned patterns help?
Learned patterns are worth evaluating when distributions and relationships matter to an analytics or ML task. Similar-looking samples are insufficient: compare the downstream task on an independent holdout and inspect rare groups and invalid combinations. Check disclosure or memorisation risk separately. A random seed does not by itself prove that a learned generator will produce identical artifacts across changed runtimes.
How do you validate a hybrid dataset?
Keep the source-derived portion and generated additions identifiable in the build definition. State which relationships must join across both parts and prevent identifier collisions. Validate business coverage, constraints and evaluation utility separately. Record the exact generator, input snapshot, model version and transformations; a label such as synthetic says little about the complete data lineage or residual risk.
Do synthetic records automatically protect privacy?
No. A learned model may reproduce or reveal information about unusual records in its training set. Evaluate the release context, possible linkage, memorisation and utility against held-out data. The ICO describes inference risk for rare training examples; an output labelled synthetic is not a privacy assessment.
What should an acceptance test measure?
Use separate checks for exact scenario rules, aggregate utility and disclosure risk. A payment-status rule can have an exact expected row set; a learned model may instead need distribution and downstream-task comparisons. Report input data, model version, sample size and failed checks; do not convert one metric into a universal score.
References and further reading
Choose your next step
- When should you choose model-driven test data generation?
- AI Test Data Generation: What Works and What Does Not
- Define requirements and a proof of concept
- What is test data? Types, examples and acceptance checks
- Test data generation: a practical first scenario
- Anonymize test data without breaking the test
- Test data without production data: choices and privacy
- Open source or enterprise test data tools?