Getting started

Overview

General Validation is built for one practical question: did the data that landed match the data you meant to land?

It starts by reading Azure Data Factory metadata: datasets, linked services, pipelines, and schemas. From there, you pair a source dataset with its target, define the checks that matter for that pair, and run the validation through ADF inside your own Azure environment.

The app stores validation metadata and result metrics. Your source rows, target rows, result Delta tables, and bad-record evidence stay in your Azure.

The flow

1. Connect Azure Data Factory for read-only metadata discovery.
2. Select the source and target datasets that should agree.
3. Define validation tests: row counts, counts, distinct counts, sums, averages, min/max checks, value comparisons, outer-value comparisons, and set checks.
4. Run the validation. General Validation compiles the tests into a reusable ADF Mapping Data Flow and ADF reads the data inside your environment.
5. Review the result: status, source value, target value, delta, threshold, diagnostics, and evidence pointers for checks that capture bad records.

What you need for a first run

• An Azure Data Factory with datasets in the runnable scope.
• A source and target dataset pair that should reconcile.
• A small set of checks your team would trust if they passed.
• Customer-owned Delta storage for validation results and bad-record evidence.

Supported today

Supported source formats today: Parquet, CSV / delimited text, Delta Lake, and the Azure SQL family (including Managed Instance and Synapse).

Non-flat formats such as JSON, REST payloads, Avro, and binary files can be cataloged for tie-out, but they are not runnable validation targets yet.