Curriculum / Foundry Data Engineering
Guard the data with expectations and Data Health
Introduction
Scenario: The trusted supply-chain backbone you joined and enriched is about to become the single source of truth for the disruption response, so a silent duplicate-key bug or a stale upstream feed would now corrupt every downstream decision. Your job this unit is to wrap that dataset in data expectations and Data Health checks so quality regressions are caught at the pipeline and surfaced before anyone acts on bad data.
Until now you have been building the trusted backbone: connecting feeds, joining and shaping them in Pipeline Builder, and computing the supplier risk score in a Code Repository. This unit shifts from producing data to defending it. A disruption-response workflow is only as reliable as its weakest input, and the failure modes that hurt most are quiet ones — a join that fans out and duplicates primary keys, a row count that silently collapses after an upstream schema change, a feed that simply stops refreshing. You will install two complementary guards: data expectations that live inside the pipeline and can fail the build, and Data Health checks that monitor the published dataset and alert when something drifts.
Data expectations are validation rules you attach to a Pipeline Builder output (or an intermediate transform). Documented expectation types include primary key and row count, and a failing expectation can fail the build and/or surface as a health check — so it acts as a gate that stops bad data from being committed in the first place. Data Health, by contrast, monitors and alerts on common dataset issues — status, freshness, size, content, and schema change — and sends in-platform notifications and emails when checks fail. Named check types include Job Status, Schedule Status, Schedule Duration, Time Since Last Updated (freshness), Data Freshness, and Sync Freshness, so you can pick exactly what you want to watch.
By the end of the unit the trusted_supply_chain dataset carries a primary-key and row-count expectation in the pipeline plus a set of Data Health checks (for example job status and time-since-last-updated) installed on the published dataset. A note on honesty before you start: most of this configuration lives inside Pipeline Builder and Data Health internals that no confirmed read API exposes. The platform can confirm that health checks exist (via a preview enumeration endpoint), and you can prove the conditions hold from the read side — distinct key count equals row count, and a recent committed transaction exists — but it cannot read back an expectation's definition or a health check's pass/fail report. Where that is the case, you self-attest and the instructor verifies the configuration directly in the UI.
Capability focus: Data expectations (PK, row count); Data Health checks (status, freshness, schema); blocking vs warning. · Artifact: Data expectations on the pipeline output plus Data Health checks installed on the trusted dataset.
Key concepts
- Data expectations: Validation rules attached to Pipeline Builder dataset outputs and intermediate transforms. Documented expectation types include primary key and row count; a failing expectation can fail the build and/or surface as a health check, making expectations a build-time gate that prevents bad data from being committed. (Expectation behavior is partial-confidence in the docs and configured in the Pipeline Builder UI, not via a confirmed GA read API.)
- Data Health: A monitoring system that watches datasets for common issues — status, freshness, size, content, and schema change — and sends in-platform notifications and emails when a check fails. It surfaces regressions on the published dataset after the build, complementing the build-time gate that expectations provide.
- Health check types: Named, configurable check types include Job Status, Schedule Status, Schedule Duration, Time Since Last Updated (freshness), Data Freshness, and Sync Freshness. You choose which checks to install on the trusted dataset based on what 'healthy' means for it (e.g., it built successfully and it built recently).
- Blocking vs. warning: Expectations and health checks can be treated as blocking (fail the build / hard-stop downstream) or as warnings (alert but allow the data through). Choosing the severity is a deliberate data-engineering decision — a duplicate primary key usually warrants blocking, while a soft freshness window may only warrant a warning.
- Transactions: Each successful build commits a transaction on the dataset. A COMMITTED transaction within an expected recency window is read-side evidence that the dataset is fresh and was actually built, and List Transactions (GA) can enumerate these — which is how freshness/status can be proven from the read API even though a health check's pass/fail report cannot.
- Primary key as grain contract: The primary-key expectation encodes the dataset's grain — the level at which one row means one real-world thing (here, one purchase order, po_id). When distinct po_id count equals the row count, the grain holds and no upstream join has fanned out; when it does not, a duplicate-key regression has slipped in.
Companion video
Data expectations & Data Health walkthrough (placeholder) · open on YouTube
Hands-on activity
each step validates · the unit completes when all steps pass- 1
Configure primary-key and row-count expectations
In the Pipeline Builder pipeline that produces the trusted dataset, add two data expectations on the output: a primary-key expectation on po_id (asserting it is unique and non-null, which encodes the dataset's grain) and a row-count expectation (asserting the count stays within a sane range so a silent collapse or explosion is caught). Decide and set the severity: a duplicate primary key should be blocking so the build fails rather than commit corrupt data, while a row-count drift may be a warning. This step is self-attested — expectation definitions and their pass/fail history are not enumerable via a confirmed GA read API, and the blocking-vs-warning behavior is partial-confidence in the docs and configured in the Pipeline Builder UI — so the instructor verifies the expectations directly in the pipeline. A failing expectation surfaces in Data Health, which is the bridge to the checks you install next.
not startedself-attestedSelf-attested: PK + row-count expectations are configured (definitions/pass-fail history are not enumerable via a confirmed GA read API).
- 2
Health checks are installed on the trusted dataset
On the published trusted_supply_chain dataset, install Data Health checks so quality regressions are surfaced after the build — for example a Job Status check (the build succeeded) and a Time Since Last Updated / freshness check (the data built recently). Data Health sends in-platform notifications and emails when a check fails, so these become the team's early warning system. The platform confirms this step via the Get Dataset Health Checks endpoint (datasets v2), which returns the RIDs of the configured checks — enumeration only, proving checks exist, not whether they currently pass. Note this is a PREVIEW endpoint (requires preview=true): if it is unreachable live it returns 'blocked' rather than silently passing, so a blocked result means 'could not confirm', not 'no checks'. Configure at least one check, then have the instructor confirm the specific check types and thresholds in the Data Health UI.
not startedinstance checkConfirms Data Health checks are configured (enumeration only; preview → blocked when unreachable).
- 3
Primary key holds in the data
Prove the primary-key expectation's condition actually holds in the data, independent of whether the expectation is configured. The check reads a sample from the output (Read Table / Aggregate) and asserts that the distinct count of po_id equals the sampled row count — i.e., there are no duplicate keys and the grain you declared is real. This is a deliberate read-side proxy: because an expectation's pass/fail status is not read-confirmable through a GA API, confirming distinct-po_id-equals-row-count is the closest observable evidence that the primary key holds. If this fails, suspect an upstream join that fanned out (a one-to-many relationship treated as one-to-one), which is the single most common way a primary-key guarantee breaks; fix the grain in the pipeline rather than papering over it with a deduplication step.
not startedinstance checkConfirms distinct primary-key count equals sampled row count (no duplicate keys) — a read-side proxy that the PK expectation holds.
- 4
Dataset is fresh / recently built
Confirm the dataset is fresh and was actually built by checking that a COMMITTED transaction exists on trusted_supply_chain within the expected recency window. List Transactions is a GA read endpoint, so this is directly observable. Treat it as a read-side proxy for the freshness and status health checks you installed in step two: a health check's pass/fail report is not confirmed to be exposed by any read API, so a recent COMMITTED transaction is the closest provable signal that the freshness/status guards have something healthy to watch. A committed transaction here demonstrates that the build pipeline ran end to end and published output — the precondition for every downstream consumer of the trusted backbone.
not startedinstance checkConfirms a COMMITTED transaction exists — a read-side proxy for the freshness/status health checks (whose pass/fail is not read-API-exposed).