Skip to content
Agile Defense

Curriculum / Foundry Data Engineering

CAPSTONEUnit 8unit-08

Capstone — ship the trusted backbone for the disruption response

not started~360 min
Integrate everything into a fresh, joined, enriched, quality-guarded, incrementally-built, branch-promoted trusted dataset (plus media + time series) ready for the Ontology.

Introduction

Scenario: The disruption-response team is ready to start building the Ontology and analyst workflows, but they can only move as fast as the data underneath them is trustworthy — so your capstone is to integrate seven units of work into one fresh, joined, enriched, quality-guarded, incrementally-built, branch-promoted trusted backbone and hand it off.

This capstone is not a new feature; it is the moment the whole pipeline has to hold together as a single trusted backbone for the supply-chain disruption response. Across the course you connected the raw feeds with Data Connection, joined and enriched them in Pipeline Builder, computed a supplier risk_score in a Code Repository, guarded the output with data expectations and Data Health, made the build incremental and put it on a Scheduler, promoted it through a branch with the propose/approve workflow, and modeled the advanced shapes (a media set of shipping documents and a time-series sensor feed). Now you prove the lineage actually connects end to end: raw_orders, raw_suppliers, raw_shipments, and raw_inventory landing as datasets, flowing into the joined trusted_supply_chain output, and into the enriched supplier_risk_scored dataset.

What makes a dataset a defensible handoff rather than just a table is everything around the rows. A Foundry dataset is a set of files plus a typed schema, and its history is a sequence of transactions — so a dataset is only ready when it has a non-empty, correctly-keyed schema and a recent COMMITTED transaction proving a build actually ran. On top of that the trusted output must satisfy its primary-key contract (one row per purchase order at the documented grain), and the pipeline that produces it must be the promoted, scheduled version living on the default branch, not a forgotten dev branch. These are the same Datasets v2 reads you have used all course — Get Dataset, Get Dataset Schema, Read Table, List Transactions, List Branches, Get Dataset Schedules — composed here to certify the backbone as a whole.

By the end you will have produced the artifact the Ontology and analyst teams have been waiting for: a trusted data backbone whose lineage, freshness, integrity, and refresh story are all verifiable, accompanied by a written handoff brief. Some of what matters most — the @incremental and trigger configuration, the expectation definitions, the Pipeline Builder and Code Repository internals, the DevOps release plan, and the narrative caveats — cannot be confirmed by any read API, so the final deliverable is a self-attested brief that closes that gap honestly. The whole point of this unit is integration over novelty: you are signing your name to the claim that this data can be trusted downstream.

Capability focus: End-to-end data backbone: sources → join+enrich → expectations/health → incremental schedule → promoted branch → media + time-series, handed off to the Ontology. · Artifact: The end-to-end trusted data backbone, ready for Ontology consumption.

Key concepts

  • Datasets and transactions: A Foundry dataset is a collection of files with a typed schema; its contents change through transactions (SNAPSHOT, APPEND, UPDATE, DELETE) recorded in build history. A dataset is only a trustworthy handoff when Get Dataset and Get Dataset Schema resolve with the expected columns and List Transactions shows a recent COMMITTED transaction — evidence a build actually ran rather than an empty placeholder.
  • Data lineage and the backbone: The trusted backbone is the connected chain of datasets — raw_orders, raw_suppliers, raw_shipments, raw_inventory → trusted_supply_chain (joined) → supplier_risk_scored (enriched). Certifying the capstone means confirming each link in that lineage exists and is schema-correct, so downstream Ontology objects build on data with a known origin.
  • Primary key and grain: The output's grain is the level it is keyed at (one row per purchase order, po_id). A read-side proxy for the primary-key contract is that the distinct count of the key equals the row count over a sample — no duplicate keys. A broken grain (often from a fan-out join) inflates rows and silently corrupts every downstream aggregate.
  • Branches, schedules, and incremental refresh: The promoted pipeline lives on the dataset's default branch (List Branches), and an attached Scheduler schedule (Get Dataset Schedules) keeps it fresh automatically. Incremental computation (the @incremental decorator) lets builds process only new data and produce APPEND transactions instead of full SNAPSHOT rewrites, which is what makes a frequently-scheduled backbone affordable.
  • Data Health and expectations: Data expectations (primary key, row count) configured in Pipeline Builder and Data Health checks (status, freshness, schema-change, Time Since Last Updated) installed on the output are the guardrails that catch regressions and notify owners. Their configuration is largely instructor-checked; only proxies like a COMMITTED transaction (freshness) and key uniqueness (PK) are read-confirmable here.
  • Foundry DevOps and the handoff: Foundry DevOps packages a data-backed workflow (pipelines, data connections, ontology, models) into a product with version/dependency management and release-channel promotion to test and production. The handoff brief documents lineage, grain, keys, refresh cadence, health checks, incremental/schedule config, and the DevOps release plan so the Ontology team can consume and operate the backbone — none of which a read API can attest to on your behalf.

Companion video

End-to-end data backbone walkthrough (placeholder) · open on YouTube

Hands-on activity

each step validates · the unit completes when all steps pass
  1. 1

    All backbone datasets exist and are schema-correct

    Certify that the full lineage of the backbone exists as schema-correct datasets before anyone builds on top of it. Using Datasets v2 GA reads (Get Dataset to resolve the RID, Get Dataset Schema to read the typed columns), confirm that all six datasets are present: the four raw feeds raw_orders, raw_suppliers, raw_shipments, and raw_inventory, the joined trusted_supply_chain output, and the enriched supplier_risk_scored output. This composite check is the read-confirmable proof that the connect → join → enrich chain from Units 1–3 is intact end to end; a missing or empty-schema dataset means a link in the lineage broke and the handoff is not ready.

    not startedinstance check

    Confirms the full lineage of datasets (raw sources, joined output, enriched output) is in place.

  2. 2

    Trusted output is non-empty and keyed (unique PK)

    Prove the trusted output is a usable, integrity-clean handoff and not just a non-empty table. Read a sample of trusted_supply_chain (Read Table / aggregate) and confirm row count > 0 and that the distinct count of the primary key po_id equals the sampled row count — the read-side proxy that the dataset holds one row per purchase order at its documented grain with no duplicate keys. This is the guardrail against a fan-out join silently multiplying rows: if distinct-key < row-count the grain is broken, every downstream aggregate is wrong, and the backbone must not be handed off until the join is corrected back to the stated grain.

    not startedinstance check

    Confirms handoff integrity: sampled rows > 0 and distinct primary-key count = sampled row count.

  3. 3

    Trusted backbone is freshly built

    Confirm the backbone is freshly built and therefore reflects current reality, not a stale snapshot from a past run. List Transactions on trusted_supply_chain and verify a recent COMMITTED transaction exists — direct evidence that a build actually ran and wrote data. This is also the read-side proxy for the freshness story you set up with Data Health (Time Since Last Updated / Data Freshness) and the Scheduler: the health check's pass/fail status itself is not exposed by a read API, so a recent committed build is what we can verify here. Conceptually this is the promoted, scheduled pipeline on the default branch keeping the trusted dataset current; the formal schedule/branch and incremental configuration were certified in Units 5–6 and are documented in the handoff brief.

    not startedinstance check

    Confirms a COMMITTED transaction on the trusted output — the backbone is built and ready for Ontology consumption.

  4. 4

    Document the handoff to the Ontology / analyst teams

    Write the handoff brief that turns a set of verified datasets into something the Ontology and analyst teams can actually adopt — and self-attest it, because most of its contents are not read-API-confirmable. Document the lineage (the six datasets and how they connect), the grain and primary key (one row per po_id), the configured expectations and Data Health checks, the @incremental and Scheduler trigger configuration (including re-trigger-on-successful-build), and the Foundry DevOps release plan for promoting test → production. State this explicitly as a self-attestation: the expectation definitions, @incremental/trigger config, Pipeline Builder and Code Repository internals, and DevOps packaging live behind product internals with no documented READ API, so the brief is your signed claim that the backbone is trustworthy. Include known caveats — the media set + Data Health reads are preview (they can return blocked live, and never silently pass), and the incremental APPEND check is only a proxy that can false-negative on a SNAPSHOT-only build — so the next team inherits the limits along with the data.

    not startedself-attested

    Self-attested: the handoff package (grain, keys, refresh cadence, known caveats) is documented for downstream consumption.