Introduction

Scenario: The raw orders, suppliers, shipments, and inventory feeds have landed as separate datasets, but the disruption-response team cannot see which late shipments threaten which purchase orders until those feeds are stitched into one trusted table. Your job is to shape and join them in Pipeline Builder into a single dataset at a documented grain — the joined backbone the rest of the supply-chain workflow will stand on.

Unit 1 landed four disconnected feeds; on their own they answer nothing. A buyer cannot ask 'which of my open purchase orders are exposed by a late shipment from a high-risk supplier' until orders, suppliers, shipments, and inventory live in one table with a clear shape. This unit builds that table entirely no-code in Pipeline Builder, Foundry's visual pipeline authoring tool. Pipeline Builder expresses logic as a graph of transformations: 'expressions' operate at the column level (deriving, cleaning, casting individual fields) and 'transforms' operate at the table level (filter, pivot, and the joins that combine datasets); user-defined functions exist as an escape hatch when the built-in library is insufficient, but you should not need them here.

The single most important decision in this unit is grain — the question 'what does one row of the output represent?' Every join either preserves the grain you intend or silently changes it. A four-way join across orders, suppliers, shipments, and inventory can fan out: if one purchase order has many shipments, an inner or left join on the order key multiplies that order's row, and any downstream count or sum of order-level values becomes wrong. So you state the grain first (one row per purchase-order / shipment line), choose each join type and key deliberately to honor it, and treat the resulting schema and primary key as the contract the output must satisfy.

Foundry cannot read the inside of your pipeline through a documented API — the node graph and the join logic you wired are not externally inspectable — so the checks in this unit confirm the artifact the pipeline produces, not the wiring that produced it. When you build the pipeline, it deploys and writes to a build-output dataset, committing a transaction. We verify that the output dataset exists, that its schema carries the merged column set and a primary key, and that a build committed rows. That gap between 'what you wired' and 'what we can confirm' is exactly why the first step is an honest self-attestation and why grain discipline is on you, not the validator.

Capability focus: Pipeline Builder transforms (expressions vs table transforms), joins, output grain, build-output datasets. · Artifact: A Pipeline Builder pipeline outputting the trusted joined dataset (orders ⋈ suppliers ⋈ shipments ⋈ inventory) at a documented grain.

Key concepts

Companion video

1. 1

   State the grain and build the pipeline graph (transforms + joins)

   Start by writing down the output grain in one sentence — one row per purchase-order / shipment line — because every join decision that follows must preserve it. In Pipeline Builder, add the four raw feeds (orders, suppliers, shipments, inventory) as inputs, use column-level expressions to clean and cast keys and dates and to standardize field names, then wire the table-level Join transforms that combine them on their keys, choosing each join type (inner vs left) deliberately so you keep the orders you need without fanning rows out. Derive the output fields the downstream workflow expects, including a late_flag (whether a shipment missed its promised date) and an exposure_usd value, and a stable primary key (po_id) for the chosen grain. This step is self-attested: Pipeline Builder's internal node graph and join logic are not exposed by any documented read API, so no automated check can confirm the wiring — only the dataset it produces is confirmable, which is why stating and honoring the grain here is entirely your responsibility.

   not startedself-attested

   Mark complete

   Self-attested: the output grain is stated and the transform/join graph is built (Pipeline Builder internals are not API-readable).

2. 2

   Trusted joined output dataset exists

   Build the pipeline so it deploys and writes its result to the build-output dataset trusted_supply_chain. Building Pipeline Builder logic produces real Foundry datasets that downstream transforms and the Ontology can consume. The check here calls Get Dataset (Datasets v2, generally available) on the expected output: a resolving dataset confirms the pipeline was actually deployed and produced an output rather than left as an unbuilt draft. If this fails, the pipeline either was never built or wrote to a different name or path than expected — reconcile the output dataset's name and location before moving on.

   not startedinstance check

   Run check

   Confirms the pipeline deployed and produced the joined output dataset.

3. 3

   Output has the expected joined schema and grain

   With the output dataset in place, confirm it carries the intended shape. The check calls Get Dataset Schema (generally available) and expects the merged column set to include po_id, supplier_id, shipment_id, late_flag, and exposure_usd — fields drawn from all four feeds plus the primary key for the stated grain. A dataset's schema is the closest externally inspectable proxy for join shape and grain: it can confirm that columns from each input survived the join and that a primary key column exists, but it cannot prove row-level uniqueness or that no fan-out occurred. Make sure the join actually surfaces every expected column with sane types; if a column is missing, a join key or a derived expression upstream is wrong.

   not startedinstance check

   Run check

   Confirms the merged column set + primary key — the read-confirmable proxy for join shape/grain.

4. 4

   Pipeline produced rows (a build committed)

   Finally, confirm the pipeline did not just define logic but executed and persisted rows. The check uses Read Table Dataset / List Transactions to verify a COMMITTED transaction exists on trusted_supply_chain — the atomic record Foundry writes when a build succeeds. A committed transaction with rows confirms the four-way join ran end to end and produced a non-empty result, closing the loop from raw feeds to a single trusted joined dataset. If the dataset exists with the right schema but has no committed transaction, the build was configured but never successfully ran; re-run it and check the build logs for a transform or join error.

   not startedinstance check

   Run check

   Confirms the join executed and committed a transaction.