Introduction

Scenario: Operations needs one reliable joined table for the daily disruption review, with late-shipment and inventory-exposure flags.

Day 2 turns yesterday's raw, scattered feeds into one trusted, reviewable table that Operations can open every morning during the disruption review. You will work in Pipeline Builder, Foundry's visual application for constructing data transformation workflows through a node-and-connector graph rather than hand-written code, adding datasets and applying transform nodes (join, filter, aggregate) and previewing results before delivery. The deliverable is a single batch-built dataset output that joins shipment and inventory data on shared keys and carries two derived signals: a late-shipment flag and an inventory-exposure flag.

This matters because a disruption response is only as good as the table underneath it. If shipments and inventory are stitched together on the wrong keys, a single shipment row can fan out against many inventory rows and silently inflate exposure counts, sending the morning standup chasing phantom problems. By being deliberate about grain and join type, and by attaching data expectations that validate schema, nulls, uniqueness, and row counts as the pipeline builds, you make the table both correct and self-policing.

By the end of the day you will have one trusted joined dataset with disruption flags, automated expectations that warn or block when quality slips, and a documented handoff note so downstream analysts inherit your assumptions rather than guessing at them.

Capability focus: Pipeline Builder, transforms, joins, data expectations, Data Health · Artifact: Trusted operational dataset

Key concepts

Companion video

1. 1

   Define the grain and review join keys

   Before opening any transform, decide what one row of the final table should represent for the disruption review (for example, one row per inbound shipment line, or one row per shipment-per-SKU) because grain dictates which keys you join on and whether counts will be trustworthy. Inspect the shipment and inventory inputs in Pipeline Builder and confirm the candidate keys (such as shipment_id, SKU, or facility) are present, consistently typed, and unique at your intended grain. Write the chosen grain and keys down now, since picking the wrong key is what causes a single shipment to fan out against many inventory rows and inflate exposure later.

   not startedself-attested

   Mark complete

   Self-attested: you stated the output grain and verified the join keys.

2. 2

   Build the trusted joined dataset

   In Pipeline Builder, add the shipment and inventory datasets as inputs, drop in a Join transform, and set the join condition on the keys you validated in the previous step. Choose the join type deliberately: a left join from shipments keeps every shipment even when no inventory match exists, while an inner join drops unmatched rows, so pick the one that matches how Operations wants to read the table. Use the preview to confirm the output row count is at your intended grain (not multiplied), and resolve any duplicate column names from the two inputs by renaming so the joined schema is unambiguous before you write a dataset output.

   not startedinstance check

   Run check

   Confirms the trusted_supply_chain dataset exists in the project.

3. 3

   Add the late-shipment and inventory-exposure flags

   Add derived columns to the joined data using transform expressions so the disruption signals live in the table itself rather than in someone's head. Compute a late-shipment flag by comparing actual or expected arrival against the promised delivery date, and an inventory-exposure flag by comparing on-hand quantity against a threshold (for example, below safety stock given the delayed shipment). Preview a few known cases to confirm the flags fire correctly, since these two columns are what the morning review will filter and sort on.

   not startedinstance check

   Run check

   Confirms the trusted dataset schema carries the late and exposure risk signals.

4. 4

   Add data expectations

   Attach data expectations to the output so quality is enforced automatically every time the pipeline builds, not checked by hand. Configure expectations that match this table's contract: schema and column types are stable, the grain key is unique (no unexpected fanout), late and exposure flags contain no nulls, and the row count stays within a sane range. Set the high-stakes checks (uniqueness, schema) to blocking/abort mode so a broken build never reaches the review, and leave softer checks in warning mode so the team is flagged without halting the daily refresh.

   not startedself-attested

   Mark complete

   Self-attested: data-expectation configuration is not API-readable (instructor-checked).

5. 5

   Document assumptions (handoff note)

   Write a short handoff note capturing the grain, the join keys and join type you chose, the exact thresholds behind each flag, and the expectations you set and why, so downstream analysts inherit your reasoning instead of reverse-engineering it. Note any known gaps (for example, facilities that lack inventory records and therefore appear without exposure values) and recommend the Data Health checks worth adding, such as a freshness check so the team is alerted if the daily inputs stop updating. Store this note with the Project alongside the dataset so the trusted table and its assumptions travel together.

   not startedself-attested

   Mark complete

   Self-attested: your handoff note makes the join logic and assumptions explainable.