Introduction

Scenario: The trusted supply-chain backbone the disruption-response team relies on is rebuilt from scratch every run, so refreshes are slow and expensive and the team cannot get same-day visibility into newly disrupted shipments. Your job is to make the transform process only new and changed transactions and to put it on an automatic schedule so the backbone stays fresh on its own.

Up to now the trusted joined-and-scored backbone has been a SNAPSHOT pipeline: every build recomputes the entire output from the full inputs, which is correct but slow and costly as the orders, shipments, and inventory feeds grow. In this unit you convert the transform to incremental computation. Foundry's @incremental decorator wraps your transform's compute function and reads the dataset's build history so that, on a normal run, only the new or changed input transactions are read and only new rows are appended to the output instead of replacing everything. That is what turns a backbone that takes minutes-to-hours into one that can refresh cheaply and often enough to keep up with a live disruption.

Incremental computation is governed by parameters you set on the decorator, and getting them right is the whole game. semantic_version is an integer you bump deliberately to force a full recompute when your logic changes (otherwise old appended output would be inconsistent with new code); snapshot_inputs names reference inputs that should always be read in full rather than incrementally (for example a small supplier-mapping table); and v2_semantics opts into the newer, clearer incremental behavior. Other parameters such as require_incremental, allow_retention, and strict_append exist for stricter guarantees. Crucially, whether a transform is configured incremental is an authoring-time property of the code — it is not exposed by any documented read API — so downstream we can only observe its effect on transactions, never the decorator itself.

Once the transform is incremental, you make it run on its own with the Scheduler. A schedule targets one or more datasets and carries a trigger: time-based (a cron-style cadence), event-driven (build when new data lands upstream), or manual. For a backbone that should drain a backlog of inputs, Foundry also offers 're-trigger upon successful build', which keeps building until all available input transactions are processed — but that option requires a target with incremental transaction limits (for datasets) or incremental batch limits (for media sets), which is exactly why the incremental work in this unit comes first. By the end you will have an incremental transform whose output grows by APPEND transactions and a Scheduler schedule that keeps the trusted dataset fresh without anyone pressing build.

Capability focus: @incremental computation (semantic_version, snapshot_inputs); Scheduler schedules + triggers (time/cron, on-new-data, re-trigger-on-success). · Artifact: An incremental transform on the output plus a Scheduler schedule keeping the trusted dataset fresh.

Key concepts

Companion video

1. 1

   Add the @incremental decorator with correct semantics

   Open the Code Repository transform that produces the trusted backbone and apply the @incremental decorator above your compute function. Set the parameters deliberately: bump semantic_version whenever your transformation logic changes so Foundry forces a clean full recompute instead of appending rows that are inconsistent with old output; list any small reference inputs (for example a supplier-mapping table) under snapshot_inputs so they are always read in full rather than incrementally; and enable v2_semantics for the current incremental behavior. This step is self-attested and instructor-checked: whether a transform is configured incremental is an authoring-time property of the code, and no documented Foundry read API exposes the decorator or its parameters — only the resulting transaction behavior is observable downstream. State which parameters you set and why so a reviewer can judge the configuration.

   not startedself-attested

   Mark complete

   Self-attested: @incremental is applied with appropriate parameters (whether a transform is incremental is not API-readable).

2. 2

   Incremental behavior shows as appended transactions

   Build the incremental transform against inputs that have grown by new transactions, then inspect the output's transaction history. Using List Transactions (a GA read endpoint) on the trusted output, you should see an APPEND, COMMITTED transaction rather than a fresh SNAPSHOT that replaces the whole dataset — APPEND is the read-observable signature that incremental computation read only the new input transactions and added rows instead of recomputing everything. Confirm at least one APPEND transaction landed. Be aware this check is a PROXY (DE-VAL2): a single SNAPSHOT-only build — such as the very first build, a forced full recompute after a semantic_version bump, or a run with no new input data — can legitimately produce a SNAPSHOT and cause this APPEND check to false-negative, so treat a missing APPEND as a prompt to verify the input actually had new transactions, not automatic proof that @incremental is misconfigured.

   not startedinstance check

   Run check

   Confirms an APPEND (not repeated SNAPSHOT) transaction — read-side evidence incremental computation ran (DE-VAL2: APPEND-match is a proxy).

3. 3

   A schedule is attached to the output

   Create a Scheduler schedule that targets the trusted output dataset so it rebuilds automatically. The schedule associates the build with the trusted dataset; confirm the association with Get Dataset Schedules (datasets v2), which returns the schedule(s) attached to the dataset. This step verifies only that a schedule exists and points at the trusted output — the existence of the schedule association — not what trigger it uses or how it is configured; the trigger internals are covered and attested in the next step. Make sure exactly the trusted backbone dataset is the schedule's target so downstream consumers refresh from the right output.

   not startedinstance check

   Run check

   Confirms a Scheduler schedule targets the trusted dataset (association only, not trigger config).

4. 4

   Configure the trigger and re-trigger-on-success

   Configure the schedule's trigger and confirm the refresh behavior matches the backbone's needs. Choose the trigger type — time-based for a fixed cadence, event-driven to build when new upstream data lands, or manual — and, if the backbone must drain a backlog of input transactions, enable 're-trigger upon successful build', which keeps building until all available inputs are processed and requires a target using incremental transaction limits (for datasets) or incremental batch limits (for media sets), which is why this depends on the incremental work done earlier. This step is self-attested and instructor-checked: trigger type and re-trigger-on-success are schedule configuration internals that no documented read API exposes beyond the schedule's existence. Record the trigger type and whether re-trigger-on-success is enabled, and the cadence or upstream event you tied it to.

   not startedself-attested

   Mark complete

   Self-attested: trigger type + re-trigger-on-successful-build are set (trigger internals are not read-confirmable beyond schedule existence).