Introduction

Scenario: The trusted joined supply-chain dataset is landing, but the disruption-response team cannot yet see which suppliers are most exposed; you need a computed supplier risk score that no-code transforms cannot express cleanly. As the data engineer, you drop into a Code Repository and author a PySpark transform that enriches the backbone with a defensible risk_score column.

Pipeline Builder carried the supply-chain backbone from raw feeds to a single trusted, joined dataset, but scoring supplier exposure is the kind of logic that outgrows a no-code canvas: it blends several signals (late-shipment rate, inventory cover, single-source concentration, lead-time variance) into one bounded number with branching rules and a documented contract. In Foundry, that logic belongs in a Code Repository, where you author Python transforms against the transforms-python API. This unit has you write a @transform_df function that reads the trusted joined dataset and writes a derived dataset, supplier_risk_scored, carrying a computed risk_score column the rest of the response workflow can rank on.

The execution model is PySpark over a Spark backend. DataFrames are immutable and lazily evaluated, so each step you write describes a transformation that Spark only materializes when the build runs; you build up the risk score by selecting and deriving columns rather than mutating rows in place. A select statement at the start and end of the transform documents the schema contract, making it explicit which columns enter and which leave, including the new risk_score that did not exist on the input. This is the discipline that keeps a derived dataset readable and reviewable months later, when someone downstream asks why a supplier scored 82.

Most of this work lives inside source files and transform definitions that Foundry does not expose through a documented read API, so the checks here verify the build's result rather than the code itself. After you commit and build, the platform confirms that supplier_risk_scored exists, that its schema gained the risk_score column absent from the input, and that the computed values actually fall in the intended [0,100] domain. You attest to the code; the read-side checks prove the transform ran and emitted sane values, which together establish that custom logic now enriches the trusted backbone.

Capability focus: Code Repositories; transforms-python API (@transform_df); DataFrame patterns; derived/computed columns. · Artifact: A Code Repository PySpark transform that reads the trusted joined dataset and writes supplier_risk_scored with a computed risk_score column.

Key concepts

Companion video

1. 1

   Write and commit the PySpark transform code

   In a Code Repository, author a Python transform using the transforms-python API: a @transform_df function that takes the trusted joined dataset as input and returns a DataFrame written to supplier_risk_scored. Inside it, derive risk_score from existing supply-chain signals (for example, late-shipment rate, inventory cover, and single-source concentration), keeping the value bounded to [0,100], and use a select at the start and end to document the schema contract — which columns you read and that risk_score is the new field you emit. Then commit so the transform definition is versioned in the repository. This step is self-attested: Foundry does not expose repository source code or transform definitions through a documented read API in this DSL, so no automated check can read your code — the later checks instead verify the dataset your build produces. State honestly whether the committed @transform_df with the select-contract and bounded scoring logic is in place; the build's output is what is confirmable, not the source.

   not startedself-attested

   Mark complete

   Self-attested: the @transform_df scoring transform is committed (Code Repository source is not API-readable).

2. 2

   Derived dataset exists after build

   Build the transform so it produces its output. Because DataFrames are lazily evaluated, none of your scoring logic actually runs until a build materializes supplier_risk_scored; the build executes the Spark plan and writes a transaction to the output dataset. The check here calls Get Dataset (a GA read endpoint) on the expected derived-output dataset and confirms it resolves to a real resource, which proves the transform was built and produced an output rather than only existing as committed code. If the dataset does not resolve, the transform has not been built successfully — rerun the build and resolve any errors before moving on, since the remaining checks all read from this output.

   not startedinstance check

   Run check

   Confirms the transform built and produced the enriched output dataset.

3. 3

   Derived schema includes the new computed column

   Confirm the transform emitted the new field, not just any output. The check calls Get Dataset Schema (GA) on supplier_risk_scored and verifies the schema includes the risk_score column. Because risk_score is a derived column computed by your PySpark logic, it is present in the output schema but absent from the trusted joined input — so its appearance in the schema is direct read-side evidence that your scoring code ran and the end-of-transform select carried it through. If risk_score is missing, the column was dropped or misnamed in the final select; align the emitted column name with the contract you documented and rebuild.

   not startedinstance check

   Run check

   Confirms the PySpark logic emitted the new risk_score column (absent from the input).

4. 4

   Computed values fall in the expected domain

   Prove the computed values are sane, not merely present. The check uses Read Table Dataset (GA) to sample rows from supplier_risk_scored and asserts that every sampled risk_score falls within [0,100] — verifying the transform's logic, not just the column's existence. This catches scoring bugs that a schema check cannot: an unbounded sum, a divide-by-zero producing nulls or infinities, or a percentage that overflows past 100. Make the bounding explicit in your PySpark (clamp or normalize the blended signals) so the domain holds for every supplier, including edge cases like a brand-new supplier with no shipment history. A passing range check is the read-side confirmation that custom risk logic now enriches the trusted backbone correctly.

   not startedinstance check

   Run check

   Confirms every sampled risk_score lies in [0,100] — the transform's computed values, not just column existence.