Curriculum / Data Engineer
Foundry Data Engineering
Build the trusted data backbone behind the supply-chain disruption scenario: Data Connection sources, Pipeline Builder (no-code) and Code Repositories (PySpark), data expectations + Data Health, incremental computation + scheduling, branch-based DevOps, plus media sets and time series for ISR/sensor data — a fresh, joined, enriched, quality-guarded dataset the Ontology and analyst teams consume.
Data engineers feeding the Ontology (coding-capable — PySpark + no-code).
The 8-unit arc
- Unit 1: Connect the source feeds (Data Connection)foundational
Configure a Data Connection source and batch syncs to ingest the raw supply-chain feeds into Foundry as datasets.
- Unit 2: Shape and join in Pipeline Builder (no-code transforms)intermediate
Clean, transform, and join the raw feeds into a single trusted joined dataset at the correct grain — all no-code.
- Unit 3: Custom logic in a Code Repository (PySpark transform)intermediate
Author a PySpark transform for logic beyond no-code — a supplier risk/exposure score — and build it to a derived dataset.
- Unit 4: Guard the data with expectations and Data Healthintermediate
Apply data expectations (primary key, row count) and configure Data Health checks so quality regressions are caught and surfaced.
- Unit 5: Make it incremental and schedule the buildsadvanced
Convert the transform to incremental computation and create a Scheduler schedule so the trusted backbone refreshes automatically and cheaply.
- Unit 6: Branch-based development and DevOps releaseadvanced
Develop pipeline changes on a branch, validate, and promote to default via propose/approve; package for release with Foundry DevOps.
- Unit 7: Advanced data shapes — media sets and time seriesadvanced
Ingest unstructured shipping documents into a media set and model IoT shipment-sensor telemetry as time series.
- Unit 8: Capstone — ship the trusted backbone for the disruption responsecapstone
Integrate everything into a fresh, joined, enriched, quality-guarded, incrementally-built, branch-promoted trusted dataset (plus media + time series) ready for the Ontology.
Connect the source feeds (Data Connection)
Configure a Data Connection source and batch syncs to ingest the raw supply-chain feeds into Foundry as datasets.
Shape and join in Pipeline Builder (no-code transforms)
Clean, transform, and join the raw feeds into a single trusted joined dataset at the correct grain — all no-code.
Custom logic in a Code Repository (PySpark transform)
Author a PySpark transform for logic beyond no-code — a supplier risk/exposure score — and build it to a derived dataset.
Guard the data with expectations and Data Health
Apply data expectations (primary key, row count) and configure Data Health checks so quality regressions are caught and surfaced.
Make it incremental and schedule the builds
Convert the transform to incremental computation and create a Scheduler schedule so the trusted backbone refreshes automatically and cheaply.
Branch-based development and DevOps release
Develop pipeline changes on a branch, validate, and promote to default via propose/approve; package for release with Foundry DevOps.
Advanced data shapes — media sets and time series
Ingest unstructured shipping documents into a media set and model IoT shipment-sensor telemetry as time series.
Capstone — ship the trusted backbone for the disruption response
Integrate everything into a fresh, joined, enriched, quality-guarded, incrementally-built, branch-promoted trusted dataset (plus media + time series) ready for the Ontology.