Curriculum / Foundry Data Engineering
Advanced data shapes — media sets and time series
Introduction
Scenario: The disruption-response backbone now carries clean, joined, quality-guarded structured feeds, but the team still cannot answer two operational questions: what do the scanned customs and bill-of-lading documents actually say, and how is the cold-chain temperature on a delayed reefer container trending right now. This unit broadens the trusted backbone to the two data shapes the disruption scenario needs but structured tables cannot hold — unstructured shipping documents in a media set and IoT shipment-sensor telemetry modeled as time series.
Up to this point the supply-chain backbone has been rectangular: rows and columns landed by Data Connection, joined in Pipeline Builder, enriched with a PySpark transform, and guarded by data expectations and Data Health. Real disruption data is not all rectangular. Carriers send scanned bills of lading, customs forms, and inspection photos; reefer containers and trucks emit continuous temperature, humidity, and GPS readings. Foundry models these as two distinct first-class shapes. A media set is a collection of media files that share a common schema type and a primary format, and it lets other datasets carry media references that point at items without copying the bytes — so a structured shipments dataset can link directly to the scanned document for each shipment. Time series, by contrast, attaches timestamped measurements to objects: a time series property (TSP) hangs a stream of (timestamp, value) points off an object, and a sensor object type gives each sensor its own series while linking back to a root object type, backed by a dataset keyed on a series ID.
The engineering reality is that both shapes straddle the dataset layer and the Ontology/metadata layer, which changes what you can prove with a read API and what you must attest by hand. The pieces that are confirmable are the ones that live as datasets: the time-series backing dataset must be physically shaped with a series-identifier column, a timestamp, and a value before a TSP can ever bind to it, and that schema is readable through the GA Get Dataset Schema endpoint. The pieces that live above the dataset — the media set's internal contents, and the sensor object type / TSP configuration (series-ID mapping, units, interpolation) — are configured in tooling whose state no GA dataset read exposes. Media-set existence can be probed, but only through media-sets v2 read endpoints that are still in PREVIEW.
By the end of this unit you will have produced a media set populated with scanned shipping documents (with media references usable from your structured datasets) and a time-series-backed dataset whose schema is ready to feed a sensor object type / TSP for shipment telemetry. Treat the unit as two parallel tracks that each end in an Ontology-facing handoff: the document track ends with importable, referenceable media, and the telemetry track ends with a correctly shaped series dataset wired to a sensor object type. Be deliberate about which claims you can machine-verify and which you must honestly self-attest, because the next unit's capstone assumes both shapes are real, not just declared.
Capability focus: Media sets (unstructured docs / media references); time series (TSP / sensor object type) over sensor telemetry. · Artifact: A media set populated with scanned shipping documents plus a time-series-backed dataset feeding a sensor object type.
Key concepts
- Media sets: A media set is a collection of media files that share a common schema type and a single primary format (for example PDF or JPEG). It is the Foundry construct for unstructured content such as scanned bills of lading, customs forms, and inspection photos, and it is populated by importing files or via a Data Connection media set sync.
- Media references: A media reference lets a structured dataset point at an item inside a media set without copying the file's bytes into the dataset. This is how a row in the shipments dataset can carry a pointer to its scanned document, keeping the structured table small while remaining linkable to the unstructured source.
- Time series and TSP: A time series property (TSP) attaches a stream of timestamped measurements to an object type, so an object such as a container can expose its temperature or location history. Time series produced this way are consumable downstream in Workshop, Quiver, and Functions.
- Sensor object type: A sensor object type gives each sensor object its own time series and links back to a root object type (for example each temperature sensor links to its container). It requires a backing dataset that carries a series ID so each row's measurement can be associated with the correct series.
- Time-series backing dataset schema: Before any TSP or sensor object type can bind, the backing dataset must be physically shaped with a series-identifier column, a timestamp column, and a value column. This dataset-level contract is the part of the time-series setup that a read API can confirm.
- Transactions and schema reads: Foundry datasets are versioned by transactions, and the GA Datasets v2 read surface includes Get Dataset Schema (GET /api/v2/datasets/{rid}/schema), which returns the dataset's columns and types — the mechanism used here to verify the telemetry backing dataset is correctly shaped.
- Preview read endpoints: Media-sets v2 read endpoints such as Get Media Set and Get Media Item Info are in PREVIEW and require preview=true. They can confirm a media set resolves but may change, and against a live tenant may return blocked rather than a silent pass — so machine confirmation of media is best-effort and explicitly flagged.
Companion video
Media sets & time series walkthrough (placeholder) · open on YouTube
Hands-on activity
each step validates · the unit completes when all steps pass- 1
Create and populate the media set with documents
Create the media set for the scanned shipping documents, choosing the correct schema type and a single primary format that matches what carriers send (for example PDF for bills of lading and customs forms), then import the documents so the set is populated. A media set holds files sharing that common schema type and primary format, and once populated it can expose media references that a structured dataset — such as your shipments table — points at without copying the underlying bytes. This step is self-attested: media-sets v2 read endpoints are in PREVIEW, and the act of creating the set, picking its format, and importing files happens in Foundry tooling whose internals no GA read API exposes, so honestly confirm that the set exists with the right format and that the expected documents landed rather than expecting a machine check to vouch for the contents.
not startedself-attestedSelf-attested: the media set is created and shipping documents imported (media-sets reads are preview; creation is instructor-checked).
- 2
Media set existence (best-effort, preview)
Confirm the populated media set actually resolves at its expected RID by probing the media-sets v2 read surface (Get Media Set / Get Media Item Info). This is an existence-and-reachability check only: a successful resolve tells you the set is present and returns its metadata or item info, not that every intended document was imported correctly. Treat the result as best-effort and PREVIEW — these endpoints require preview=true and are subject to change, and against a live tenant the read may return blocked rather than silently passing, so a blocked result means unverified, not failed. Pair this proxy with the self-attestation from the previous step rather than relying on it alone to prove the document track is complete.
not startedinstance checkConfirms the media set resolves (existence-only; preview → blocked when unreachable).
- 3
Time-series backing dataset has the required series schema
Shape the time-series backing dataset so it can physically back a TSP and sensor object type, then verify its schema. The dataset must carry a series-identifier column (series_id) so each row associates with the right sensor stream, a timestamp column marking when the measurement was taken, and a value column holding the reading (for example container temperature). This is the machine-confirmable part of the time-series track: the GA Get Dataset Schema endpoint reads the dataset's columns and types directly, so the check confirms series_id, timestamp, and value are present and the dataset is correctly shaped to back a series. Make sure the grain is one row per (series_id, timestamp) — a backing dataset that fans out to multiple values per series per timestamp will break the series contract even though the schema columns still appear present.
not startedinstance checkConfirms the dataset is shaped to back a time series (series-ID + timestamp + value columns).
- 4
Sensor object type / TSP configured on the Ontology
Configure the sensor object type / TSP on the Ontology so the shaped backing dataset becomes a usable time series for the disruption response. This means mapping the series ID to the backing dataset's series_id column, declaring units for the value (for example degrees Celsius), setting interpolation behavior, and linking the sensor object type back to its root object type (for example each sensor to its container) so each object exposes its own series. This step is self-attested: time-series metadata configuration lives in the Ontology / time-series setup tooling, not in the dataset itself, so no dataset-level read here can confirm the mapping, units, or interpolation. Honestly attest that the sensor object type / TSP is configured and that the series resolves end to end, since the dataset schema check from the prior step proves only that the table is shaped, not that the TSP was bound.
not startedself-attestedSelf-attested: the sensor object type / time-series property is configured (time-series metadata is not a dataset-level read here).