Introduction

Scenario: A supplier-side disruption is rippling through the network, and the response team has no single trustworthy view of orders, suppliers, shipments, and inventory. As the data engineer, you stand up the first link of the trusted backbone: connecting the four raw source feeds into Foundry so every downstream join, score, and alert has real data to stand on.

This unit is the foundation of the entire disruption-response backbone: getting raw supply-chain data out of external systems and into Foundry as datasets you can build on. Nothing downstream — no join, no risk score, no Data Health check, no Ontology — exists until the source feeds are landing reliably. Data Connection is the capability that does this: it synchronizes data from external systems into Foundry for use in data integration, modeling, and the Ontology, and the syncs you configure produce the Foundry datasets that the rest of the course transforms.

The mental model you need is the three-layer split that trips up most newcomers. A source represents a single external system (a relational database, an SFTP server, a cloud bucket) together with the connection details and credentials used to reach it; a sync is the configured job that actually moves data from that source into a specific Foundry dataset. You author one source, then attach multiple syncs to it — here, one batch sync per feed landing raw orders, suppliers, shipments, and inventory. Foundry offers out-of-the-box connectors for common system types (relational databases, FTPS, HDFS, S3, SFTP, local directories) plus a generic connector for arbitrary systems, and supports several sync styles — batch/standard, file-based, streaming, media set, and CDC. For this unit you use plain batch/file syncs.

A second distinction matters for how the work runs: a source can use the Foundry-managed runtime, where Foundry hosts the connection directly, or an agent-based runtime, where a lightweight agent runs inside the source network to reach systems Foundry cannot connect to directly (for example, on-prem databases behind a firewall). The choice does not change the datasets that land, but it changes how you authenticate and where the connection lives. Your deliverable for this unit is concrete: a working Data Connection source plus four batch syncs that have actually committed data, leaving raw_orders, raw_suppliers, raw_shipments, and raw_inventory sitting in the training Project with real schemas and real rows.

Capability focus: Data Connection source vs connection vs sync; Foundry-managed vs agent-based runtime; batch/file syncs landing raw datasets. · Artifact: A Data Connection source plus 4 batch syncs landing raw orders, suppliers, shipments, and inventory datasets.

Key concepts

Companion video

1. 1

   Create the source and authenticate the connection

   Create the Data Connection source for the external system that holds the supply-chain feeds — pick whichever fits the training environment (a relational database, an SFTP server, or a cloud bucket like S3) and use the matching out-of-the-box connector. Decide between the Foundry-managed runtime and an agent-based runtime based on whether Foundry can reach the system directly: if the source sits on-prem behind a firewall, you will need the agent-based runtime, otherwise Foundry-managed is simpler. Enter the connection details and authenticate so the source shows a healthy, connected status. This step is self-attested: source configuration and credentials are deliberately not exposed by any documented Foundry read API, so the automated check cannot confirm it. What the platform can later confirm are the output datasets your syncs produce — so make sure the connection genuinely authenticates here before moving on, because every following step depends on it.

   not startedself-attested

   Mark complete

   Self-attested: a Data Connection source is created and the connection authenticates (config/credentials are not API-readable).

2. 2

   Each raw feed lands as a dataset

   Attach one batch (or file-based) sync per feed to the source you just created, targeting four distinct Foundry datasets in the training Project: raw_orders, raw_suppliers, raw_shipments, and raw_inventory. Configure each sync to pull the corresponding source table or file and write it as a snapshot into its dataset, then run all four. The check for this step calls Get Dataset (datasets v2, GA) on each of the four expected dataset RIDs/paths and confirms each resolves — returning its rid, name, and parentFolderRid. That proves the synced output datasets exist in the Project as real resources, not just sync definitions. Use the exact dataset names above so the four feeds map cleanly onto the join you build in the next unit.

   not startedinstance check

   Run check

   Confirms all four raw feeds resolve as datasets with a non-empty schema.

3. 3

   Raw datasets carry the expected ingest schema

   Confirm that ingestion wrote a real, typed schema and not an empty placeholder. The check runs Get Dataset Schema (GET /api/v2/datasets/{rid}/schema, GA) on raw_shipments and verifies the expected source columns are present — at minimum shipment_id, order_id, and status. Because a batch sync infers and writes column types as it lands data, a populated schema is direct evidence that the sync read the source structure correctly; an empty or missing schema usually means the sync ran against the wrong table, an empty file, or a mis-mapped column set. If the schema is missing those columns, fix the sync's source selection or column mapping and re-run before continuing — the join and risk score downstream reference these exact fields, so getting the shipments shape right here prevents silent failures later.

   not startedinstance check

   Run check

   Confirms the shipments sync wrote a typed schema with the expected columns.

4. 4

   Sync actually committed data (a build ran)

   Prove a sync actually ran and committed data, rather than just creating an empty dataset. The check calls List Transactions (datasets v2, GA) on raw_orders and confirms at least one COMMITTED transaction exists. In Foundry, every successful write to a dataset is recorded as a committed transaction in its history, so a COMMITTED transaction is the read-confirmable signal that rows landed — a dataset can exist with a schema yet still hold zero committed writes if the sync was defined but never executed successfully. If no committed transaction appears, trigger the raw_orders sync, watch it complete without error, and verify the transaction is recorded. With the source connected, four datasets present, the shipments schema typed, and orders showing a committed write, the raw feeds are in Foundry and the trusted backbone has its first link.

   not startedinstance check

   Run check

   Confirms a sync ran and committed a transaction (not just an empty dataset).