Skip to content
Agile Defense

Curriculum / Foundry Data Engineering

ADVANCEDUnit 6unit-06

Branch-based development and DevOps release

not started~300 min
Develop pipeline changes on a branch, validate, and promote to default via propose/approve; package for release with Foundry DevOps.

Introduction

Scenario: The trusted supply-chain backbone is now live and feeding the disruption response, which means you can no longer edit it in place — a bad change would break every downstream consumer at once. You need to evolve the pipeline safely on a branch, get the change reviewed and promoted to the default branch, and define how the whole workflow ships from test to production.

Until now you have built the backbone by editing the pipeline directly. Once a dataset is load-bearing for an active disruption response, that habit becomes dangerous: an in-place change is live the moment you save it, with no review and no isolation. This unit moves you to a branch-based way of working that mirrors GitFlow. Foundry's recommended branching and release process uses a protected master (production) branch, a dev (staging) branch, and short-lived feature branches off of dev; the formal release branch is typically skipped. You will create a feature branch on the trusted_supply_chain pipeline, make your change there, and leave the default branch untouched until the change is vetted.

The change itself is small and concrete so the mechanics stay in focus: you will add a derived risk_tier column (a banded label over the existing risk_score) on a branch named dev-risk-score. In Pipeline Builder, promoting that change is not a raw git merge but a propose-change / approve-change workflow — a developer proposes the change, a second team member approves it, and only approved changes deploy to the default branch. Branch protection and fallback branches reinforce that the default branch is never edited casually. This review gate is the data-engineering equivalent of a pull request: it is how a second set of eyes catches a fan-out join or a grain change before it reaches production.

Promotion to the default branch is where the pipeline integrates the change, not the end of the release story. To move a whole data-backed workflow between environments, Foundry DevOps packages the ontology, pipelines, datasets, and data connections as a versioned product with automated dependency management and release-channel-based release management, so you can deploy the same product to test and then to production (and even publish to the Marketplace). In this unit you will both perform the branch-and-promote loop on your own pipeline and write down the DevOps release plan that would carry the trusted backbone from test to prod, distinguishing what a READ API can actually confirm from what only review and packaging configuration can.

Capability focus: GitFlow-style branching; propose/approve review; Foundry DevOps packaging + release channels for test→prod promotion. · Artifact: A development branch with a proposed-and-approved change merged to default, plus a documented DevOps release plan.

Key concepts

  • GitFlow-style branching/release process: The recommended Foundry process uses a protected master (production) branch, a dev (staging) branch, and short-lived feature branches; the dedicated release branch is typically skipped. Feature work happens on a branch so the default branch keeps serving downstream consumers unchanged until a vetted promotion.
  • Pipeline Builder branches (propose/approve): Pipeline Builder branching uses a propose-change / approve-change workflow — a developer works on a branch and proposes a change, another team member approves it, and only approved changes deploy to the default branch. Branch protection and fallback branches are available to keep the default branch safe.
  • Default branch vs. feature branch: A non-default (feature) branch isolates in-progress edits from the default branch that production reads. Promotion is what merges the feature branch's logic — and the resulting schema and data — onto the default branch; before promotion the default branch's output is unchanged.
  • Foundry DevOps: Foundry DevOps packages data-backed workflows (ontology, AI models, pipelines, data connections) as products with automated version and dependency management, and uses release-channel-based release management to deploy to test and production environments, including the Marketplace. It is the mechanism for promoting a whole workflow across environments, distinct from branch promotion inside a single pipeline.
  • Release channels: A release channel governs which version of a packaged DevOps product an environment receives, enabling controlled test→prod promotion of the same product rather than rebuilding it per environment.
  • Datasets v2 read endpoints (branch/schema/transactions): List Branches (GA) returns the branches on a dataset, Get Dataset Schema (GA) returns the column schema for a given branch, and List Transactions shows committed transactions — these are the READ APIs that can confirm branch existence and a promoted schema, but cannot see proposal/approval state or DevOps packaging.

Companion video

Branching & Foundry DevOps walkthrough (placeholder) · open on YouTube

Hands-on activity

each step validates · the unit completes when all steps pass
  1. 1

    Create a development branch on the output dataset

    Create a non-default feature branch named dev-risk-score on the trusted_supply_chain pipeline so your work is isolated from the default branch that downstream consumers read. This follows the GitFlow-style model — the default branch stays untouched while you develop on a short-lived feature branch off of dev. The check uses the datasets v2 List Branches endpoint (GA) on trusted_supply_chain and confirms that dev-risk-score exists alongside the default branch; it verifies only that the branch was created, not what the branch contains yet. Create the branch before you make any edits so every change you make lands on the branch and never on the default branch.

    not startedinstance check

    Confirms a non-default dev branch exists for isolated development.

  2. 2

    Propose and approve the change

    On the dev-risk-score branch, make your change — add the derived risk_tier column that bands the existing risk_score into ordered tiers — and then run the Pipeline Builder propose-change / approve-change workflow: propose the change from the branch, and have a second team member approve it so it can deploy to the default branch. This step is MANUAL because proposal and approval are Pipeline Builder review state, internal to the tool, and no documented Foundry dataset READ API exposes whether a change was proposed or approved. Honestly self-attest that the review actually happened with a real second reviewer; do not approve your own change. The review gate is the point of branching — it is where a reviewer catches grain or join regressions before they reach production, and the platform cannot confirm that judgment for you.

    not startedself-attested

    Self-attested: the propose-change / approve-change review workflow was followed (proposal state is not API-exposed).

  3. 3

    Merged change reflected on the default branch

    After the approved change deploys to the default branch, confirm the promotion landed by inspecting the default branch's output. The check uses Get Dataset Schema (GA) on trusted_supply_chain (default branch) and confirms the schema now includes the risk_tier column you added on dev-risk-score; List Transactions should also show a newly committed transaction on default. Together these confirm the change was promoted off the branch and is now live on the default branch that downstream consumers read. If risk_tier is present on dev-risk-score but absent on default, the change was not actually promoted — the schema read is reporting the default branch's real state, so re-check that approval completed and the build ran on default.

    not startedinstance check

    Confirms the promoted change: the default-branch output schema now includes the added column.

  4. 4

    Package the workflow with Foundry DevOps for release

    Write the Foundry DevOps release plan that would carry the trusted backbone from test to production: identify the product to package (the pipeline plus its datasets, and any data connections and ontology objects it depends on), define the release channels for test and prod, and describe how version and dependency management promotes the same product across environments. This step is MANUAL because DevOps product definition and release-channel configuration live in the DevOps service and are not exposed by a documented Foundry dataset READ API in this DSL — only the branch state (List Branches) and the promoted schema (Get Dataset Schema) from earlier steps are read-confirmable. Be explicit in the plan about that boundary: branch-and-promote is what the checks can prove, while packaging and release-channel promotion are self-attested design, not API-verified facts.

    not startedself-attested

    Self-attested: a DevOps product/release plan is defined for test→prod promotion (DevOps package config is not API-exposed).