Introduction

Scenario: Before the supply-chain disruption-response recommendation is released to the operations team, leadership wants evidence — not a vibe — that recommendDisruptionResponse gives trustworthy answers across the kinds of disruptions it will face. You build an AIP Evals suite that scores the function on representative disruption cases so the team can release with confidence and compare versions over time.

Unit 5 shifts you from building the recommendation to proving it. You already have a published AIP Logic function, recommendDisruptionResponse, that takes a DisruptionEvent and returns a structured recommendation. The open question for a decision-support tool that operators will act on is whether its answers hold up across the range of disruptions it will see — a single late shipment, a sole-source supplier outage, a customs hold — not just the one case you happened to test by hand. AIP Evals is the testing environment that answers this: it lets you evaluate an AIP Logic function, an AIP Chatbot function, or a code-authored function against a curated set of cases and read the results as metrics, so 'it worked when I tried it' becomes a repeatable, comparable measurement.

An AIP Evals suite has four moving parts that map directly to today's work. A target function is the thing under test — here, the published recommendDisruptionResponse. Test cases pair an input (a seeded DisruptionEvent) with an expected output (the recommendation you would accept as correct). Evaluators are evaluation functions that score each run against its expected output; AIP Evals ships built-ins such as Exact string match, and you can author custom evaluators for domain logic. Each evaluator carries a metric objective: either a Boolean objective (true is the pass) or a numeric objective you want to maximize or minimize. Running the suite produces metrics per test case and in aggregate, and those aggregates are comparable between runs — which is how you tell whether a new function version is actually better than the last before you publish it.

Be clear-eyed about what this curriculum can and cannot machine-verify. The recommendation function is published, so its query type can be executed by API and you can confirm it runs on a seeded case and returns the expected output shape — that is the one instance check this unit asserts. Everything inside AIP Evals itself, though — the suite, its test cases, the evaluators and their objectives, and the metrics a run produces — is not exposed by any documented Foundry read API. That is not a gap in your work; it is a property of the platform. So most of this unit is honestly self-attested against the AIP Evals metrics dashboard, and the discipline is to do the evaluation rigorously even though a script cannot read the result back out.

Capability focus: AIP Evals (test cases, evaluators, metric objectives); proving function reliability before release. · Artifact: An evaluation suite over the recommendation function with multiple test cases and a metrics report.

Key concepts

Companion video

1. 1

   Target function executes on each eval input

   AIP Evals targets a function, so the first thing to establish is that the function the suite will score actually runs on the cases you intend to seed. Because recommendDisruptionResponse is published, it has a queryApiName and is addressable by the Execute Query API (POST /api/v2/functions/queries/{queryApiName}/execute, preview=true), which runs the latest published version against a parameters map and returns the result as JSON. Execute it on a representative seeded test case (the capstone disruption, disruptionId DISR-CAP-1) and confirm the returned recommendation carries the expected structured keys — action, rationale, and confidence. This is the only part of the unit a script can verify, and it verifies the target by execution and output shape, never by reading the function's blocks, prompt, or output schema, none of which any read API exposes. Note the contrast with a pre-publish function: had you left recommendDisruptionResponse merely saved, it would have no queryApiName and Execute Query could not address it — which is precisely why the eval target must be the published function.

   not startedinstance check

   Run check

   Executes the published target Logic function on a representative seeded test case and confirms expected keys are present.

2. 2

   Test cases of inputs + expected outputs defined

   Define the suite's test cases, each pairing a disruption input with the recommendation output you would accept as correct. Curate cases that span the disruptions this function will face in production — a single late shipment, a sole-source supplier outage, a customs/regulatory hold, a low-severity blip that should not escalate — so the suite measures behavior across the real distribution rather than one convenient case. For each case, record the input DisruptionEvent and the expected recommendation, choosing expected values you can actually score against (for example a specific recommended action, or a severity tier). This step is self-attested: no documented Foundry read API enumerates AIP Evals test cases (R-EVAL1), so the cases are confirmed only in the AIP Evals dashboard. Build them with the same rigor you would if a grader could read them — the value is in the coverage, not in any external check.

   not startedself-attested

   Mark complete

   Self-attested: the eval suite pairs disruption inputs with expected recommendation outputs.

3. 3

   Evaluator configured with an objective

   Add at least one evaluator and give it a metric objective. Start with a built-in such as Exact string match when the expected output is a discrete value like a recommended action or severity tier; reach for a custom evaluator when correctness needs domain logic the built-ins cannot express (for example, accepting any of several acceptable actions, or scoring a confidence band). Whichever you choose, set its objective explicitly: a Boolean objective where true is the pass, or a numeric objective you want to maximize or minimize. The objective is what turns a raw score into a verdict, so state plainly what 'good enough to release' means for this function. This step is self-attested — evaluator type and objective configuration are not exposed by any read API (R-EVAL1) and are confirmed only in the AIP Evals dashboard.

   not startedself-attested

   Mark complete

   Self-attested: at least one evaluator with a metric objective is added.

4. 4

   Suite run produces a metrics report meeting the objective

   Run the suite and read the metrics report. AIP Evals computes a metric for each test case and presents both per-case results and an aggregate, letting you see not just whether the function passed overall but which specific disruption cases it handled poorly. Confirm the run meets the objective you set, and treat any failing case as a signal to revise the function — then re-run, because the aggregate metrics are comparable between runs and are exactly how you justify that a new version is better than the last before you publish it. This step is self-attested via the AIP Evals dashboard: run results and metrics are not retrievable through any documented read API (R-EVAL1). Capture a screenshot or written record of the per-case and aggregate metrics as your release evidence, since that dashboard view is the only place the result lives.

   not startedself-attested

   Mark complete

   Self-attested via the AIP Evals dashboard: the suite run meets its objective.