Skip to content

Deploy and operate

Deterministic testing

@yielded/agent-testing replaces the model provider with scripted responses while retaining the real interpreter, tools, and output validation. Use it to verify integration behavior without credentials, network calls, or model variance.

import {
class ScriptedModel

Inspection service for a deterministic LanguageModel backed by finite scripted turns.

ScriptedModel
} from "@yielded/agent-testing/scripted-model";
import {
import Model
Model
} from "effect/ai";
const
const TestModel: Model.Model<"scripted", ScriptedModel | LanguageModel, never>
TestModel
=
import Model
Model
.
const make: <"scripted", "test-model", ScriptedModel | LanguageModel, never>(provider: "scripted", modelName: "test-model", layer: Layer<ScriptedModel | LanguageModel, never, never>) => Model.Model<"scripted", ScriptedModel | LanguageModel, never>

Creates a Model from a provider name and a Layer that constructs AI services.

Example (Providing model metadata)

import { Effect, Layer } from "effect"
import { Model } from "effect/ai"
const model = Model.make("amazon-bedrock", "claude-3-5-haiku", Layer.empty)
const program = Effect.gen(function*() {
const provider = yield* Model.ProviderName
const modelName = yield* Model.ModelName
return { provider, modelName }
}).pipe(Effect.provide(model))
await Effect.runPromise(program) // => { provider: "amazon-bedrock", modelName: "claude-3-5-haiku" }

@stability ― unstable

@category ― constructors

@since ― 4.0.0

make
(
"scripted",
"test-model",
class ScriptedModel

Inspection service for a deterministic LanguageModel backed by finite scripted turns.

ScriptedModel
.
ScriptedModel.layer(turns: ReadonlyArray<ScriptedTurnInput>): Layer<LanguageModel | ScriptedModel, never, never>

Provides the native Effect AI LanguageModel and this inspection service. Supplying the extra inspection service does not add it to model-call requirements. Each model invocation consumes one turn before assertion and turn-kind validation.

layer
([
{
_tag: "Stream"
_tag
: "Stream",
parts: readonly (StreamPartEncoded | {
readonly type: "tool-call";
readonly id: string;
readonly name: string;
readonly params: unknown;
readonly providerExecuted?: boolean | undefined;
readonly metadata?: {
readonly [x: string]: Json;
} | undefined;
} | {
readonly type: "tool-result";
readonly id: string;
readonly name: string;
readonly result: unknown;
readonly isFailure: boolean;
readonly providerExecuted?: boolean | undefined;
readonly metadata?: {
readonly [x: string]: Json;
} | undefined;
readonly preliminary?: boolean | undefined;
})[]
parts
: [
{
BasePartEncoded<"text-start", TextStartPartMetadata>.type: "text-start"

The type of this response part.

type
: "text-start",
TextStartPartEncoded.id: string

Unique identifier for this text chunk.

id
: "answer" },
{
BasePartEncoded<"text-delta", TextDeltaPartMetadata>.type: "text-delta"

The type of this response part.

type
: "text-delta",
TextDeltaPartEncoded.id: string

Unique identifier matching the corresponding text chunk.

id
: "answer",
TextDeltaPartEncoded.delta: string

The incremental text content to add.

delta
: "ready" },
{
BasePartEncoded<"text-end", TextEndPartMetadata>.type: "text-end"

The type of this response part.

type
: "text-end",
TextEndPartEncoded.id: string

Unique identifier matching the corresponding text chunk.

id
: "answer" },
{
BasePartEncoded<"finish", FinishPartMetadata>.type: "finish"

The type of this response part.

type
: "finish",
FinishPartEncoded.reason: "error" | "length" | "stop" | "content-filter" | "tool-calls" | "pause" | "other" | "unknown"

The reason why the model finished generating the response.

reason
: "stop",
FinishPartEncoded.usage: {
readonly inputTokens: {
readonly uncached?: number | undefined;
readonly total?: number | undefined;
readonly cacheRead?: number | undefined;
readonly cacheWrite?: number | undefined;
};
readonly outputTokens: {
readonly total?: number | undefined;
readonly text?: number | undefined;
readonly reasoning?: number | undefined;
};
}

Token usage statistics for the request.

usage
: {
inputTokens: {
readonly uncached?: number | undefined;
readonly total?: number | undefined;
readonly cacheRead?: number | undefined;
readonly cacheWrite?: number | undefined;
}

Information about input (i.e. prompt) token utilization.

inputTokens
: {
total?: number | undefined

The total of number of input (i.e. prompt) tokens used.

total
: 1 },
outputTokens: {
readonly total?: number | undefined;
readonly text?: number | undefined;
readonly reasoning?: number | undefined;
}

Information about the output (i.e. response) tokens used.

outputTokens
: {
total?: number | undefined

The total of number of output (i.e. response) tokens used.

total
: 1 } },
},
],
termination: {
readonly _tag: "Complete";
} | {
readonly _tag: "Fail";
readonly description: string;
} | {
readonly _tag: "Hang";
}
termination
: {
_tag: "Complete"
_tag
: "Complete" },
},
]),
);

Provide TestModel in place of the provider Layer for a text-output Agent. Keep the application’s runtime and handler composition. Each scripted Turn is consumed once; inspection exposes requests, remaining turns, and assertExhausted. Hooks can inspect the normalized prompt or observe stream finalization. Scripts can also emit tool calls, fail, or wait for interruption.

Choose assertions for your integration’s actual risks: authorized tools, expected output, resource cleanup, or the outcome after restart. A scripted model proves how the runtime handles the supplied response; it does not prove that a real model will choose that response.

The testing package includes executable travel-planner and researcher fixtures. Use live evaluation separately when the question is model quality.

Run the shared storage-port cases against disposable storage to check each adapter’s contract. The @yielded/agent/testing/*-store-conformance modules and @yielded/agent/testing/submission-ledger-conformance expose those cases independently.

For coordinator recovery, run durableCoordinatorConformance from @yielded/agent-testing/durable-coordinator-conformance in a TestClock-backed environment such as it.effect. Provide one combined Layer containing SubmissionLedger, ThreadStore, SettlementPublisher, StorageOwner, and Digest from effect/crypto/Digest. Use pristine disposable storage with native Thread verification. The helper discovers six scenario paths, injects faults, and checks bounded recovery through settled history; violations fail with DurableCoordinatorConformanceViolation. See the coordinator conformance module for its contract.

Injected faults do not establish recovery after process loss or eviction. Exercise actual restarts separately against the host and storage you deploy. Likewise, the reusable @yielded/agent-testing/code-executor-conformance checks executor behavior and declared metadata; verify isolation against the actual executor.