Suites and cases

A suite is a system prompt and shared setup for many cases. Each case is one eval: it fills in the prompt, sets up the workspace and checks the result.

import { command, repo, suite } from "sphynx-sh";

export default suite({
  id: "api",
  source: repo("acme/api@8f31c4a"),
  prompt: "{{task}}",
  cases: [
    {
      id: "health-route",
      variables: { task: "Add a /health route that returns 200" },
      validate: command("bun test test/health.test.ts"),
    },
  ],
  variants: [{ harness: "codex", model: "model-id" }],
  trials: 3,
});

Running this file starts a batch: one run per case per variant, each with trials trials in their own sandboxes. A suite and a case each need an id: a handle of up to 100 lowercase letters, digits and single hyphens, such as health-route. A case id is unique within your organization. Its history is kept under it, so keep it stable. When a case’s definition changes, later runs record a new version of the case.

Case field	Purpose
`id`	Stable handle the case’s history is kept under (required)
`name`	Label for the case’s row in the grid. Defaults to the id
`variables`	Values for prompt variables such as `{{task}}`
`source`	Workspace for this case, overriding the suite’s
`prepare`	Typed setup that runs before the agent
`cache`	One prepared directory to restore and save
`tags`	Up to 12 labels to group and filter cases by
`user`	Makes the case a conversation instead of one prompt
`maxTurns`	Most turns a conversation may hold. Defaults to 8
`timeoutMs`	How long the agent may work across the whole trial. Defaults to 15 minutes
`validate`	How the trial passes: a validator, a model judge, `command("...")`, or an array of validators and judges (required)

The suite’s name also defaults to its id.

Sources

source is the workspace the agent starts in. Leave it out for an empty workspace.

import { files, repo } from "sphynx-sh";

source: repo("acme/api@8f31c4a");
source: files({ "src/add.ts": "export const add = () => 0;" });

repo() accepts owner/repo, owner/repo@ref or a clone URL. A case’s source overrides the suite’s.

Prepare

prepare receives typed filesystem and process helpers. Whatever it returns reaches the validator as prepared.

import type { Prepare } from "sphynx-sh";

export const install: Prepare = async ({ cached, exec }) => {
  if (!cached) await exec("npm", ["ci"], { timeoutMs: 600_000 });
  return { port: 4173 };
};

{
  cache: { key: "npm-lock-v1", path: "node_modules" },
  prepare: install,
}

cached is true when the cache directory was restored. The cache path must stay inside the workspace. prepare can return a key it creates, such as a test API key. Sphynx hands the value to your validator and never prints it in logs or the terminal or stores it as evidence. Output the prepare prints itself is logged while it runs, with anything that looks like a key replaced by [redacted].

Limits

timeoutMs is how long the agent may work across all of a trial’s turns, from 1,000 to 3,600,000 milliseconds. It defaults to 15 minutes. Time the person spends replying does not count. When the agent runs past it, the harness is stopped and the trial ends as timed out, never as a pass or a fail. maxTurns caps how many times the person speaks in a conversation, the opening prompt included. It takes 1 to 50 and defaults to 8.

{
  id: "long-migration",
  maxTurns: 4,
  timeoutMs: 1_800_000,
  validate: migrated,
}

Changing either limit creates a new version of the case.

Validate

command("bun test") runs a shell command in the workspace after the agent finishes. The trial passes when it exits 0. Use a validator when the check is clearer in TypeScript than in a shell command:

import type { Validator } from "sphynx-sh";

export const hasHealthRoute: Validator = async ({ readText }) => ({
  message: "the health route exists",
  passed: (await readText("src/routes.ts")).includes("/health"),
});

Set validate: hasHealthRoute on the case. Sphynx bundles the function and its imports with the case. A validator can also return a plain boolean. The validator receives answer, transcript, turns, readText, exists, exec, prepared, and the call logs of any mocks (api, cli, mcp). Every check of a trial receives the same context object, so checks can share work, such as one request to your app, by caching on it. answer() is the agent’s last message and transcript() is every agent message joined. turns() is the whole conversation, turn by turn, with the messages, commands, tool calls and file changes of each. A case without user has one turn. See what a validator reads. Pass an array to combine validators and judges, up to 20 per case: validate: [hasHealthRoute, correctness]. A command() can’t go in an array. Use model judges for checks that need interpretation. Every code validator runs, even after an earlier one fails, so each one records its own result and pass rate. The trial passes only when all of them pass.

Source capture

The dashboard’s Validation panel shows the validator source as you wrote it, captured at compile time. Each run keeps its own snapshot. Only TypeScript and JavaScript files under the eval’s nearest package.json directory are captured. Dependencies and files outside that directory are not. Keep secrets out of those files, or set captureSource: false on the suite.

Run

sphynx eval runs every *.eval.ts file it finds. Pass a path to run one: sphynx eval ./smoke.eval.ts. Add --case <id> to run one case from the file. See variants for harnesses, models and sandboxes, and profiles for custom harness configuration.

Getting started

Define evals

Run evals

Suites and cases

Sources

Prepare

Limits

Validate

Source capture

Run

​Sources

​Prepare

​Limits

​Validate

​Source capture

​Run

Sources

Prepare

Limits

Validate

Source capture

Run