> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sphynx.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Suites and cases

> Define what the agent does and how it passes

A suite is a system prompt and shared setup for many cases. Each case is one eval: it fills in the prompt, sets up the workspace and checks the result.

```ts theme={null}
import { command, repo, suite } from "sphynx-sh";

export default suite({
  id: "api",
  source: repo("acme/api@8f31c4a"),
  prompt: "{{task}}",
  cases: [
    {
      id: "health-route",
      variables: { task: "Add a /health route that returns 200" },
      validate: command("bun test test/health.test.ts"),
    },
  ],
  variants: [{ harness: "codex", model: "model-id" }],
  trials: 3,
});
```

Running this file starts a batch: one run per case per variant, each with `trials` trials in their own sandboxes.

A suite and a case each need an `id`: a handle of up to 100 lowercase letters, digits and single hyphens, such as `health-route`. A case id is unique within your organization. Its history is kept under it, so keep it stable. When a case's definition changes, later runs record a new version of the case.

| Case field | Purpose |
| - | - |
| `id` | Stable handle the case's history is kept under (required) |
| `name` | Label for the case's row in the grid. Defaults to the id |
| `variables` | Values for prompt variables such as `{{task}}` |
| `source` | Workspace for this case, overriding the suite's |
| `prepare` | Typed setup that runs before the agent |
| `cache` | One prepared directory to restore and save |
| `tags` | Up to 12 labels to group and filter cases by |
| `user` | Makes the case a [conversation](/evals/conversations) instead of one prompt |
| `maxTurns` | Most turns a conversation may hold. Defaults to 8 |
| `timeoutMs` | How long the agent may work across the whole trial. Defaults to 15 minutes |
| `validate` | How the trial passes: a validator, a [model judge](/evals/judges), `command("...")`, or an array of validators and judges (required) |

The suite's `name` also defaults to its `id`.

## Sources

`source` is the workspace the agent starts in. Leave it out for an empty workspace.

```ts theme={null}
import { files, repo } from "sphynx-sh";

source: repo("acme/api@8f31c4a");
source: files({ "src/add.ts": "export const add = () => 0;" });
```

`repo()` accepts `owner/repo`, `owner/repo@ref` or a clone URL. A case's `source` overrides the suite's.

## Prepare

`prepare` receives typed filesystem and process helpers. Whatever it returns reaches the validator as `prepared`.

```ts theme={null}
import type { Prepare } from "sphynx-sh";

export const install: Prepare = async ({ cached, exec }) => {
  if (!cached) await exec("npm", ["ci"], { timeoutMs: 600_000 });
  return { port: 4173 };
};
```

```ts theme={null}
{
  cache: { key: "npm-lock-v1", path: "node_modules" },
  prepare: install,
}
```

`cached` is `true` when the cache directory was restored. The cache path must stay inside the workspace.

`prepare` can return a key it creates, such as a test API key. Sphynx hands the value to your validator and never prints it in logs or the terminal or stores it as evidence. Output the prepare prints itself is logged while it runs, with anything that looks like a key replaced by `[redacted]`.

## Limits

`timeoutMs` is how long the agent may work across all of a trial's turns, from 1,000 to 3,600,000 milliseconds. It defaults to 15 minutes. Time the person spends replying does not count. When the agent runs past it, the harness is stopped and the trial ends as timed out, never as a pass or a fail.

`maxTurns` caps how many times the person speaks in a [conversation](/evals/conversations), the opening prompt included. It takes 1 to 50 and defaults to 8.

```ts theme={null}
{
  id: "long-migration",
  maxTurns: 4,
  timeoutMs: 1_800_000,
  validate: migrated,
}
```

Changing either limit creates a new version of the case.

## Validate

`command("bun test")` runs a shell command in the workspace after the agent finishes. The trial passes when it exits 0.

Use a validator when the check is clearer in TypeScript than in a shell command:

```ts theme={null}
import type { Validator } from "sphynx-sh";

export const hasHealthRoute: Validator = async ({ readText }) => ({
  message: "the health route exists",
  passed: (await readText("src/routes.ts")).includes("/health"),
});
```

Set `validate: hasHealthRoute` on the case. Sphynx bundles the function and its imports with the case. A validator can also return a plain boolean.

The validator receives `answer`, `transcript`, `turns`, `readText`, `exists`, `exec`, `prepared`, and the call logs of any [mocks](/evals/mocks) (`api`, `cli`, `mcp`). Every check of a trial receives the same context object, so checks can share work, such as one request to your app, by caching on it.

`answer()` is the agent's last message and `transcript()` is every agent message joined. `turns()` is the whole conversation, turn by turn, with the messages, commands, tool calls and file changes of each. A case without `user` has one turn. See [what a validator reads](/evals/conversations#what-a-validator-reads).

Pass an array to combine validators and judges, up to 20 per case: `validate: [hasHealthRoute, correctness]`. A `command()` can't go in an array. Use [model judges](/evals/judges) for checks that need interpretation.

Every code validator runs, even after an earlier one fails, so each one records its own result and pass rate. The trial passes only when all of them pass.

### Source capture

The dashboard's **Validation** panel shows the validator source as you wrote it, captured at compile time. Each run keeps its own snapshot. Only TypeScript and JavaScript files under the eval's nearest `package.json` directory are captured. Dependencies and files outside that directory are not. Keep secrets out of those files, or set `captureSource: false` on the suite.

## Run

`sphynx eval` runs every `*.eval.ts` file it finds. Pass a path to run one: `sphynx eval ./smoke.eval.ts`. Add `--case <id>` to run one case from the file.

See [variants](/evals/variants) for harnesses, models and sandboxes, and [profiles](/evals/profiles) for custom harness configuration.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.