# Show which organization a credential uses Source: https://docs.sphynx.sh/api-reference/auth/show-which-organization-a-credential-uses /openapi.json post /v1/auth.whoami Runs, suites, and variables set with this credential land in this organization. # Get a batch Source: https://docs.sphynx.sh/api-reference/batches/get-a-batch /openapi.json post /v1/evals.batches.get # List batches Source: https://docs.sphynx.sh/api-reference/batches/list-batches /openapi.json post /v1/evals.batches.list Newest first. Pass the `next` cursor from a response to read the page after it; a null `next` means there are no more. # Start a batch for a suite Source: https://docs.sphynx.sh/api-reference/batches/start-a-batch-for-a-suite /openapi.json post /v1/evals.batches.start Starts one run per case and variant, and returns the batch while trials continue in the background. The command harness needs a profile with a run command; other harnesses take no run. Any harness may take an install command. # Get a case, its variants and its versions Source: https://docs.sphynx.sh/api-reference/cases/get-a-case-its-variants-and-its-versions /openapi.json post /v1/evals.cases.get # List cases Source: https://docs.sphynx.sh/api-reference/cases/list-cases /openapi.json post /v1/evals.cases.list # Start a batch for a case Source: https://docs.sphynx.sh/api-reference/cases/start-a-batch-for-a-case /openapi.json post /v1/evals.cases.run Runs the case's newest version on the variants named, or on every variant it has run on, so they can be compared. # List variables and subscriptions Source: https://docs.sphynx.sh/api-reference/environment/list-variables-and-subscriptions /openapi.json post /v1/environment.list Secret values are never returned, only a preview. # Remove a variable Source: https://docs.sphynx.sh/api-reference/environment/remove-a-variable /openapi.json post /v1/environment.remove # Set variables Source: https://docs.sphynx.sh/api-reference/environment/set-variables /openapi.json post /v1/environment.set # API overview Source: https://docs.sphynx.sh/api-reference/introduction Call the Sphynx API over HTTP The public API is at `https://api.sphynx.sh/v1`. Every operation is a `POST` with a JSON body, reads included, and there are no path or query parameters: ```text theme={null} POST /v1/evals.batches.start POST /v1/evals.batches.get POST /v1/evals.cases.run ``` ## Authentication Pass an API key as a bearer token: ```bash theme={null} curl https://api.sphynx.sh/v1/evals.batches.list \ -H "authorization: Bearer $SPHYNX_API_KEY" \ -H "content-type: application/json" \ -d '{"limit":20}' ``` A key belongs to one organization. OAuth access tokens also work. Reading evals needs `evals:read`. Starting a batch needs `evals:write`. ## Start a batch `evals.batches.start` creates a batch that runs every case of a suite on every variant. It returns right away with the batch `id` and a `runs` list holding one run per case per variant, each with its `caseId` and `variantId`, while the trials continue on the server: ```bash theme={null} curl https://api.sphynx.sh/v1/evals.batches.start \ -H "authorization: Bearer $SPHYNX_API_KEY" \ -H "content-type: application/json" \ -d '{ "suite": { "id": "smoke", "name": "Smoke", "prompt": "{{task}}" }, "cases": [{ "id": "writes-hello", "variables": { "task": "Create hello.txt containing exactly hello" }, "verify": "test \"$(cat hello.txt)\" = hello" }], "variants": [{ "harness": "codex", "model": "gpt-5.6-sol", "sandbox": "daytona" }], "trials": 3 }' ``` `trials` is the number of trials in each run. `verify` is the shell command that decides whether a trial passed, which the SDK writes as `validate: command("...")`. Poll `evals.batches.get` with the returned `id` until `status` is `finished` or `failed`. The API can't run the `local` sandbox. Run those with `sphynx eval --local`. Starting a batch is not idempotent. If the response is lost, list recent batches with `evals.batches.list` before retrying. ## Run a case again `evals.cases.run` starts a batch for a stored case's newest version. Name variant ids from `evals.cases.get` to pick variants, or leave `variants` out to run every variant the case has run on. `trials` defaults to 1: ```bash theme={null} curl https://api.sphynx.sh/v1/evals.cases.run \ -H "authorization: Bearer $SPHYNX_API_KEY" \ -H "content-type: application/json" \ -d '{"id":"writes-hello","trials":3}' ``` ## Limits | Limit | Value | | - | - | | Trials per run | 1 to 10 | | Cases per batch | 100 | | Variants per batch | 20 | | Trials per batch (cases x variants x trials) | 100 | | Batches running at once per organization | 3 | A batch over a size limit is refused with `400`. A start while three batches are running is refused with `409`. Retry once one finishes. ## Errors | Status | Meaning | | - | - | | `400` | Invalid request body | | `401` | Missing or invalid bearer token | | `403` | Missing permission | | `404` | Resource not found | | `409` | Request conflicts with current state | The errors above return JSON with a `message`. The [TypeScript client](/sdk/client) decodes responses and throws `SphynxError` on failure. # List models available to a harness Source: https://docs.sphynx.sh/api-reference/models/list-models-available-to-a-harness /openapi.json post /v1/evals.models.list # Add a version Source: https://docs.sphynx.sh/api-reference/prompts/add-a-version /openapi.json post /v1/prompts.update Content is versioned, so updating a prompt appends a version rather than overwriting one. Earlier versions stay readable. # Create a prompt Source: https://docs.sphynx.sh/api-reference/prompts/create-a-prompt /openapi.json post /v1/prompts.create Creates the prompt and its first version in one call. # List prompts Source: https://docs.sphynx.sh/api-reference/prompts/list-prompts /openapi.json post /v1/prompts.list Up to 100 prompts in the organization, without content. # Promote a version to a channel Source: https://docs.sphynx.sh/api-reference/prompts/promote-a-version-to-a-channel /openapi.json post /v1/prompts.promote Points a channel, such as production, at a version. This is how a version goes live without callers changing anything. # Resolve a prompt Source: https://docs.sphynx.sh/api-reference/prompts/resolve-a-prompt /openapi.json post /v1/prompts.get Returns the content a caller should send to a model. With no selector this follows the organization's default channel. # Get a run and its trials Source: https://docs.sphynx.sh/api-reference/runs/get-a-run-and-its-trials /openapi.json post /v1/evals.runs.get # List a case's runs Source: https://docs.sphynx.sh/api-reference/runs/list-a-cases-runs /openapi.json post /v1/evals.runs.list Newest first, 20 to a page. # Get a suite and the setup its cases share Source: https://docs.sphynx.sh/api-reference/suites/get-a-suite-and-the-setup-its-cases-share /openapi.json post /v1/evals.suites.get # List suites Source: https://docs.sphynx.sh/api-reference/suites/list-suites /openapi.json post /v1/evals.suites.list Most recently active first. Pass the `next` cursor from a response to read the page after it; a null `next` means there are no more. # Environment Source: https://docs.sphynx.sh/environment The keys and subscriptions your evals use Every run needs a key for its [harness](/evals/variants), because model usage is charged to your account. Set one under **Settings > Environment** before you start your first batch. The eval API does not accept raw model keys. | Surface | Credential | | - | - | | Sphynx CLI and SDK | `SPHYNX_API_KEY` | | Harness | Its key or subscription in Settings > Environment | | Hosted sandbox | None. Uses Sphynx's account unless you set your own | | Judge that calls a provider, or a simulated person | Its provider key in Settings > Environment | | Mock product MCP or CLI | None | | [Sphynx MCP](/tools/mcp) | OAuth 2.1 | ## Variables Settings > Environment holds one table of variables and subscriptions. Choose **Add variable** and enter `NAME=value` pairs. You can paste a whole `.env` file into any name field, and known names autocomplete as you type. Adding a name that already exists replaces its value. Values are encrypted on arrival. A secret is never shown again, only its last four characters, such as `••••7f2a`, and a short secret shows none. A plain value, such as a URL, stays readable. Names starting with `SPHYNX_` are reserved. ## Known names Sphynx picks these up on its own. You don't name them anywhere. | Name | Used by | | - | - | | `ANTHROPIC_API_KEY` | Claude Code, Anthropic judges | | `OPENAI_API_KEY` | Codex by key, OpenAI judges | | `AI_GATEWAY_API_KEY` | FX | | `GEMINI_API_KEY` | Gemini CLI, Google judges | | `CURSOR_API_KEY` | Cursor Agent | | `DASHSCOPE_API_KEY`, optional `QWEN_BASE_URL` | Qwen Code | | `XAI_API_KEY`, `MOONSHOT_API_KEY`, `DEEPSEEK_API_KEY`, `GROQ_API_KEY`, `OPENROUTER_API_KEY`, `TYPESAFE_API_KEY` | Judges and simulated people | | `DAYTONA_API_KEY` | Daytona sandboxes on your account | | `E2B_API_KEY` | E2B sandboxes on your account | | `UPSTASH_BOX_API_KEY` | Upstash Box sandboxes on your account | | `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | Modal sandboxes on your account | | `CLOUDFLARE_API_TOKEN`, optional `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_SANDBOX_API_KEY`, `CLOUDFLARE_SANDBOX_URL` | Cloudflare sandboxes on your account | | `VERCEL_TOKEN`, `VERCEL_TEAM_ID`, `VERCEL_PROJECT_ID` | Vercel sandboxes on your account | A harness gets only its own key. Judge and sandbox keys stay on the server and never enter the agent's sandbox. Without a sandbox key, trials run on Sphynx's hosted account. ## Subscriptions Choose **Add subscription** to run a harness on a plan instead of a key. | Subscription | Runs | How | | - | - | - | | ChatGPT | Codex | Sign in with a device code. It renews itself | | OpenCode | OpenCode | Paste the output of `cat ~/.local/share/opencode/auth.json` | | Pi | Pi | Paste the output of `cat ~/.pi/agent/auth.json` | Codex uses a ChatGPT subscription over `OPENAI_API_KEY` when both are set. A ChatGPT subscription is not an OpenAI API key, so OpenAI judges still need `OPENAI_API_KEY`. ## Your own variables Any other name is yours. It reaches a run only when the variant's [profile](/evals/profiles) names it: ```json profile.json theme={null} { "variables": ["SEARCH_API_KEY"] } ``` A named variable reaches the profile's process, the install command, the case's prepare step and code checks. A hosted batch whose profile names a variable that is not set is refused before it starts: ```text theme={null} A profile names variables that are not set. Set SEARCH_API_KEY in Settings > Environment. ``` ## From the terminal ```bash theme={null} sphynx env list sphynx env set ANTHROPIC_API_KEY sphynx env import .env sphynx env remove SEARCH_API_KEY ``` `set` prompts for the value instead of taking it as an argument, since an argument ends up in shell history and the process list. In a script, pipe it in: ```bash theme={null} echo "$ANTHROPIC_API_KEY" | sphynx env set ANTHROPIC_API_KEY ``` See the [CLI](/tools/cli#environment-variables) for every flag. Subscriptions are added in the dashboard. ## Judges and simulated people A [judge](/evals/judges) that names a `harness` runs on that harness's key or subscription. A judge declared with `provider: "openai"` needs `OPENAI_API_KEY`. A case that states a [human](/evals/conversations) needs a model to play them. `openai`, `anthropic`, `google`, `xai`, `moonshotai`, `deepseek`, `groq` and `openrouter` all serve the same chat-completions API, so the human runs on the first of their keys it finds. Provider judges call OpenAI's responses API, which the others don't serve, so a case scored by one needs `OPENAI_API_KEY`. On a self-hosted deployment, `OPENAI_API_KEY` in the server and worker environment works as a fallback. A missing judge key is not a startup error. The judge that needs it fails when a case reaches it, and names what was missing. Keep keys out of eval definitions and fixtures. Runs read everything they need from Settings > Environment. ## Everyone or only me A variable for **Everyone in the organization** reaches every run in it. One for **Only me** reaches only your runs, and overrides the organization's value of the same name, so you can run on your own account without changing what your team uses. An API key acts for the organization, so it uses only the organization's values. ## Missing or removed ```text theme={null} Set ANTHROPIC_API_KEY in Settings > Environment to run claude ``` When the harness key is missing, the batch is refused, nothing is charged and no sandbox opens. Removing a variable leaves past runs readable. A run already holding the value finishes. # Suites and cases Source: https://docs.sphynx.sh/evals/cases Define what the agent does and how it passes A suite is a system prompt and shared setup for many cases. Each case is one eval: it fills in the prompt, sets up the workspace and checks the result. ```ts theme={null} import { command, repo, suite } from "sphynx-sh"; export default suite({ id: "api", source: repo("acme/api@8f31c4a"), prompt: "{{task}}", cases: [ { id: "health-route", variables: { task: "Add a /health route that returns 200" }, validate: command("bun test test/health.test.ts"), }, ], variants: [{ harness: "codex", model: "model-id" }], trials: 3, }); ``` Running this file starts a batch: one run per case per variant, each with `trials` trials in their own sandboxes. A suite and a case each need an `id`: a handle of up to 100 lowercase letters, digits and single hyphens, such as `health-route`. A case id is unique within your organization. Its history is kept under it, so keep it stable. When a case's definition changes, later runs record a new version of the case. | Case field | Purpose | | - | - | | `id` | Stable handle the case's history is kept under (required) | | `name` | Label for the case's row in the grid. Defaults to the id | | `variables` | Values for prompt variables such as `{{task}}` | | `source` | Workspace for this case, overriding the suite's | | `prepare` | Typed setup that runs before the agent | | `cache` | One prepared directory to restore and save | | `tags` | Up to 12 labels to group and filter cases by | | `user` | Makes the case a [conversation](/evals/conversations) instead of one prompt | | `maxTurns` | Most turns a conversation may hold. Defaults to 8 | | `timeoutMs` | How long the agent may work across the whole trial. Defaults to 15 minutes | | `validate` | How the trial passes: a validator, a [model judge](/evals/judges), `command("...")`, or an array of validators and judges (required) | The suite's `name` also defaults to its `id`. ## Sources `source` is the workspace the agent starts in. Leave it out for an empty workspace. ```ts theme={null} import { files, repo } from "sphynx-sh"; source: repo("acme/api@8f31c4a"); source: files({ "src/add.ts": "export const add = () => 0;" }); ``` `repo()` accepts `owner/repo`, `owner/repo@ref` or a clone URL. A case's `source` overrides the suite's. ## Prepare `prepare` receives typed filesystem and process helpers. Whatever it returns reaches the validator as `prepared`. ```ts theme={null} import type { Prepare } from "sphynx-sh"; export const install: Prepare = async ({ cached, exec }) => { if (!cached) await exec("npm", ["ci"], { timeoutMs: 600_000 }); return { port: 4173 }; }; ``` ```ts theme={null} { cache: { key: "npm-lock-v1", path: "node_modules" }, prepare: install, } ``` `cached` is `true` when the cache directory was restored. The cache path must stay inside the workspace. `prepare` can return a key it creates, such as a test API key. Sphynx hands the value to your validator and never prints it in logs or the terminal or stores it as evidence. Output the prepare prints itself is logged while it runs, with anything that looks like a key replaced by `[redacted]`. ## Limits `timeoutMs` is how long the agent may work across all of a trial's turns, from 1,000 to 3,600,000 milliseconds. It defaults to 15 minutes. Time the person spends replying does not count. When the agent runs past it, the harness is stopped and the trial ends as timed out, never as a pass or a fail. `maxTurns` caps how many times the person speaks in a [conversation](/evals/conversations), the opening prompt included. It takes 1 to 50 and defaults to 8. ```ts theme={null} { id: "long-migration", maxTurns: 4, timeoutMs: 1_800_000, validate: migrated, } ``` Changing either limit creates a new version of the case. ## Validate `command("bun test")` runs a shell command in the workspace after the agent finishes. The trial passes when it exits 0. Use a validator when the check is clearer in TypeScript than in a shell command: ```ts theme={null} import type { Validator } from "sphynx-sh"; export const hasHealthRoute: Validator = async ({ readText }) => ({ message: "the health route exists", passed: (await readText("src/routes.ts")).includes("/health"), }); ``` Set `validate: hasHealthRoute` on the case. Sphynx bundles the function and its imports with the case. A validator can also return a plain boolean. The validator receives `answer`, `transcript`, `turns`, `readText`, `exists`, `exec`, `prepared`, and the call logs of any [mocks](/evals/mocks) (`api`, `cli`, `mcp`). Every check of a trial receives the same context object, so checks can share work, such as one request to your app, by caching on it. `answer()` is the agent's last message and `transcript()` is every agent message joined. `turns()` is the whole conversation, turn by turn, with the messages, commands, tool calls and file changes of each. A case without `user` has one turn. See [what a validator reads](/evals/conversations#what-a-validator-reads). A check takes its name from the function, so `hasHealthRoute` shows as hasHealthRoute in the transcript and the dashboard. A function built by another function has no name and shows as Validator 1. Give it one with `named`: ```ts theme={null} import { named, type Validator } from "sphynx-sh"; const reports = (status: string): Validator => async ({ answer }) => (await answer()).includes(status); validate: [named("reports created", reports("created"))] ``` Pass an array to combine validators and judges, up to 20 per case: `validate: [hasHealthRoute, correctness]`. A `command()` can't go in an array. Use [model judges](/evals/judges) for checks that need interpretation. Every code validator runs, even after an earlier one fails, so each one records its own result and pass rate. The trial passes only when all of them pass. ### Source capture The dashboard's **Validation** panel shows the validator source as you wrote it, captured at compile time. Each run keeps its own snapshot. Only TypeScript and JavaScript files under the eval's nearest `package.json` directory are captured. Dependencies and files outside that directory are not. Keep secrets out of those files, or set `captureSource: false` on the suite. ## Run `sphynx eval` runs every `*.eval.ts` file it finds. Pass a path to run one: `sphynx eval ./smoke.eval.ts`. Add `--case ` to run one case from the file. See [variants](/evals/variants) for harnesses, models and sandboxes, and [profiles](/evals/profiles) for custom harness configuration. # Command harness Source: https://docs.sphynx.sh/evals/command-harness Run your own agent as a process inside the sandbox The `command` harness runs a process you own instead of a coding agent Sphynx installs. Sphynx starts it in the sandbox, passes it the case through environment variables, reads events from its stdout, and scores the trial with the case's `validate`. ```ts theme={null} variants: [ { harness: "command", model: "openai/gpt-5.5", profile: { name: "sample", dir: "./profile" }, sandbox: "e2b", }, ] ``` The command harness needs a [profile](/evals/profiles). Its `run` is the command line, run with `bash -c` from the workspace. Its `install` runs once while the sandbox is prepared. The profile directory ships everything else the process needs. ## Environment | Variable | Value | | - | - | | `SPHYNX_PROMPT` | The case prompt | | `SPHYNX_MODEL` | The variant's `model`, verbatim | | `SPHYNX_HOME` | The sandbox home directory | | `SPHYNX_WORKSPACE` | The workspace, and the working directory when the process starts | | `SPHYNX_SYSTEM_PROMPT_FILE` | Path to the profile's system prompt, when it has one | | `SPHYNX_TRACE_LOG` | Where the shell recorder appends what it sees | The profile's `env` is set too, and so are the [variables](/environment#your-own-variables) it names. Stdin is closed. ## Runner kit Write the process in TypeScript with `sphynx-sh/runner`. It reads the environment and prints the event lines for you, and it rejects a line the server would not accept. ```ts theme={null} import { createEmitter, env } from "sphynx-sh/runner"; const { model, prompt } = env(); const emit = createEmitter(); emit.started({ model, sessionId: "run-1" }); emit.toolCall({ name: "search", input: { query: prompt }, output: ["total.ts"] }); emit.message({ text: "The total lives in total.ts." }); emit.usage({ inputTokens: 900, outputTokens: 40 }); emit.finished("done"); ``` Set the profile's `run` to start the script, for example `"run": "bun run agent.ts"`. `env()` returns `prompt`, `model`, `workspace`, `home`, `systemPromptFile` and `traceLog`. It throws when a variable the harness always sets is missing, so a script run by hand fails with the name of the variable. | Method | Line it prints | | - | - | | `started({ sessionId, model })` | `Started` | | `message({ text, role?, usage? })` | `Message`. `role` defaults to `assistant` | | `toolCall({ name, input, output?, error?, status?, callId? })` | `ToolCall`. A value that is not a string is written as JSON | | `usage({ inputTokens, outputTokens, cacheReadTokens?, cacheWriteTokens? })` | `Usage`. Counts must be whole numbers, zero or more | | `command({ command, exitCode, output })` | `Command` | | `fileChange(paths)` | `FileChange` | | `finished(reason)` | `Finished`. A second call prints nothing | Every event carries the time it was printed. To collect lines instead of printing them, pass `createEmitter({ write, now })`. ## Events The kit prints the lines below. They are also the protocol for a process in any other language. Print one JSON object per line on stdout. Lines with a known `_tag` are recorded and every other line is ignored, so ordinary logging is fine. `at` is epoch milliseconds and optional. A line without it is stamped when it is read. ```json theme={null} {"_tag":"Started","sessionId":"run-42","model":"openai/gpt-5.5"} {"_tag":"Message","role":"assistant","text":"Reading the failing test first."} {"_tag":"Command","command":"bun test","exitCode":1,"output":"expect(5).toBe(6)\n"} {"_tag":"FileChange","paths":["/workspace/src/total.ts"]} {"_tag":"ToolCall","callId":"call_7","name":"search","input":"{\"query\":\"total\"}","status":"completed"} {"_tag":"Usage","inputTokens":9120,"outputTokens":312,"cacheReadTokens":8000} {"_tag":"Finished","reason":"done","at":1788300005000} ``` | Line | Recorded as | | - | - | | `Started` | The session id and model. Optional. Without it the trial has no session id | | `Message` | A conversation turn. `role` is `assistant` or `user`. The last assistant message is the answer validators read | | `Command` | A shell command with its exit code and output. `exitCode` is required but may be `null` | | `FileChange` | Absolute paths written | | `ToolCall` | A tool call by name. `callId` and `status` are required but may be `null` | | `Usage` | Tokens for one turn, added to the trial's total. `cacheReadTokens` and `cacheWriteTokens` default to 0, `totalTokens` to input plus output | | `Finished` | Ends the journal with a reason. A reason that starts with `failed:` ends the trial unscored, with the rest of the reason as why | ## Exit A non-zero exit does not fail the trial or discard earlier events. If the process never prints `Finished`, Sphynx records one with the reason `exit N`, and the trial is scored. A `Finished` reason that starts with `failed:` is the one exception. The trial is not scored, and the CLI and the trial page show the reason, so a refused model or a missing key reads as what it is. The process shares the case's [time limit](/evals/cases#limits) with every other turn of the trial, 15 minutes by default. If it is still running when the limit runs out, it is stopped and the trial ends as timed out. ## The recorder Sphynx sources a small script into every non-interactive bash the process starts, through `BASH_ENV`. Its `DEBUG` trap appends one line per command to `SPHYNX_TRACE_LOG`. Each line becomes a `Command` in the journal with a `null` exit code, unless you already printed a `Command` with exactly the same text. The log is cleared after each turn, so a turn records only the commands it ran. It only sees bash and zsh, the two shells with a `DEBUG` trap: * `bash -c` and nested bash scripts are traced line by line. * `sh -c` on Debian is dash, which has no `DEBUG` trap. Only the `sh -c …` invocation is recorded. * Shells spawned by Node, Python or other runtimes are invisible. A Python agent leaves one line: its own invocation. * Exit codes are never captured, because the trap fires before the command runs. To get inner commands and exit codes into the journal, print `Command` lines yourself. # Conversations Source: https://docs.sphynx.sh/evals/conversations Test an agent over a conversation, not a single prompt A case sends one prompt and reads one answer unless it sets `user`. With `user`, the agent's reply goes to a simulated person, they answer what it actually said, and the harness continues the same session. That lets an eval check things like whether the agent asked before it acted. ```ts theme={null} import { suite, type Validator } from "sphynx-sh"; import { human } from "sphynx-sh/validators"; const askedFirst: Validator = async ({ cli, turns }) => { const conversation = await turns(); const approved = conversation.findIndex((turn) => turn.userText.toLowerCase().includes("yes") ); const calls = await cli.calls(); return { passed: approved > 0 && calls.length > 0 }; }; export default suite({ id: "pricing", prompt: "Read USER_REQUEST.md and model the pricing.", cases: [ { id: "asks-before-it-pushes", user: human({ goal: "Get Pro live in the sandbox, not only written to a file.", prompt: `Pro is $20 a month with 500 messages. Hard limit, no overage. If the agent asks whether to push or go live: yes, go ahead.`, }), validate: askedFirst, }, ], variants: [{ harness: "codex", model: "gpt-5.6-sol", sandbox: "e2b" }], trials: 3, }); ``` ## Goal and prompt `goal` (up to 2,000 characters) is what the person wants done. They pursue it every turn and push back when the agent stalls. `prompt` (up to 8,000 characters) is the private brief they answer from. They do not volunteer it. A broad question gets everything in the brief that answers it. A question never asked gets nothing, so an agent that guesses instead of asking will contradict facts it was never told. Write the brief the way the person would say it, one fact per line. The person is told not to invent anything outside it. ## Who plays the person By default a model plays the person. It is fixed for all cases, not chosen per case, and defaults to `gpt-5.4-mini`. A weak simulator can approve a summary that contradicts its own brief, so the case passes for the wrong reason. The `SPHYNX_USER_MODEL` environment variable on the runner overrides it. That model needs its provider key in [Settings > Environment](/environment), `OPENAI_API_KEY` for the default. Without one, a batch that contains a case with a simulated person is refused when it starts. To play the person through a harness instead, name one and a model: ```ts theme={null} user: human({ goal: "Get Pro live in the sandbox, not only written to a file.", prompt: "Pro is $20 a month with 500 messages.", harness: "codex", model: "gpt-5.6-luna", }), ``` `harness` is any harness except `command`, and `model` is required with it. The person runs on that harness's key or subscription, so a Codex person can run on a ChatGPT subscription and needs no `OPENAI_API_KEY`. Without one, the trial ends with an error that names the missing harness. A harness person gets a sandbox of its own, separate from the agent's workspace, kept for the whole conversation. It is told to reply only in words, never to run commands or touch files. On `codex` and `claude`, the person keeps one session for the whole conversation, and each later turn sends only the agent's latest reply. Other harnesses start a new session every turn and are sent everything said so far. In a local run, Sphynx leases a key for every harness the suite needs, including one that plays the person. Each is used only for its own harness. Whoever plays the person, model or harness and model, is part of the variant, so results from different people are never mixed. Changing it creates a new variant. A harness and model named in the case are also part of the case, so changing them creates a new case version. ## Ending A conversation ends when the person says they are done, which they decide from the agent's reply, not a turn count. It also ends after the case's `maxTurns`, eight by default. See [limits](/evals/cases#limits). If the harness reports no session to continue, the conversation stops after the first turn instead of starting over with a second opening prompt. `codex` and `claude` continue sessions. Other harnesses hold a one-turn conversation. If the person stops answering mid-conversation (for example, the user model or harness fails), the trial ends with an error instead of a pass or fail. A conversation cut short would otherwise look like one the agent finished. ## A fixed script When the order of events matters more than the conversation, give the replies up front: ```ts theme={null} import { script } from "sphynx-sh/validators"; user: script(["yes, go ahead", "that looks right"]); ``` A script takes 1 to 24 replies and ends when they run out. It never reads the agent's words, so it costs nothing and never varies. Use it to test what happens after an approval, not whether the agent earned one. ## What a validator reads `turns()` returns each turn in order, with `index`, `userText`, `agentText`, `commandCount` and `events`. `events` is everything that happened in that turn, in order, in the same shape the dashboard shows: | `_tag` | Fields | | - | - | | `message` | `role` (`user` or `assistant`), `text` | | `command` | `command`, `exitCode`, `output` | | `fileChange` | `paths` written | | `toolCall` | `name`, `input`, `output`, `error`, `status` | Each turn opens with the person's `message`. Command output and tool input, output and errors keep their first 4,000 characters, and `outputTruncated`, `inputTruncated` or `errorTruncated` says when something was cut. ```ts theme={null} const readSkillBeforeWriting: Validator = async ({ turns }) => { const [first] = await turns(); const read = first.events.findIndex( (event) => event._tag === "toolCall" && event.input?.includes("SKILL.md") ); const wrote = first.events.findIndex((event) => event._tag === "fileChange"); return { passed: read !== -1 && (wrote === -1 || read < wrote) }; }; ``` `commandCount` is the number of commands the agent ran in the turn. It ties calls to turns. The call log is in order, so the first turn owns the first `commandCount` calls, the second owns the next, and so on. A case without `user` has one turn: the prompt and everything the agent did in reply. ## Cost Every turn is a model call for the agent and a cheaper one for the person. A conversation costs several times a single-prompt case, and one where the person never finishes costs every turn up to `maxTurns`. The trial's cost shows the person on its own line, priced at the published rate of the model playing the person, so it never inflates the agent's model cost or token counts. Give the person a goal they can recognize as met. # Import a case file Source: https://docs.sphynx.sh/evals/import Turn existing JSON or YAML cases into a TypeScript suite `sphynx eval import` converts an existing case file into a `*.eval.ts` suite. ```bash theme={null} sphynx eval import --format evals-json ./evals.json --out ./release-notes.eval.ts ``` Without `--out`, the suite prints to stdout. A summary of converted and unconverted assertions goes to stderr. | `--format` | Input | | - | - | | `evals-json` | One JSON file holding every case | | `yaml` | One YAML file per case, or a directory of them | The generated suite has a placeholder variant (`codex` on `gpt-5.6-sol`) and `trials: 3`, with each case's text passed through a `task` variable. ## evals-json ```json theme={null} { "skill_name": "release notes", "evals": [ { "id": 1, "name": "Lists the breaking change", "prompt": "Read CHANGELOG.md and write the release notes.", "expected_output": "A short note naming the breaking change.", "files": ["fixtures/changelog"], "assertions": [ { "text": "names the removed flag", "kind": "content_contains_all", "needles": ["--legacy", "removed"] } ] } ] } ``` `skill_name` and `evals` are required, and each case needs `id`, `prompt` and `assertions`. An assertion is a structured object or a prose string. | `kind` | Generated check | | - | - | | `content_contains_any` | `containsAny(answer, needles)` | | `content_contains_all` | `containsAll(answer, needles)` | | `content_contains_none` | `containsNone(answer, needles)` | Matching is case-insensitive. Prose assertions cannot be converted safely, so each becomes a placeholder that always fails: ```ts theme={null} validate: async (context) => { const answer = await context.answer(); return [ /* Write this check, then delete the line under it: */ /* The summary reads as though a maintainer wrote it. */ unwritten("The summary reads as though a maintainer wrote it."), ].every(Boolean); }, ``` `expected_output` becomes a comment, not a check. `files` names directories but not their contents, so each case starts with `files({})` and a comment naming them. ## yaml A directory imports every `*.yaml` and `*.yml` file in it, sorted by name. ```bash theme={null} sphynx eval import --format yaml ./cases --out ./browsing.eval.ts ``` ```yaml theme={null} name: Checkout flow task: Open the demo storefront and pay with the test card judge_context: - Reach the basket page before paying - Use the test card, never a real one max_steps: 12 ``` `name` and `task` are required. The task becomes the prompt. Each `judge_context` entry needs human judgment, so it becomes a failing placeholder. A case without `judge_context` gets one placeholder asking what a good answer is. `max_steps` becomes a comment, because Sphynx does not cap agent steps. YAML cases start with `files({})` because the format carries no source files. ## Finish the suite 1. Review the generated file. 2. Replace each `files({})` with fixture files or a [repository source](/evals/cases). 3. Set the harness, model and sandbox for each variant. 4. Replace every `unwritten(...)` placeholder with a real check. 5. Run it with `sphynx eval ./suite.eval.ts`. An invalid file fails with its path, case and field. A directory is never partially imported. # Model judges Source: https://docs.sphynx.sh/evals/judges Score answers with a model you choose Use code for exact checks. Add a judge for qualities that need interpretation, such as correctness or clarity. ```ts theme={null} import { suite } from "sphynx-sh"; import { judge } from "sphynx-sh/validators"; const correctness = judge({ name: "correctness", harness: "codex", model: "gpt-5.6-sol", prompt: "The answer explains why the result is correct, without inventing facts.", expected: "2 + 2 = 4", choices: { correct: 1, partial: 0.5, incorrect: 0 }, threshold: 1, }); export default suite({ id: "arithmetic", prompt: "What is 2 + 2? Explain briefly.", cases: [{ id: "addition", validate: correctness }], variants: [{ harness: "codex", model: "gpt-5.6-sol", sandbox: "e2b" }], trials: 3, }); ``` The judge model is independent of the variant being measured. Sphynx requests exactly the model you name. If it is unavailable, the judgment is unscored. There is no fallback model. ## Combine checks `validate` accepts one validator or judge, or an array of them: ```ts theme={null} validate: [checkToolCalls, correctness]; ``` Code checks run in the trial's sandbox. Judges run after it. A trial passes only when every code check passes and every judge meets its threshold. A passing judge cannot override a failed code check. ## Configuration | Field | Meaning | | - | - | | `name` | Judge name, unique within the case | | `harness` or `provider` | A harness (any except `command`), or `provider: "openai"` for a direct model call | | `model` | Exact model identifier | | `prompt` | Instructions for the judgment, separate from the case prompt | | `expected` | Optional reference answer | | `files` | Optional workspace files the judge reads, 1 to 8 paths relative to the workspace | | `choices` | Labels mapped to scores from 0 to 1 (1 to 20 choices) | | `threshold` | Inclusive pass threshold, default 1 | | `timeoutMs` | Total judging timeout, default 120,000, range 1,000 to 300,000 | A case accepts up to 20 validators. `judge()` checks its options when you call it. ## Grade workspace files Use `files` when the result is a file, not the final answer. A judge that grades a blog post can read the post itself: ```ts theme={null} const quality = judge({ name: "post quality", harness: "codex", model: "gpt-5.6-sol", prompt: "The post is accurate, clear and ready to publish.", choices: { publish: 1, revise: 0 }, files: ["out/post.md"], }); ``` Sphynx reads each file after the code checks run, before the trial's sandbox closes. The judge gets the full text of each file with its path. Files are not cut like the conversation. Secrets are redacted the same way as in the answer. Each file can hold up to 32,000 characters. All judge files in a case together can hold up to 96,000. A path must be relative to the workspace and cannot contain `..`. If a file is missing or over a limit, the judge is not asked. Its judgment is invalid with an error that names the file, and the trial is void. It never counts as a pass or a fail. ## Authentication A harness judge runs on that harness's key or subscription in Settings > Environment. A Codex judge can use a ChatGPT subscription and needs no `OPENAI_API_KEY`. The key stays on the server and never enters the agent's sandbox. A `provider: "openai"` judge needs `OPENAI_API_KEY`. See [environment](/environment). ## Evidence and isolation A judge receives the rendered case prompt, the conversation, the final answer and `expected`, if set. The conversation lists what happened in order: each user and agent message, each command with its exit code and output, each tool call, and each file written. A case without `user` is a one-turn conversation that opens with the prompt, so a judge can still ask what the agent did before it answered. Command output, tool input and output, and long messages are cut to 4,000 characters. A conversation longer than 48,000 characters keeps its start and end and marks how many steps were left out of the middle. A judge does not see the rest of the workspace, MCP or CLI configuration, or sandbox environment. To grade a file the agent wrote, name it in [`files`](#grade-workspace-files). A harness judge starts in a fresh sandbox after the trial's sandbox closes. It is told to grade only the supplied evidence. If it uses tools anyway, the judgment is invalid. Tool use is detected, not blocked by a firewall. To assert on tool use, keep a code validator that reads the mock call log. An OpenAI judge uses structured outputs with storage disabled. Both kinds must pick one of the declared choices and give a short reason. A timeout, refusal, invalid response or provider failure gives `score: null` and voids the trial, so it is not counted as a pass or a fail. ## Results Each trial's judgments carry `name`, `model`, `evaluator`, `choice`, `score`, `threshold`, `reason`, `durationMs`, `error` and the tokens the judge used as `usage`. The dashboard shows them apart from code checks. Judging has its own line in the trial's cost, apart from the agent's model. Each judge is priced at its model's published rate. If a judge reported no usage, or its model has no published rate, the line reads as not known and the trial's cost is marked incomplete. The sandbox a harness judge runs in is not priced. For the request, raw response and provider metadata, open the judge entry in the trial's `validations`. Invalid response text is kept even when no score is assigned. See [validation evidence](/evals/results#validation-evidence) for capture limits and privacy settings. Changing a judge changes the case definition, so later runs record a new case version. Calibrate each judge prompt against known good and bad answers before trusting it, and use several trials: judgments vary even with the same model and prompt. # Run evals locally Source: https://docs.sphynx.sh/evals/local Run an eval on your machine so it can reach local services A cloud sandbox cannot reach `localhost`. When the thing you are testing runs on your machine, `--local` runs the eval there instead: ```bash theme={null} sphynx eval --local ./checkout.eval.ts ``` Every case runs on every variant, with the suite's number of trials, and the results print in the terminal. `--case ` and `--variant` narrow what runs. `--variant` takes a variant's label, `harness/model` or `harness/model@profile`, or a profile name, and repeats for more. See [variants](/evals/variants#pick-variants-from-the-command-line). Without a file, every `*.eval.ts` under the current directory runs. A local run always waits and prints a transcript, so it can't take `--no-wait`, `--json` or `--output`. ## Reaching a local service Addresses in `.env.local` and `.env` in the directory you run the command from are passed to the case, so the agent, the prepare step and `validate` can reach a service you already have running: ```bash theme={null} # .env API_BASE=http://localhost:1234 ``` ```ts theme={null} cases: [ { id: "checkout-completes", validate: command('test "$(curl -s $API_BASE/health)" = ok'), }, ] ``` `.env.local` wins over `.env`, and your shell's environment wins over both. ## What is forwarded Addresses only. A variable is forwarded when both are true: * Its name is `URL`, `URI`, `HOST`, `PORT`, `ENDPOINT`, `ORIGIN` or `BASE`, or ends in `_` plus one of them. * Its value is an `http(s)` address with no credentials, a hostname, or a port. `DATABASE_URL=postgres://user:password@host/db` has an address name but a password in its value, so it is withheld. To pass a secret, name it in the profile's `variables`. See [keys and variables](#keys-and-variables). Anything forwarded reaches the agent's context and the trial's journal. ## Keys and variables With an API key set, Sphynx leases what the suite needs from [Settings > Environment](/environment): the key for each harness the suite needs, the variant's and any harness that plays the person or judges, the judge keys, and the variables the profile names. Each harness key is used only for its own harness. The lease expires after 15 minutes and is held in memory, never written to disk. Sandbox keys are never leased, because a local batch opens no cloud sandbox. A named variable set in your own shell wins over the leased value. Without an API key, the run reads known names such as `ANTHROPIC_API_KEY`, and the variables the profile names, from your shell. ## What gets recorded With an API key set, `--local` starts a local batch: one run per case per variant, marked local and visible in the dashboard with its journal. Each trial is priced like a hosted one when its report arrives, with the model, the simulated user and the judges on separate lines. Your machine is listed as the sandbox and costs nothing. Without an API key, nothing is sent anywhere and the results only print. An empty or blank `SPHYNX_API_KEY` counts as no key. The exit code reflects the results either way, so CI can gate on it: ```bash theme={null} sphynx eval --local ./checkout.eval.ts || exit 1 ``` ## When a trial cannot finish Each trial ends with its own outcome, and the run moves on to the next one. A trial that runs past the case's `timeoutMs` is timed out. A trial whose setup breaks, such as a prepare step that exits with an error or a harness that will not start, is void with the reason. Neither is scored as a pass or a fail. The summary at the end lists every trial with its outcome, then the link to the batch when it is recorded: ```text theme={null} checkout (checkout.eval.ts): 3 trials on this machine, 1 passed, 1 void, 1 timed out ✓ passed completes on codex/gpt-5.6-sol 41.2s ○ void refunds on codex/gpt-5.6-sol 2.1s The prepare step seeds-orders exited with status 3 ○ timed out retries on codex/gpt-5.6-sol@strict 5m00s The agent ran past its time limit of 5m 26k tokens (25k in, 1k out), $0.46 est. model $0.42, simulated user $0.03, judges $0.01 Results: https://www.sphynx.sh/evals/batch_... ``` The command exits with 2 when any trial did not pass, so CI fails on a void or timed out trial as well as a failed one. `--fail-on never` exits 0 whatever the trials did. Ctrl+C stops the run straight away and closes the batch, and the trials it never reached show as void. ## When the network drops A local run keeps going when Sphynx stops answering for a moment. Starting the batch, leasing keys, reporting a trial and finishing the batch each try again for up to two minutes on a dropped connection or a 429, 502, 503 or 504, and the terminal says so while it waits. If the CLI cannot reach Sphynx at all when the run starts, because the address does not resolve, the connection is refused, TLS fails or the server never answers, it stops within 5 seconds and names the address it tried. Check your network, or set `SPHYNX_BASE_URL` if your Sphynx server is at another address. A trial whose report still does not land shows as void, not missing. The run closes its batch however it ends: done, failed, timed out or stopped with Ctrl+C. While it runs, the CLI checks in with Sphynx every minute. If the machine goes quiet for 10 minutes, because it crashed, lost the network or was closed, the batch stops counting toward your organization's 3 batches at once and is marked failed soon after. Trials it never reported show as void. If the machine comes back, the CLI stops the run, says the batch was closed and exits with 1. Run the eval again for a full result. An older CLI never checks in, so its batch follows the six-hour rule in [results](/evals/results) instead. ## Requirements The agent runs as you, in a temporary workspace with its own `HOME`. It is not a container: a case can read and write anything you can. Only run evals you wrote or trust. Each harness version installs once into `~/.sphynx/local/harness` and is shared by every run after, so the first time you use a harness version locally is slower. Local runs can go side by side: each trial gets its own `HOME` and temp folder, and a failed install says what went wrong. bun and npm in a profile's `install` share one download cache, so a package fetched once is not fetched again by later trials. ## Disk space Local runs keep their files in `~/.sphynx/local`, or wherever `SPHYNX_LOCAL_ROOT` points: | Folder | Holds | | - | - | | `harness` | One install per harness version, a few hundred MB each | | `cache` | What a case's `cache` keeps between runs | | `packages` | The bun and npm download caches profile installs share | | `staging` | Installs and cache entries being written | Each run tidies up as it starts. It removes installs, cache entries and package caches unused for 30 days, staging left by a run that crashed, and the shared `home` folder older CLIs used. Installs no longer keep the npm cache they were built with. Nothing a run is using is touched. To see how much space local runs take, run `du -sh ~/.sphynx/local/*`. To reclaim all of it, delete `~/.sphynx/local` while no local run is going. The next run installs what it needs again. # Mocks Source: https://docs.sphynx.sh/evals/mocks Give agents typed local MCP servers, CLIs and HTTP APIs Mocks are TypeScript objects. Their schemas type your handlers and validate requests and responses at runtime. Each trial gets its own copy of every MCP server, CLI and HTTP API. Keep fixtures in shared files and assertions in validators. ```bash theme={null} npm install sphynx-sh zod ``` ## Share fixtures Keep fixture data and schemas separate from each interface: ```ts fixtures/users.ts theme={null} import { z } from "zod"; export const User = z.object({ id: z.string(), name: z.string() }); export const GetUserInput = z.object({ id: z.string() }); export const user = { id: "user_fixture", name: "Ada", } satisfies z.infer; export const getUser = (id: string) => { if (id !== user.id) throw new Error(`User not found: ${id}`); return user; }; ``` ## MCP ```ts mcp/users.ts theme={null} import { resource, server, tool } from "sphynx-sh/mcp"; import { GetUserInput, User, getUser, user } from "../fixtures/users"; export const usersMcp = server({ name: "users", version: "1.0.0", tools: [ tool({ name: "users_get", inputSchema: GetUserInput, outputSchema: User, handler: ({ id }) => getUser(id), }), ], resources: [ resource({ name: "current_user", uri: "users://current", outputSchema: User, handler: () => user, }), ], }); ``` MCP mocks support tools and fixed-URI resources. Schemas must implement Standard Schema and expose JSON Schema. The examples use Zod 4. ## CLI ```ts cli/users.ts theme={null} import { cli, command } from "sphynx-sh/cli"; import { GetUserInput, User, getUser } from "../fixtures/users"; export const usersCli = cli({ path: "users", version: "1.0.0", commands: [ command({ path: ["get"], inputSchema: GetUserInput, outputSchema: User, options: { id: { description: "User ID", type: "string" } }, handler: ({ id }) => getUser(id), }), ], }); ``` * `cli.path` is the executable name. `command.path` is a non-empty list of argv segments. * Every input key needs an `options` entry. Its flag is the key in kebab case. Set `name` (with the leading `--`) to override it. * String options accept `--id value` or `--id=value`. Boolean options are `true` when present, so make them optional or give them a schema default. * Positional arguments are not supported. The runtime generates help and version output, validates output, prints JSON to stdout and exits nonzero on failure. `sphynx-sh` also exports a `command` for `validate`. If one file needs both, import one under another name. ## Attach and validate Attach MCP and CLI mocks on the suite, then check the call logs in a validator. ```ts users.eval.ts theme={null} import { suite, type Validator } from "sphynx-sh"; import { usersCli } from "./cli/users"; import { GetUserInput, user } from "./fixtures/users"; import { usersMcp } from "./mcp/users"; const requestedFixture = (input: unknown) => { const result = GetUserInput.safeParse(input); return result.success && result.data.id === user.id; }; const validate: Validator = async ({ answer, cli, mcp }) => { const [text, cliCalls, mcpCalls] = await Promise.all([ answer(), cli.calls("users"), mcp.calls("users"), ]); const called = cliCalls.some(({ input }) => requestedFixture(input)) || mcpCalls.some( ({ input, kind, name }) => kind === "tool" && name === "users_get" && requestedFixture(input) ); return { passed: called && text.includes(user.name) }; }; export default suite({ id: "users-interface", prompt: "Use an available interface to retrieve user_fixture and report its name.", mcp: [usersMcp], cli: [usersCli], cases: [{ id: "gets-user", validate }], variants: [{ harness: "codex", model: "model-id" }], trials: 1, }); ``` Check both the recorded call and the outcome. Call inputs and outputs are `unknown`, so parse them with the shared schema before asserting on values. ## HTTP APIs Define endpoints with `api()` and `endpoint()`. Standard Schema types the handler input and checks each response against the schema for its status code. Handlers are plain TypeScript. ```ts mocks/catalog.ts theme={null} import { api, endpoint } from "sphynx-sh/api"; import { z } from "zod"; const item = { id: "fixture", name: "Test item" }; const Item = z.object({ id: z.string(), name: z.string() }); export const catalog = api({ name: "catalog", endpoints: [ endpoint({ method: "GET", path: "/items/:id", inputSchema: z.object({ params: z.object({ id: z.string() }) }), responses: { 200: Item, 404: z.object({ message: z.string() }), }, handler: ({ params }) => params.id === item.id ? { status: 200, body: item } : { status: 404, body: { message: "Item not found" } }, }), ], }); ``` Move `item` and `Item` into a shared fixture file to reuse them with MCP and CLI mocks. API names use lowercase letters, digits and hyphens. ### Attach an API ```ts catalog.eval.ts theme={null} import { suite, type Validator } from "sphynx-sh"; import { catalog } from "./mocks/catalog"; const validate: Validator = async ({ api, answer }) => ({ passed: (await api.calls("catalog")).some( ({ path, status }) => path === "/items/fixture" && status === 200 ) && (await answer()).includes("Test item"), }); export default suite({ id: "catalog-api", api: [catalog], prompt: "Use the local catalog API to retrieve fixture and report its name.", cases: [{ id: "retrieve-item", validate }], variants: [{ harness: "codex", model: "model-id" }], trials: 1, }); ``` In each trial, Sphynx starts a loopback HTTP server on a free port and adds its base URL and endpoint list to the agent's input. Your prompt is unchanged. Curl, Python and SDK clients can call it directly. Prepare functions and validators get the URL from `await api.url("catalog")`. No environment variables or global `fetch` are replaced. ### Requests and responses * The input schema receives `{ params, query, headers, body }`. Headers are lowercase. Repeated query values are arrays. An empty body is `null`, and a nonempty body must be JSON. Use schema transforms for coercion. * Handlers return `{ status, body, headers? }`. Use `null` for bodyless responses such as 204. The runtime sets content type and framing headers. * An optional endpoint `description` is shown to the agent. * Error responses you declare, such as 404 or 429, are normal mock behavior. Unmatched routes return 404 and are recorded with `matched: false`. * A handler that throws, a response that fails its schema, or a body that cannot be serialized fails the mock. Nothing is forwarded upstream. Handlers take a second argument, `{ signal, log }`. Call `log(value)` to attach evidence to the call, and use `signal` to cancel async work. ### Evidence HTTP calls appear in the trial's Calls view and are available to judges. `api.calls(name?)` returns records with method, path, status, timing, whether a route matched, input, output, handler logs and errors. Calls to `api.url` and `api.calls` appear in the validator's evidence. Input, output and log values are bounded snapshots with `text`, `format`, `state` and `truncated` fields. Check those before parsing captured JSON. Authorization headers, cookies and common secret fields are redacted before recording. Add more field names (case-insensitive) with `api({ redact: ["accessToken"], ... })`. Before a call is stored in the trial, anything that looks like a key and every secret the trial knows, such as its harness and sandbox credentials, are also replaced with `[redacted]`, as described in [secrets in evidence](/evals/results#secrets-in-evidence). A secret with no known shape inside a string is still recorded, so keep credentials out of fixtures. ### Validator-owned servers Use `withApi()` when a validator needs its own server with hidden test data. The server closes when the callback finishes or fails. ```ts theme={null} import { withApi } from "sphynx-sh/api"; await withApi({ api: catalog, run: async ({ url, calls }) => { const response = await fetch(`${url}/items/fixture`); console.info("HTTP evidence", await calls()); return response.json(); }, }); ``` These calls are returned by `calls()`, not the trial's call log. Log them inside the validator to include them in its logs. ### Limits * JSON HTTP endpoints only. No proxying, streaming, WebSockets or automatic recording. * Request bodies up to 1 MiB, handlers up to 30 seconds, and 256 requests per mock runtime. * `api.calls()` inside a validator returns values up to 64,000 characters. The trial stores each value at up to 16,000 characters, cut after redaction. Mocks never call an upstream service, but they do not stop the agent from contacting other hosts. Use a sandbox egress policy for network isolation. ## Where mocks run Mocks apply to every variant in the suite. Put variants in separate eval files when they need different mocks. CLI mocks are installed for every harness. MCP mocks are configured for every [built-in harness](/evals/variants), but not the [command harness](/evals/command-harness). Handlers need no credentials for the service they simulate. Fixture data is bundled into the sandbox, so keep secrets out of it. Running the eval still needs `SPHYNX_API_KEY` and the harness key in [Settings > Environment](/environment). # Harness profiles Source: https://docs.sphynx.sh/evals/profiles Version the files and settings you layer on a harness A profile is the files and settings you layer onto a harness. It is part of the variant: ```ts theme={null} variants: [ { harness: "opencode", model: "anthropic/claude-sonnet-4.6" }, { harness: "opencode", model: "anthropic/claude-sonnet-4.6", profile: { name: "docs-writer", dir: "./profile" }, sandbox: "daytona", }, ] ``` The first variant is the stock harness. The second is your configured one. Each variant keeps its own runs, so you read a profile change against that profile's history. ## The directory `dir` resolves against the eval file. Sphynx reads two directories and nothing else: ``` profile/ home/ .config/opencode/opencode.json workspace/ AGENTS.md .claude/skills/review/SKILL.md system.md profile.json README.md ``` | Path | Where it lands | | - | - | | `home/**` | The sandbox home directory, after the harness writes its own credentials and config, so a profile can override them | | `workspace/**` | The workspace, after the source is cloned | Files outside those two directories are ignored unless the manifest names them, so a `README.md` can document the profile without shipping. `node_modules` and `.git` are skipped at any depth. ## The manifest `profile.json` holds what is not a file. It is optional. ```json profile.json theme={null} { "systemPrompt": "system.md", "env": { "DOCS_BASE_URL": "https://example.com" }, "variables": ["SEARCH_API_KEY"], "install": "pip install -r requirements.txt", "run": "python run.py" } ``` | Key | Meaning | | - | - | | `systemPrompt` | The agent's system prompt. See [System prompts](#system-prompts) | | `env` | Literal environment variables for every process the trial starts. Never secrets | | `variables` | Names from [Settings > Environment](/environment) the trial gets. See [secrets](#secrets) | | `install` | A command run once while the sandbox is prepared. Any harness | | `run` | The command that starts your process. [Command harness](/evals/command-harness) only, and required there | If `systemPrompt`, `install` or `run` names a file inside the profile directory, the file's content is used. Otherwise the value is used as written, so `"systemPrompt": "system.md"` ships the file and `"systemPrompt": "Be terse."` ships the sentence. A value that names a file outside the directory is an error. `env` values are always literal. `install` runs after both file stages, so it can read what the profile shipped, and before the case's prepare step. A profile that sets `run` on any harness other than `command` is rejected. Never put a key in `env`. A profile is committed beside the eval file, stored unencrypted and readable by anyone who can read its runs. ## Secrets Set a secret in [Settings > Environment](/environment), then name it in `variables`. The profile holds only the name. The value reaches the profile's process, the install command, the case's prepare step and code checks. A hosted batch whose profile names a variable that is not set is refused before it starts: ```text theme={null} A profile names variables that are not set. Set SEARCH_API_KEY in Settings > Environment. ``` A built-in harness picks up its own key, such as `ANTHROPIC_API_KEY` for `claude`, so you don't name it. A [command harness](/evals/command-harness) gets only what its profile names. Keys only a judge or sandbox uses, such as `E2B_API_KEY` or `GROQ_API_KEY`, stay on the server. A profile that names one is refused. Anyone who can run evals can name a variable in a profile and read it, so only set values every eval author is trusted with. ## Limits | Limit | Value | | - | - | | Files | 256 | | Characters per file | 96,000 | | Characters in total | 2,000,000 | | Path length | 512 characters, under `home/` or `workspace/`, no `..` segment | Files are text. A file containing a NUL byte is skipped, so a checked-in binary does not fail the compile. ## System prompts The prompt is always written into the sandbox. Each harness receives it differently: | Harness | Delivery | | - | - | | `claude` | `--append-system-prompt-file`. A profile also adds `--add-dir` on the workspace, plus `--settings` and `--mcp-config` when it ships `.claude/settings.json` or `.mcp.json`. Claude runs with `--bare`, so a `CLAUDE.md` is not discovered and must be passed this way | | `opencode` | `OPENCODE_CONFIG_CONTENT`, merging the profile's own config with the prompt's path in `instructions` | | `codex` | `-c developer_instructions=…`, carrying the text, added to the built-in prompt rather than replacing it | | `gemini`, `qwen`, `pi`, `fx`, `cursor` | Prepended to the instructions file the agent reads, `GEMINI.md` for Gemini and `AGENTS.md` for the rest | | `command` | `SPHYNX_SYSTEM_PROMPT_FILE`, holding the path | For file-based delivery, check the trajectory to confirm the harness read the instructions. ## Identity The profile **name** is part of the variant. Two profiles on the same harness, model and sandbox are two variants with separate histories, the same as two harnesses. The profile **version** is not. It is a hash of the files, system prompt, environment, install and run commands. When you edit a profile, the next batch adds a run on the same variant, and each run records the version it used. Renaming a file changes the version. Renaming the profile creates a new variant. ```ts theme={null} const [latest, previous] = ( await sphynx.evals.runs.list({ caseId: "writes-docs", variant: variantId }) ).runs; console.log(previous?.profileVersion, latest?.profileVersion); ``` ## Comparing fairly Ship the same skills to every variant. Skills live in agent-specific directories such as `.claude/skills` and `.agents/skills`, so write them under `workspace/` into every directory the compared harnesses read. Otherwise you measure the skills, not the harness. Hold the model constant. If a profile needs a provider another variant cannot use, or the model ids do not overlap, the variants are not comparable. Say so next to the results instead of reading the difference. Change one thing at a time. If the profile and the model both change between runs, you cannot tell which one moved the result. # Results Source: https://docs.sphynx.sh/evals/results Read a batch, its runs and each trial Starting evals creates a batch with one run per case per variant. Each run holds its trials. `evals.batches.start` returns the batch id and, for each case and variant, a run with its `caseId` and `variantId`, while trials continue on the server. `startAndWait` resolves once the batch stops: ```ts theme={null} const batch = await sphynx.evals.batches.startAndWait(smoke); ``` To wait on a batch you started earlier, pass its id to `evals.batches.wait({ id })`. Both poll `evals.batches.get`, backing off as the batch goes on, and both take `onProgress`, `signal`, `timeoutMs`, `pollIntervalMs` and `maxIntervalMs`. Neither cancels the batch. On timeout or abort it keeps going, and `evals.batches.get` can still read it. | Batch status | Meaning | | - | - | | `running` | At least one run has not finished | | `finished` | Every run completed, including any void trials | | `failed` | The batch stopped early. `failure` says why | Each run in `batch.runs` names its `case`, `suite` and `variant`, and has its own status, trials and distribution. ## Run distribution A run's pass rate is taken across its trials: ```ts theme={null} const { passRate, passed, failed, voided, scored, deterministic } = batch.runs[0].distribution; ``` Only scored trials count. When `scored` is zero, `passRate` is zero, which is not a result. `deterministic` is true when at least two scored trials all agree on pass or fail and their command counts are within four of each other. ## Trial status | Status | Meaning | | - | - | | `queued` | Not started | | `running` | The harness is working | | `passed` | Every check in `validate` passed | | `failed` | A check ran and did not pass | | `void` | Required evidence is missing | A void trial is an infrastructure failure, not a verdict. `voidFields` names the missing evidence where it can be identified. When a trial stopped before it could be scored, `failure` says why, such as `The agent ran past its time limit of 5m` or a setup step that failed; it is `null` for a trial that was scored. The dashboard shows the same reason on the trial. Keep void trials out of pass rates in reports and CI. Coding trials take minutes, so a batch that stays `running` is usually still working. A server restart does not resume active work, and runs abandoned for six hours are marked failed. A [local batch](/evals/local#when-the-network-drops) from a CLI that checks in is marked failed once its machine has been quiet for 10 minutes. ## Trajectory Each trial has an ordered `trajectory` of commands, messages, tool calls and file changes: ```ts theme={null} for (const event of batch.runs[0].trials[0].trajectory) { if (event._tag === "command") { console.log(event.command, event.exitCode, event.output); } } ``` Trials also record command counts, changed files, model and sandbox time, the sandbox id, and token counts when the harness reports them. Command output and tool payloads are capped at 4,000 characters, and truncation is marked. Usage carries a warning when a trial re-read a large context on every turn or got nothing from cache. Both mean the trial paid again and again for something it read once. See [what a trial costs](/guides/ci#what-a-trial-costs). Start with the distribution, then compare a passing and a failing trajectory from the same run. The pass rate tells you what happened. The command counts tell you how the attempts differed. ## Validation evidence Each trial's `validations` holds the code checks, judges and command that scored it: ```ts theme={null} for (const validation of batch.runs[0].trials[0].validations) { console.log(validation.name, validation.status, validation.durationMs); console.log(validation.input, validation.output, validation.error); } ``` Code checks record their return value, context calls and results, logs and exceptions. Judges record the prompt, raw response, provider metadata and parsed judgment. Commands record the command, stdout, stderr and exit code. `failed` means a check finished and did not pass, `error` means it could not finish, and `skipped` means it never ran. Every code check runs even when an earlier one fails or crashes, so each keeps its own result. A check is skipped only when the trial ended before it could start, or when it is a judge and the trial was already void. Runs from before capture existed have no evidence to show. Each validator gets a 64,000-character budget. Single values cap at 16,000 characters, and at most 64 calls or log entries are kept. Source snapshots cap at 100 files and 1,000,000 characters, and older runs without one show "Source unavailable." Truncation affects stored evidence, never the score. Inputs and logs can contain your data. Set `captureValidation: false` on the suite to keep verdicts and timing but store no payloads. `captureSource` controls source snapshots separately. Model reasoning is never collected. ### Secrets in evidence Sphynx does not store the value your `prepare` returns. A check can read `prepared` and pass or fail on it, and its evidence shows no input. Before evidence is stored or shown, every value, log line and message is checked for credentials. Anything that looks like a key is replaced with `[redacted]`. That covers keys starting with `sk-`, `sk_live_`, `am_sk_`, `anp_`, `ghp_`, `github_pat_`, `xox`, `AKIA` and `AIza`, JSON web tokens, private key blocks, the token after `Bearer`, and the password in a URL such as `postgres://app:password@db`. Every value of the harness and sandbox credentials the trial ran with is replaced too, even when a check reads it back from a file in the sandbox. A forwarded host variable is replaced when its name says it is secret, such as `STRIPE_SECRET_KEY`, or when its value looks like a token. A forwarded address such as `http://localhost:3000` stays readable, and only a password or token inside it is replaced. The agent's journal gets the same treatment before it is stored, streamed to the dashboard or printed in your terminal. That covers commands, their output, tool calls, messages and changed file paths. Failure reasons, such as the error output of a prepare or a clone, are redacted too. Code checks still read what the agent actually wrote, so redaction never changes their verdict. Judges read the redacted conversation, so a judge model is never sent a credential Sphynx recognizes. The files a trial saves from its workspace are redacted the same way before they are stored. Only small text files are saved, so a binary file is never saved or changed. A file whose redacted text still holds something that looks like a password assignment is not saved at all. Text is redacted before it is cut to the limits above, so a key that crosses a limit never leaves a fragment behind. A secret with no known shape, such as a plain password, is stored if a check or the agent prints it. Log a summary instead of the value. Evidence stored before redaction existed is kept as it was. If an older run recorded a key, rotate that key. ## Trigger Each batch records a `trigger` (Dashboard, CLI, CI, API or MCP), and its runs inherit it. Sphynx sets it for you. The CLI records CLI, or CI with a link to the GitHub Actions attempt when it runs there. Batches started through the API record API. The trigger describes the caller. It grants and verifies nothing. Running a case again records the new trigger, and older runs show Unknown. # Variants Source: https://docs.sphynx.sh/evals/variants The harness, model and sandbox a case runs on A variant is the harness, model and sandbox (plus an optional [profile](/evals/profiles)) a case runs on. A case can run on many variants, and comparing them, for example Codex against Claude Code, is the main reason to have more than one. ```ts theme={null} { harness: "codex", model: "gpt-5.6-sol" } ``` | Field | Purpose | | - | - | | `harness` | The agent that does the work | | `model` | The model the harness uses | | `sandbox` | Where each trial runs. Optional, defaults to `e2b` | | `profile` | Optional `{ name, dir }`: files and settings layered on the harness. See [profiles](/evals/profiles) | ## Harnesses | Agent | `harness` | Environment | Reports | | - | - | - | - | | Codex | `codex` | ChatGPT subscription or `OPENAI_API_KEY` | Commands, files, tokens | | Claude Code | `claude` | `ANTHROPIC_API_KEY` | Commands, files, tokens | | OpenCode | `opencode` | OpenCode subscription | Commands, files, tokens | | Pi | `pi` | Pi subscription | Commands, files, tokens | | Gemini CLI | `gemini` | `GEMINI_API_KEY` | Commands, files, tokens | | Qwen Code | `qwen` | `DASHSCOPE_API_KEY` | Commands, files, tokens | | Cursor Agent | `cursor` | `CURSOR_API_KEY` | Commands, files | | FX | `fx` | `AI_GATEWAY_API_KEY` | Tool names, answer | Each harness needs its key or subscription in [Settings > Environment](/environment). Harnesses that report less are scored the same way but leave less trajectory and cost data. To run your own agent, use the [command harness](/evals/command-harness). For Claude Code, follow the [setup guide](/guides/claude-code) and check one trial before comparing models. ## Models Model ids differ by harness. List the ones a harness accepts: ```ts theme={null} const { models } = await sphynx.evals.models.list({ harness: "codex" }); ``` You don't have to list models before starting a batch. Lists come from Sphynx's Codex cache, models.dev or a fixed list, depending on the harness, so an empty result means the source is unavailable, not that the harness has no models. ## Sandboxes A variant without `sandbox` runs on `e2b`. The others are `daytona`, `upstash`, `modal`, `cloudflare` and `vercel`. Every trial gets its own isolated workspace. Hosted sandboxes can't reach services on your machine. To test against one, run the suite with `sphynx eval --local`. See [local runs](/evals/local). The sandbox is part of the variant, so changing it starts a new history. If a variant ran on a sandbox other than `e2b`, keep naming it to keep adding to that history. ## Compare variants Change one field at a time: ```ts theme={null} variants: [ { harness: "codex", model: "gpt-5.6-sol" }, { harness: "codex", model: "gpt-5.6-terra" }, ] ``` Starting the suite creates a batch with one run per case per variant, so the variants sit side by side. A case has one variant for each combination of harness, model, sandbox, profile name and simulated-user model it has run on. Changing any of them creates a new variant with its own history. A new harness version stays in the same history, and each run records the version it used. Use a [profile](/evals/profiles) to compare an agent's stock and configured setups. ## Pick variants from the command line Every variant in a suite file has a label. It is `harness/model`, plus `@` and the profile name when it has a profile. `sphynx eval` prints these labels, and `--variant` takes them back: ```ts theme={null} variants: [ { harness: "codex", model: "luna", profile: { name: "autumn-setup", dir: "./autumn" } }, { harness: "codex", model: "luna" }, ] ``` | `--variant` | Runs | | - | - | | `codex/luna` | The variant with no profile | | `codex/luna@autumn-setup` | The variant with the profile | | `autumn-setup` | The variant with the profile, named by the profile alone | A label matches its own variant first. `harness/model` also picks the one variant on that model with a profile, when no variant without one exists. When a name matches no variant, or more than one, the command stops and lists the labels to choose from. Repeat `--variant` to run more than one. ## Read a variant's history ```ts theme={null} const { runs, total } = await sphynx.evals.runs.list({ caseId: "writes-hello" }); ``` Runs come newest first, 20 to a page. Pass `variant` (a variant id) to read one variant, and `page` for older runs. `evals.cases.get({ id })` lists a case's variants with their ids. To run a case again, `evals.cases.run({ id, trials, variants })` starts a batch that runs the case's newest version on the variant ids you name, or on every variant it has run on when you leave `variants` out. `trials` defaults to 1. # Quickstart Source: https://docs.sphynx.sh/getting-started Define a suite and run it on a coding agent `sphynx-sh` is the TypeScript package for defining and running evals for coding agents. Suites, mocks and validators live in your repository next to the code they test. ## Prerequisites Set your harness key, such as `OPENAI_API_KEY` for Codex, under [Settings > Environment](/environment). Then create an API key with `evals:write` and `evals:read` under **Settings > API keys**. ```bash theme={null} export SPHYNX_API_KEY="anp_..." ``` Keep keys in Settings > Environment, never in eval definitions. Hosted sandboxes and mocks need no keys of their own. ## Install ```bash npm theme={null} npm install sphynx-sh ``` ```bash bun theme={null} bun add sphynx-sh ``` ## Define a suite Create `smoke.eval.ts`: ```ts theme={null} import { command, suite } from "sphynx-sh"; export default suite({ id: "smoke", prompt: "{{task}}", cases: [ { id: "writes-hello", variables: { task: "Create hello.txt containing exactly hello" }, validate: command('test "$(cat hello.txt)" = hello'), }, ], variants: [{ harness: "codex", model: "gpt-5.6-sol" }], trials: 3, }); ``` ## Run it ```bash theme={null} npx sphynx-sh eval ./smoke.eval.ts ``` The CLI compiles the file on your machine, including imported fixtures and validators, and starts a batch. Each trial runs in its own hosted sandbox. The [GitHub Action](/guides/ci) runs the same command, and you can also [start a batch from your own code](/guides/run-from-code). ## Vocabulary * **Suite:** a system prompt and shared setup for many cases. It needs an `id`, and its `name` defaults to the id. * **Case:** one eval. It needs an `id`, and its `name` defaults to the id. It sets up a scenario and checks the result with `validate`. A case gets a new version when its definition changes. * **Variant:** the harness, model and sandbox, plus an optional profile, a case runs on. * **Run:** one case on one variant. It holds the trials. * **Trial:** one attempt, in its own sandbox. A run's pass rate is taken across its trials. * **Batch:** the runs started together. Starting evals creates a batch. The suite above has one case, one variant and three trials, so its batch holds one run with three trials. Two cases on three variants with three trials each would be six runs and 18 trials. ## Package exports | Import | Use | | - | - | | `sphynx-sh` | [`suite`, `command`, `repo`, `files`, `empty` and validator types](/evals/cases) | | `sphynx-sh` | [`Sphynx` client for batches, cases and runs](/sdk/client) | | `sphynx-sh/mcp` | [`server`, `tool`, `resource`](/evals/mocks#mcp) | | `sphynx-sh/cli` | [`cli`, `command`](/evals/mocks#cli) | | `sphynx-sh/api` | [`api`, `endpoint`, `withApi`](/evals/mocks#http-apis) | | `sphynx-sh/validators` | [`judge`](/evals/judges), [`human`, `script`](/evals/conversations) | | `sphynx-sh/eval` | [`compileEval`, `compileDefinition`](/guides/run-from-code) | Definitions are plain objects and handlers are ordinary functions. Mock inputs use Standard Schema for types and runtime checks (the examples use Zod). You don't need Effect or a separate model client. Add sources, preparation and typed validators. Give agents typed MCP servers, CLIs and HTTP APIs. # Run evals in CI Source: https://docs.sphynx.sh/guides/ci Gate pull requests on eval results Install `sphynx-sh` in your project and commit the lockfile. Name suite files `*.eval.ts` so the action can find them. Create an API key with `evals:read` and `evals:write` and save it as the `SPHYNX_API_KEY` repository secret. Use a dedicated organization for CI. In that organization, set the harness key under **Settings > Environment** for **Everyone in the organization**. API keys cannot use personal values. Provider keys stay in Sphynx, not in GitHub or the suite. ```yaml theme={null} name: Agent evals on: pull_request: branches: [main] permissions: contents: read jobs: eval: if: >- github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository runs-on: ubuntu-latest timeout-minutes: 30 steps: - uses: actions/checkout@v6 with: ref: ${{ github.event.pull_request.head.sha }} persist-credentials: false - uses: oven-sh/setup-bun@v2 - run: bun install --frozen-lockfile - uses: charlietlamb/sphynx/.github/actions/eval@COMMIT_SHA with: api-key: ${{ secrets.SPHYNX_API_KEY }} ``` Replace `COMMIT_SHA` with a reviewed action commit. The action runs the `sphynx` CLI you installed, with Node 20 or later. It does not install dependencies. Each suite file starts one batch: one run per case per variant. The job summary links each batch as soon as it starts. The dashboard labels these batches **CI** and links back to the GitHub Actions attempt, read from GitHub's environment with no extra setup. | Input | Default | Purpose | | - | - | - | | `api-key` | Required | Sphynx API key | | `working-directory` | `.` | Project with `sphynx-sh` installed | | `file` | Discover `*.eval.ts` | Run one suite file | | `case` | Every case | Run one case. With `file`, picks it from the file. Without, runs the stored case again | | `variant` | Every variant | Variants to run, separated by commas or new lines: `harness/model`, `harness/model@profile` or a profile name for a file, variant ids for a stored case | | `fail-on` | `failures` | `failures`, `strict` or `never` | | `timeout` | `1200` | Seconds to wait per batch | | `github-token` | None | Token that posts the results as a check run. Needs `checks: write` | * `failures` fails the job when any run fails or has a trial that did not pass. * `strict` also fails when a run or trial is missing or unfinished. * `never` ignores trial results. Under every gate, a batch that fails to start, fails, or times out fails the job. The step exits 0 when the gate passes, 2 when it fails, and 1 when a batch could not start or finish. | Output | Value | | - | - | | `report` | Path to a JSON file with each suite file's batch ID, batch and problems | | `batch-ids` | The batch IDs started, separated by commas | | `conclusion` | `success`, `failure`, or `neutral` when a batch was started without waiting | To keep the report, upload it with `actions/upload-artifact` and `if: always()`. ## What runs A case runs in an empty workspace unless it or its suite names a [`source`](/evals/cases#sources), such as `repo("acme/api@8f31c4a")`. The CLI does not point it at the checked-out commit for you. The checkout above uses the PR head, not GitHub's merge commit, so the suites and validators are the ones in the PR. This example skips fork and Dependabot PRs because they don't receive repository secrets. Keep unit tests in a separate job so they still run. Don't use `pull_request_target` to run untrusted suites with a secret. The job is the PR check. For a separate check named `sphynx` on the PR head commit, pass `github-token: ${{ github.token }}` and grant the job `checks: write`. Its verdict follows the selected gate. Outside the action, the same command is: ```sh theme={null} bunx --no-install sphynx eval --fail-on failures --timeout 1200 --output results.json ``` ## Keep comparisons stable When you test one change, keep the rest of the [variant](/evals/variants) the same so its history stays comparable. Pin fixture repositories to a commit. A new harness release stays on the same variant, and each run records the harness version it used. Use at least three trials for a gating check. One trial shows an outcome, not whether it repeats. ## Control the batch size ```text theme={null} trials in a batch = cases x variants x trials per run ``` Five cases, two variants and three trials open 30 sandboxes. Keep pull request grids small and run larger comparisons on a schedule. A run holds 1 to 10 trials and a batch at most 100. See [API limits](/api-reference/introduction#limits). The CLI records batch IDs before waiting and never retries a start. If CI times out or is cancelled, the batch keeps running in Sphynx. Open its link before starting another. ## What a trial costs An agent re-sends its whole conversation every turn, so anything it reads early is paid for again on every later turn. Loading a large skill, or reading a dependency to work out its API, is the usual reason one trial costs ten times another with the same number of turns. Give the agent the interface instead of letting it search. A short reference in the workspace costs less than the files it would otherwise read, and keeps cost steadier: the trials that explore are the ones whose cost varies most. Each trial reports its token counts. The dashboard flags two things to act on: context that grew fast, and nothing served from cache. A grid multiplies both, so 30 sandboxes each carrying a large context cost 30 times as much. # Run Claude Code evals Source: https://docs.sphynx.sh/guides/claude-code Set your Anthropic key once, check one trial, then compare models Sphynx runs the real Claude Code CLI in a sandbox. You don't install Claude locally or write an authentication script. ## Set your Anthropic key 1. Select the organization that will own the evals. 2. Open **Settings > Environment** and choose **Add variable**. 3. Enter `ANTHROPIC_API_KEY` with your Anthropic API key and choose **Everyone in the organization**, so CLI, SDK and CI batches can use it. 4. Save it. From a terminal, `sphynx env set ANTHROPIC_API_KEY` does the same. It needs an API key with `credentials:write`, as the [CLI guide](/tools/cli) describes. The key stays in Sphynx. A local Claude login or Claude subscription does not authenticate these trials: Sphynx uses [Claude Code's bare mode](https://code.claude.com/docs/en/headless#start-faster-with-bare-mode), which requires API credentials. Create a Sphynx API key with `evals:read` and `evals:write` in the same organization and set it as `SPHYNX_API_KEY` in your shell or CI secrets. Your eval file and GitHub workflow hold no Anthropic key. Hosted E2B sandboxes need no key of their own. ## Check one trial Install `sphynx-sh`, then create `claude.eval.ts`: ```ts theme={null} import { command, suite } from "sphynx-sh"; export default suite({ id: "claude-smoke", prompt: "Create hello.txt containing exactly hello.", cases: [ { id: "writes-hello", validate: command('test "$(cat hello.txt)" = hello'), }, ], variants: [{ harness: "claude", model: "claude-haiku-4-5-20251001" }], trials: 1, }); ``` ```sh theme={null} npx sphynx-sh eval ./claude.eval.ts ``` This starts a batch with one run: the one case on the one variant, with one trial. A passing trial confirms the key, the Claude install, its file tools and your check. Open the printed batch link to see the commands and output. One trial says nothing about how reliably the case passes. ## Compare models Keep the cases and add variants: ```ts theme={null} variants: [ { harness: "claude", model: "claude-haiku-4-5-20251001" }, { harness: "claude", model: "claude-sonnet-5" }, { harness: "claude", model: "claude-opus-5" }, ] ``` The next batch holds one run per case per variant, so the models appear side by side. Use [exact model IDs](https://platform.claude.com/docs/en/models/overview) your Anthropic account can call. Calls are billed to that account. To test tool use, add [MCP, CLI or HTTP mocks](/evals/mocks). Sphynx sets them up in each sandbox for Claude the same way as for other harnesses. A batch is capped at 100 trials (cases x variants x trials). Split a larger grid into separate suites. See [API limits](/api-reference/introduction#limits), and [CI](/guides/ci) for trial counts and the GitHub Action. ## If setup fails * **Set ANTHROPIC\_API\_KEY in Settings > Environment to run claude:** check the organization, and that the key is set for everyone in it. * **Authentication rejected:** set `ANTHROPIC_API_KEY` again with a working key. * **Model unavailable:** check the exact ID and your account's access. * **Works locally but fails in CI:** an API key uses the organization's values, not your personal ones or your local login. Compiling the suite or passing local mock tests does not prove a hosted Claude trial works. Check one trial's result before growing the grid. # Eval an eve agent Source: https://docs.sphynx.sh/guides/eve Run an agent built on eve as a command harness An agent built on [eve](https://github.com/vercel/eve) runs in Sphynx as a [command harness](/evals/command-harness). `sphynx-sh/runner/eve` starts the agent, sends it the case prompt, and turns what eve streams into the trial journal. You write no event mapping yourself. ## The profile Put the agent in the profile's `workspace`, with a `package.json` that installs `sphynx-sh` and `eve` 0.58: ``` profile/ profile.json workspace/ package.json agent/ agent.ts instructions.md tools/ create_post.ts ``` ```json profile.json theme={null} { "install": "npm install", "run": "node_modules/.bin/sphynx-eve" } ``` `sphynx-eve` starts `eve dev` in the workspace, waits until it answers, and runs one session with the case prompt. When the agent lives in a subdirectory, pass it: `node_modules/.bin/sphynx-eve apps/agent`. Read the variant's model in the agent, so each variant runs the model it names: ```ts agent/agent.ts theme={null} import { defineAgent } from "eve"; export default defineAgent({ model: process.env.SPHYNX_MODEL ?? "openai/gpt-5.5", }); ``` Set the model key as `AI_GATEWAY_API_KEY` in [Settings > Environment](/environment), never in the profile, and name it in the manifest so the agent gets it: ```json profile.json theme={null} { "install": "npm install", "run": "node_modules/.bin/sphynx-eve", "variables": ["AI_GATEWAY_API_KEY"] } ``` ## The suite ```ts theme={null} import { suite, type Validator } from "sphynx-sh"; const created: Validator = async ({ answer }) => JSON.parse(await answer()).status === "created"; export default suite({ id: "changelog-writer", name: "Changelog writer", prompt: "Write one changelog for acme/app from the last 7 days.", cases: [{ id: "changelog", name: "changelog", validate: created }], variants: [ { harness: "command", model: "openai/gpt-5.5", profile: { name: "eve-writer", dir: "./profile" }, }, ], }); ``` ## What is recorded | eve event | Journal | | - | - | | The session id | `Started`, with the variant's model | | `actions.requested` and its `action.result` | One `ToolCall` with the input, output, status and error, matched by call id | | `result.completed` | The answer validators and judges read, as JSON. The runner sends the agent's `outputSchema` with the prompt, since eve applies it only when a turn asks for it, so eve makes the model satisfy it and the result wins over any message text | | The last `message.completed` that is not a tool call | The answer when the agent has no output schema | | `step.completed` | `Usage` for that model call | | `session.waiting` or `session.completed` | `Finished` with `completed` | | `turn.failed` or `session.failed` | `Finished` with `failed:` and eve's message | | `input.requested` or `authorization.required` | `Finished` with `parked:` and what the agent asked | Other events are ignored. ## A run that did not finish never passes A failed session ends the trial unscored, and the CLI and the trial page show eve's message as the reason. A parked session records no answer, so a check that reads the answer fails, and the reason is on the `Finished` line in the journal. A tool that needs approval parks the session, because nobody is there to approve it. Turn approval off for the tools a case uses. `sphynx-eve` exits 0 when the session completed, 1 when it failed, and 3 when it parked. ## From code The pieces are exported for a script of your own: ```ts theme={null} import { createEmitter } from "sphynx-sh/runner"; import { runEve, serveEve } from "sphynx-sh/runner/eve"; const server = await serveEve({ cwd: "./agent" }); const outcome = await runEve({ emitter: createEmitter(), model: "openai/gpt-5.5", prompt: "Write one changelog.", url: server.url, }); await server.close(); ``` `reduceEve` is the mapping on its own. It takes the state and one eve event and returns the next state with the lines to print, so you can test it against a recorded stream. # Run evals from code Source: https://docs.sphynx.sh/guides/run-from-code Start a batch from your own code and act on the result Export a suite from a `*.eval.ts` file, start it from your code, and wait for the batch: ```ts theme={null} import { Sphynx } from "sphynx-sh"; import { smoke } from "./evals/smoke.eval"; const sphynx = new Sphynx(); const { id } = await sphynx.evals.batches.start(smoke); const batch = await sphynx.evals.batches.wait({ id }); if (batch.status === "failed") { throw new Error(batch.failure ?? "Eval batch failed"); } const failing = batch.runs.filter(({ trials }) => trials.some(({ status }) => status !== "passed") ); if (failing.length) process.exitCode = 1; await sphynx.dispose(); ``` `batches.start` creates a batch with one run per case per variant and returns its ID. `wait` polls until the batch is no longer `running`. Pass `timeoutMs` (default one hour) or `signal` to stop waiting. Neither cancels the batch. `batches.startAndWait` does both in one call. To run one stored case again, `evals.cases.run({ id: "writes-hello" })` starts a batch for its newest version on every variant it has run on. Pass `variants` (variant ids) to narrow it, and `trials` (default 1) for more attempts. To compile without the client, import from `sphynx-sh/eval`: `compileEval("./evals/smoke.eval.ts")` compiles a file by path, and `compileDefinition(smoke)` returns the request without sending it. See the [API client](/sdk/client). # Introduction Source: https://docs.sphynx.sh/introduction What is Sphynx? Sphynx gives you a declarative way to run evals against [Claude Code](https://claude.com/product/claude-code), [Codex](https://openai.com/codex/) and any other harness you like. You write evals, validators, judges and whatever else you need in TypeScript. These are then run in a sandbox environment of your choice. Sandboxes can also be compared against one another. ```ts hello.eval.ts theme={null} import { suite } from "sphynx-sh"; export default suite({ id: "hello", name: "hello", prompt: "Create hello.txt containing exactly hello", variants: [{ harness: "codex", model: "gpt-6-astra" }], trials: 1, cases: [ { id: "writes-hello", name: "writes hello", validate: async ({ readText }) => ({ passed: (await readText("hello.txt")) === "hello", }), }, ], }); ``` ## When to use Sphynx Sphynx is a great fit if you're building a product which uses or will itself be used by harnesses like [Claude Code](https://claude.com/product/claude-code). In this case you can't just analyze the request/response passed to a model as there's a lot more to consider. The harness is doing real work in an environment and the evals need to be aware of this. Another common use case would be to compare how harnesses are able to use your product. Lets say you ship a change to your MCP server, you want to be sure that the end user who uses it in claude code won't see any regressions in performance. These are the problems that Sphynx is designed to solve. You write TypeScript and you're easily able to evaluate the performance of your product with modern harnesses. Define and run an eval. Compose evals, mocks and validators. # API client Source: https://docs.sphynx.sh/sdk/client Start batches and read runs from TypeScript ## Create a client ```ts theme={null} import { Sphynx } from "sphynx-sh"; const sphynx = new Sphynx(); ``` The client reads `SPHYNX_API_KEY` and throws if it is missing. Pass `apiKey` or `baseUrl` to override: ```ts theme={null} const sphynx = new Sphynx({ apiKey: process.env.MY_SPHYNX_KEY, baseUrl: "http://127.0.0.1:3003", }); ``` To see which organization a key acts for: ```ts theme={null} const { organization, credential, permissions } = await sphynx.whoami(); ``` `credential` is `{ kind: "apiKey", name, start }` for a key, where `start` is its first characters, never the secret. ## Evals | Method | Returns | | - | - | | `evals.batches.start(suite)` | The new batch's id, and a run per case and variant with its `caseId` and `variantId`. Trials continue on the server | | `evals.batches.startAndWait(suite)` | The batch, once it stops | | `evals.batches.wait({ id })` | A batch you already started, once it stops | | `evals.batches.get({ id })` | One batch with its runs and trials | | `evals.batches.list({ cursor?, limit? })` | A page of batches, newest first, and the next cursor | | `evals.cases.list({ suite?, tag?, cursor?, limit? })` | A page of cases, with each variant's newest run | | `evals.cases.get({ id })` | A case, its versions and the variants it has run on | | `evals.cases.run({ id, trials?, variants? })` | A new batch that runs the case's newest version on the variant ids named, or on every variant it has run on. `trials` defaults to 1 | | `evals.runs.list({ caseId, variant?, page? })` | A page of a case's runs, newest first | | `evals.runs.get({ id })` | One run and its trials | | `evals.models.list({ harness })` | Models available to a harness | Validators, preparation, profiles and mocks are written in an eval file. Import the suite and pass it in: ```ts evals/smoke.eval.ts theme={null} export const smoke = suite({ ... }); ``` ```ts theme={null} import { smoke } from "./evals/smoke.eval"; const batch = await sphynx.evals.batches.startAndWait(smoke); ``` `suite` records the file it was defined in, so `start` and `startAndWait` can compile it and ship its validators and mock servers, with everything they close over, to the sandbox. The client can't start a batch on the `local` sandbox. Use `sphynx eval --local` for that. ## Errors and cleanup ```ts theme={null} import { SphynxError } from "sphynx-sh"; try { await sphynx.evals.batches.get({ id: "missing" }); } catch (error) { if (error instanceof SphynxError) { console.error(error.status, error.message); } } await sphynx.dispose(); ``` Invalid request fields fail before any network call. API failures carry an HTTP status when one is available. When Sphynx cannot be reached, the error says so and names the address it tried. The first request checks that the server answers and gives up within 5 seconds if it does not. After that, a read that gets no answer in 30 seconds gives up with an error that says the server was slow, not unreachable. A change is never cut off once the server has it, so it cannot be applied twice. Date fields are Effect `DateTime.Utc` values. Read `epochMillis` for a number. # Sphynx CLI Source: https://docs.sphynx.sh/tools/cli Run eval suites and set keys from a terminal ## Install ```bash npx theme={null} npx sphynx-sh --help ``` ```bash global theme={null} npm install -g sphynx sphynx --help ``` Set `SPHYNX_API_KEY` before commands that call Sphynx. `--help`, `--version`, `eval import` and `eval --local` work without it. ## Commands | Command | Purpose | | - | - | | `sphynx eval [file] [options]` | Run suite files, or a stored case, as batches and gate on the results | | `sphynx eval import --format [--out ]` | Turn a case file a team already wrote into a suite | | `sphynx env ` | Set the keys and variables evals use | | `sphynx whoami [--json]` | Show the organization and key `SPHYNX_API_KEY` acts for | | `sphynx --completions ` | Print completions for zsh, bash or fish | ## Check which organization a key uses ```bash theme={null} sphynx whoami ``` ```text theme={null} organization Acme organization slug acme organization id org_01J9... key ci key starts with anp_AbCd permissions evals:read, evals:write ``` Every run, suite and variable a key touches belongs to this organization. `--json` prints the same facts as JSON. The secret is never printed. `sphynx eval` also prints the organization once before its first batch. ## Run evals ```bash theme={null} sphynx eval sphynx eval ./evals/smoke.eval.ts sphynx eval ./evals/smoke.eval.ts --case writes-hello --variant codex/gpt-5.6-sol sphynx eval --case writes-hello ``` With no file, the CLI finds every `*.eval.ts` under the current directory (skipping `node_modules`, `dist`, `build` and hidden folders) and runs them one after another. Each suite file starts one batch: one run per case per variant, each with the suite's number of trials. The command waits for the batch and applies the gate. In an interactive terminal the grid updates as trials finish. Otherwise it prints the final grid once. With `--case` and no file, the CLI runs a case Sphynx already stores again, on its newest version. | Option | Purpose | | - | - | | `--case ` | Run one case. With a file, picks that case from it. Without one, runs the stored case again | | `--variant ` | Run one variant: `harness/model`, `harness/model@profile` or a profile name for a file, or a variant id for a stored case. Repeat it for more | | `--local` | Run every case on every variant, with the suite's trials, on this machine instead of a cloud sandbox. See [local runs](/evals/local) | | `--fail-on` | Gate, `failures` by default: fail when any run fails or has a trial that did not pass. `strict` also fails when a run or trial is missing or unfinished. `never` ignores trial results | | `--timeout` | Seconds to wait per batch, default `1200`. A timeout fails the command but does not cancel the batch | | `--output` | Write a JSON report with each file's batch ID, batch and problems | | `--no-wait` | Start each batch and print its ID and runs, without waiting | | `--json` | Print each finished batch as JSON instead of the grid | Under every gate, a batch that fails to start, fails, or times out fails the command. `sphynx eval import` reads an existing case file (or a directory of them) and prints a suite, or writes it with `--out`. It converts the mechanical assertions and leaves prose ones failing until someone writes the check. It reports how many of each on stderr. See [importing a case file](/evals/import). ## Environment variables `sphynx env` manages the keys and variables in [Settings > Environment](/environment). It needs a key with `credentials:read` to list and `credentials:write` to change. | Command | Purpose | | - | - | | `sphynx env list [--json]` | List variables and subscriptions. Secrets show only a preview | | `sphynx env set NAME [--plain] [--personal]` | Set one variable. `--plain` keeps the value readable. `--personal` sets it only for you | | `sphynx env import [file] [--personal]` | Set every variable in a `.env` file, or in stdin when the file is left out | | `sphynx env remove NAME [--personal]` | Remove a variable | `set` prompts for the value, or reads it from stdin when stdin is not a terminal (a pipe, a file or a heredoc), so the secret is never a command line argument. Pipe it from a variable or a file rather than typing it into a heredoc, which your shell may keep in its history: ```bash theme={null} echo "$KEY" | sphynx env set ANTHROPIC_API_KEY ``` Subscriptions, such as ChatGPT for Codex, are added in the dashboard. ## Exit codes | Code | Meaning | | - | - | | `0` | Every batch passed the gate | | `2` | The gate failed | | `1` | An error, such as a batch that could not start or finish | | `130` | Interrupted | Failures print without a stack trace. Status messages go to stderr. When Sphynx cannot be reached, the error says so and names the address it tried. The first request checks that the server answers and gives up within 5 seconds if it does not. After that, a read that gets no answer in 30 seconds gives up with an error that says the server was slow, not unreachable. A change is never cut off once the server has it, so it cannot be applied twice. ## Environment | Variable | Purpose | | - | - | | `SPHYNX_API_KEY` | API key | | `SPHYNX_BASE_URL` | API origin, default `https://api.sphynx.sh` | | `SPHYNX_WEB_URL` | Origin for batch links, default `https://www.sphynx.sh` | | `GITHUB_TOKEN` | Optional. With `GITHUB_REPOSITORY` set, `sphynx eval` posts a check named `sphynx` on the commit (the PR head on pull requests) | # Sphynx MCP Source: https://docs.sphynx.sh/tools/mcp Start and read evals from an MCP client Connect to the hosted server: ```json Claude Desktop theme={null} { "mcpServers": { "sphynx": { "url": "https://mcp.sphynx.sh/mcp" } } } ``` ```bash Claude Code theme={null} claude mcp add --transport http sphynx https://mcp.sphynx.sh/mcp ``` The server uses OAuth 2.1. Your browser opens to sign in and approve access, so the MCP configuration holds no API key. Reading evals needs `evals:read`. Starting a batch needs `evals:write`. ## Eval tools | Tool | Purpose | | - | - | | `start_suite_batch` | Start a batch: every case of a suite on every variant | | `start_case_batch` | Start a batch that runs a stored case's newest version on the variant ids named, or on every variant it has run on | | `get_batch` | Read a batch with its runs and their trials | | `list_batches` | List batches, newest first | | `list_cases` | List cases, optionally in one suite or with one tag | | `get_case` | Read a case, its versions and the variants it has run on, with their ids | | `list_runs` | List a case's runs, newest first, optionally on one variant | | `get_run` | Read one run and its trials | | `list_models` | List the models a harness can use | `start_suite_batch` and `start_case_batch` return the batch ID and one run ID per case per variant right away. Poll `get_batch` until its `status` is no longer `running`. `start_case_batch` runs 1 trial unless you pass `trials`. `list_batches` and `list_cases` return `next`. Pass it as `cursor` to read the following page.