Import a case file

sphynx eval import converts an existing case file into a *.eval.ts suite.

sphynx eval import --format evals-json ./evals.json --out ./release-notes.eval.ts

Without --out, the suite prints to stdout. A summary of converted and unconverted assertions goes to stderr.

`--format`	Input
`evals-json`	One JSON file holding every case
`yaml`	One YAML file per case, or a directory of them

The generated suite has a placeholder variant (codex on gpt-5.6-sol) and trials: 3, with each case’s text passed through a task variable.

evals-json

{
  "skill_name": "release notes",
  "evals": [
    {
      "id": 1,
      "name": "Lists the breaking change",
      "prompt": "Read CHANGELOG.md and write the release notes.",
      "expected_output": "A short note naming the breaking change.",
      "files": ["fixtures/changelog"],
      "assertions": [
        {
          "text": "names the removed flag",
          "kind": "content_contains_all",
          "needles": ["--legacy", "removed"]
        }
      ]
    }
  ]
}

skill_name and evals are required, and each case needs id, prompt and assertions. An assertion is a structured object or a prose string.

`kind`	Generated check
`content_contains_any`	`containsAny(answer, needles)`
`content_contains_all`	`containsAll(answer, needles)`
`content_contains_none`	`containsNone(answer, needles)`

Matching is case-insensitive. Prose assertions cannot be converted safely, so each becomes a placeholder that always fails:

validate: async (context) => {
  const answer = await context.answer();

  return [
    /* Write this check, then delete the line under it: */
    /* The summary reads as though a maintainer wrote it. */
    unwritten("The summary reads as though a maintainer wrote it."),
  ].every(Boolean);
},

expected_output becomes a comment, not a check. files names directories but not their contents, so each case starts with files({}) and a comment naming them.

yaml

A directory imports every *.yaml and *.yml file in it, sorted by name.

sphynx eval import --format yaml ./cases --out ./browsing.eval.ts

name: Checkout flow
task: Open the demo storefront and pay with the test card
judge_context:
  - Reach the basket page before paying
  - Use the test card, never a real one
max_steps: 12

name and task are required. The task becomes the prompt. Each judge_context entry needs human judgment, so it becomes a failing placeholder. A case without judge_context gets one placeholder asking what a good answer is. max_steps becomes a comment, because Sphynx does not cap agent steps. YAML cases start with files({}) because the format carries no source files.

Finish the suite

Review the generated file.
Replace each files({}) with fixture files or a repository source.
Set the harness, model and sandbox for each variant.
Replace every unwritten(...) placeholder with a real check.
Run it with sphynx eval ./suite.eval.ts.

An invalid file fails with its path, case and field. A directory is never partially imported.

Fundamentals

Workflows

Advanced evals

Tools

Import a case file

evals-json

yaml

Finish the suite

​evals-json

​yaml

​Finish the suite

evals-json

yaml

Finish the suite