Quickstart

​

Prerequisites

export SPHYNX_API_KEY="anp_..."

​

Install

npm install sphynx-sh

bun add sphynx-sh

​

Define a suite

import { command, suite } from "sphynx-sh";

export default suite({
  id: "smoke",
  prompt: "{{task}}",
  cases: [
    {
      id: "writes-hello",
      variables: { task: "Create hello.txt containing exactly hello" },
      validate: command('test "$(cat hello.txt)" = hello'),
    },
  ],
  variants: [{ harness: "codex", model: "gpt-5.6-sol" }],
  trials: 3,
});

​

Run it

npx sphynx-sh eval ./smoke.eval.ts

​

Vocabulary

• Suite: a system prompt and shared setup for many cases. It needs an id, and its name defaults to the id.
• Case: one eval. It needs an id, and its name defaults to the id. It sets up a scenario and checks the result with validate. A case gets a new version when its definition changes.
• Variant: the harness, model and sandbox, plus an optional profile, a case runs on.
• Run: one case on one variant. It holds the trials.
• Trial: one attempt, in its own sandbox. A run’s pass rate is taken across its trials.
• Batch: the runs started together. Starting evals creates a batch.

​

Package exports