Run evals from code

Export a suite from a *.eval.ts file, start it from your code, and wait for the batch:

import { Sphynx } from "sphynx-sh";
import { smoke } from "./evals/smoke.eval";

const sphynx = new Sphynx();
const { id } = await sphynx.evals.batches.start(smoke);
const batch = await sphynx.evals.batches.wait({ id });

if (batch.status === "failed") {
  throw new Error(batch.failure ?? "Eval batch failed");
}

const failing = batch.runs.filter(({ trials }) =>
  trials.some(({ status }) => status !== "passed")
);

if (failing.length) process.exitCode = 1;
await sphynx.dispose();

batches.start creates a batch with one run per case per variant and returns its ID. wait polls until the batch is no longer running. Pass timeoutMs (default one hour) or signal to stop waiting. Neither cancels the batch. batches.startAndWait does both in one call. To run one stored case again, evals.cases.run({ id: "writes-hello" }) starts a batch for its newest version on every variant it has run on. Pass variants (variant ids) to narrow it, and trials (default 1) for more attempts. To compile without the client, import from sphynx-sh/eval: compileEval("./evals/smoke.eval.ts") compiles a file by path, and compileDefinition(smoke) returns the request without sending it. See the API client.

Getting started

Define evals

Run evals

Run evals from code