API client

Create a client

import { Sphynx } from "sphynx-sh";

const sphynx = new Sphynx();

The client reads SPHYNX_API_KEY and throws if it is missing. Pass apiKey or baseUrl to override:

const sphynx = new Sphynx({
  apiKey: process.env.MY_SPHYNX_KEY,
  baseUrl: "http://127.0.0.1:3003",
});

To see which organization a key acts for:

const { organization, credential, permissions } = await sphynx.whoami();

credential is { kind: "apiKey", name, start } for a key, where start is its first characters, never the secret.

Evals

Method	Returns
`evals.batches.start(suite)`	The new batch’s id, and a run per case and variant with its `caseId` and `variantId`. Trials continue on the server
`evals.batches.startAndWait(suite)`	The batch, once it stops
`evals.batches.wait({ id })`	A batch you already started, once it stops
`evals.batches.get({ id })`	One batch with its runs and trials
`evals.batches.list({ cursor?, limit? })`	A page of batches, newest first, and the next cursor
`evals.cases.list({ suite?, tag?, cursor?, limit? })`	A page of cases, with each variant’s newest run
`evals.cases.get({ id })`	A case, its versions and the variants it has run on
`evals.cases.run({ id, trials?, variants? })`	A new batch that runs the case’s newest version on the variant ids named, or on every variant it has run on. `trials` defaults to 1
`evals.runs.list({ caseId, variant?, page? })`	A page of a case’s runs, newest first
`evals.runs.get({ id })`	One run and its trials
`evals.models.list({ harness })`	Models available to a harness

Validators, preparation, profiles and mocks are written in an eval file. Import the suite and pass it in:

evals/smoke.eval.ts

export const smoke = suite({ ... });

import { smoke } from "./evals/smoke.eval";

const batch = await sphynx.evals.batches.startAndWait(smoke);

suite records the file it was defined in, so start and startAndWait can compile it and ship its validators and mock servers, with everything they close over, to the sandbox. The client can’t start a batch on the local sandbox. Use sphynx eval --local for that.

Errors and cleanup

import { SphynxError } from "sphynx-sh";

try {
  await sphynx.evals.batches.get({ id: "missing" });
} catch (error) {
  if (error instanceof SphynxError) {
    console.error(error.status, error.message);
  }
}

await sphynx.dispose();

Invalid request fields fail before any network call. API failures carry an HTTP status when one is available. When Sphynx cannot be reached, the error says so and names the address it tried. The first request checks that the server answers and gives up within 5 seconds if it does not. After that, a read that gets no answer in 30 seconds gives up with an error that says the server was slow, not unreachable. A change is never cut off once the server has it, so it cannot be applied twice. Date fields are Effect DateTime.Utc values. Read epochMillis for a number.

Getting started

Define evals

Run evals

Create a client

Evals

Errors and cleanup

​Create a client

​Evals

​Errors and cleanup

Create a client

Evals

Errors and cleanup