Mocks

Mocks are TypeScript objects. Their schemas type your handlers and validate requests and responses at runtime. Each trial gets its own copy of every MCP server, CLI and HTTP API. Keep fixtures in shared files and assertions in validators.

npm install sphynx-sh zod

Keep fixture data and schemas separate from each interface:

fixtures/users.ts

import { z } from "zod";

export const User = z.object({ id: z.string(), name: z.string() });
export const GetUserInput = z.object({ id: z.string() });

export const user = {
  id: "user_fixture",
  name: "Ada",
} satisfies z.infer<typeof User>;

export const getUser = (id: string) => {
  if (id !== user.id) throw new Error(`User not found: ${id}`);
  return user;
};

MCP

mcp/users.ts

import { resource, server, tool } from "sphynx-sh/mcp";
import { GetUserInput, User, getUser, user } from "../fixtures/users";

export const usersMcp = server({
  name: "users",
  version: "1.0.0",
  tools: [
    tool({
      name: "users_get",
      inputSchema: GetUserInput,
      outputSchema: User,
      handler: ({ id }) => getUser(id),
    }),
  ],
  resources: [
    resource({
      name: "current_user",
      uri: "users://current",
      outputSchema: User,
      handler: () => user,
    }),
  ],
});

MCP mocks support tools and fixed-URI resources. Schemas must implement Standard Schema and expose JSON Schema. The examples use Zod 4.

CLI

cli/users.ts

import { cli, command } from "sphynx-sh/cli";
import { GetUserInput, User, getUser } from "../fixtures/users";

export const usersCli = cli({
  path: "users",
  version: "1.0.0",
  commands: [
    command({
      path: ["get"],
      inputSchema: GetUserInput,
      outputSchema: User,
      options: { id: { description: "User ID", type: "string" } },
      handler: ({ id }) => getUser(id),
    }),
  ],
});

cli.path is the executable name. command.path is a non-empty list of argv segments.
Every input key needs an options entry. Its flag is the key in kebab case. Set name (with the leading --) to override it.
String options accept --id value or --id=value. Boolean options are true when present, so make them optional or give them a schema default.
Positional arguments are not supported.

The runtime generates help and version output, validates output, prints JSON to stdout and exits nonzero on failure. sphynx-sh also exports a command for validate. If one file needs both, import one under another name.

Attach and validate

Attach MCP and CLI mocks on the suite, then check the call logs in a validator.

users.eval.ts

import { suite, type Validator } from "sphynx-sh";
import { usersCli } from "./cli/users";
import { GetUserInput, user } from "./fixtures/users";
import { usersMcp } from "./mcp/users";

const requestedFixture = (input: unknown) => {
  const result = GetUserInput.safeParse(input);
  return result.success && result.data.id === user.id;
};

const validate: Validator = async ({ answer, cli, mcp }) => {
  const [text, cliCalls, mcpCalls] = await Promise.all([
    answer(),
    cli.calls("users"),
    mcp.calls("users"),
  ]);
  const called =
    cliCalls.some(({ input }) => requestedFixture(input)) ||
    mcpCalls.some(
      ({ input, kind, name }) =>
        kind === "tool" && name === "users_get" && requestedFixture(input)
    );
  return { passed: called && text.includes(user.name) };
};

export default suite({
  id: "users-interface",
  prompt: "Use an available interface to retrieve user_fixture and report its name.",
  mcp: [usersMcp],
  cli: [usersCli],
  cases: [{ id: "gets-user", validate }],
  variants: [{ harness: "codex", model: "model-id" }],
  trials: 1,
});

Check both the recorded call and the outcome. Call inputs and outputs are unknown, so parse them with the shared schema before asserting on values.

HTTP APIs

Define endpoints with api() and endpoint(). Standard Schema types the handler input and checks each response against the schema for its status code. Handlers are plain TypeScript.

mocks/catalog.ts

import { api, endpoint } from "sphynx-sh/api";
import { z } from "zod";

const item = { id: "fixture", name: "Test item" };
const Item = z.object({ id: z.string(), name: z.string() });

export const catalog = api({
  name: "catalog",
  endpoints: [
    endpoint({
      method: "GET",
      path: "/items/:id",
      inputSchema: z.object({ params: z.object({ id: z.string() }) }),
      responses: {
        200: Item,
        404: z.object({ message: z.string() }),
      },
      handler: ({ params }) => params.id === item.id
        ? { status: 200, body: item }
        : { status: 404, body: { message: "Item not found" } },
    }),
  ],
});

Move item and Item into a shared fixture file to reuse them with MCP and CLI mocks. API names use lowercase letters, digits and hyphens.

Attach an API

catalog.eval.ts

import { suite, type Validator } from "sphynx-sh";
import { catalog } from "./mocks/catalog";

const validate: Validator = async ({ api, answer }) => ({
  passed: (await api.calls("catalog")).some(
    ({ path, status }) => path === "/items/fixture" && status === 200
  ) && (await answer()).includes("Test item"),
});

export default suite({
  id: "catalog-api",
  api: [catalog],
  prompt: "Use the local catalog API to retrieve fixture and report its name.",
  cases: [{ id: "retrieve-item", validate }],
  variants: [{ harness: "codex", model: "model-id" }],
  trials: 1,
});

In each trial, Sphynx starts a loopback HTTP server on a free port and adds its base URL and endpoint list to the agent’s input. Your prompt is unchanged. Curl, Python and SDK clients can call it directly. Prepare functions and validators get the URL from await api.url("catalog"). No environment variables or global fetch are replaced.

Requests and responses

The input schema receives { params, query, headers, body }. Headers are lowercase. Repeated query values are arrays. An empty body is null, and a nonempty body must be JSON. Use schema transforms for coercion.
Handlers return { status, body, headers? }. Use null for bodyless responses such as 204. The runtime sets content type and framing headers.
An optional endpoint description is shown to the agent.
Error responses you declare, such as 404 or 429, are normal mock behavior. Unmatched routes return 404 and are recorded with matched: false.
A handler that throws, a response that fails its schema, or a body that cannot be serialized fails the mock. Nothing is forwarded upstream.

Handlers take a second argument, { signal, log }. Call log(value) to attach evidence to the call, and use signal to cancel async work.

Evidence

HTTP calls appear in the trial’s Calls view and are available to judges. api.calls(name?) returns records with method, path, status, timing, whether a route matched, input, output, handler logs and errors. Calls to api.url and api.calls appear in the validator’s evidence. Input, output and log values are bounded snapshots with text, format, state and truncated fields. Check those before parsing captured JSON. Authorization headers, cookies and common secret fields are redacted before recording. Add more field names (case-insensitive) with api({ redact: ["accessToken"], ... }). Before a call is stored in the trial, anything that looks like a key and every secret the trial knows, such as its harness and sandbox credentials, are also replaced with [redacted], as described in secrets in evidence. A secret with no known shape inside a string is still recorded, so keep credentials out of fixtures.

Validator-owned servers

Use withApi() when a validator needs its own server with hidden test data. The server closes when the callback finishes or fails.

import { withApi } from "sphynx-sh/api";

await withApi({
  api: catalog,
  run: async ({ url, calls }) => {
    const response = await fetch(`${url}/items/fixture`);
    console.info("HTTP evidence", await calls());
    return response.json();
  },
});

These calls are returned by calls(), not the trial’s call log. Log them inside the validator to include them in its logs.

Limits

JSON HTTP endpoints only. No proxying, streaming, WebSockets or automatic recording.
Request bodies up to 1 MiB, handlers up to 30 seconds, and 256 requests per mock runtime.
api.calls() inside a validator returns values up to 64,000 characters. The trial stores each value at up to 16,000 characters, cut after redaction.

Mocks never call an upstream service, but they do not stop the agent from contacting other hosts. Use a sandbox egress policy for network isolation.

Where mocks run

Mocks apply to every variant in the suite. Put variants in separate eval files when they need different mocks. CLI mocks are installed for every harness. MCP mocks are configured for every built-in harness, but not the command harness. Handlers need no credentials for the service they simulate. Fixture data is bundled into the sandbox, so keep secrets out of it. Running the eval still needs SPHYNX_API_KEY and a model connection.

Getting started

Define evals

Run evals

MCP

CLI

Attach and validate

HTTP APIs

Attach an API

Requests and responses

Evidence

Validator-owned servers

Limits

Where mocks run

​Share fixtures

​MCP

​CLI

​Attach and validate

​HTTP APIs

​Attach an API

​Requests and responses

​Evidence

​Validator-owned servers

​Limits

​Where mocks run

Share fixtures

MCP

CLI

Attach and validate

HTTP APIs

Attach an API

Requests and responses

Evidence

Validator-owned servers

Limits

Where mocks run