Skip to main content
Sphynx runs the real Claude Code CLI in a sandbox. You don’t install Claude locally or write an authentication script.

​
Set your Anthropic key

  1. Select the organization that will own the evals.
  2. Open Settings > Environment and choose Add variable.
  3. Enter ANTHROPIC_API_KEY with your Anthropic API key and choose Everyone in the organization, so CLI, SDK and CI batches can use it.
  4. Save it.
From a terminal, sphynx env set ANTHROPIC_API_KEY does the same. It needs an API key with credentials:write, as the CLI guide describes. The key stays in Sphynx. A local Claude login or Claude subscription does not authenticate these trials: Sphynx uses Claude Code’s bare mode, which requires API credentials. Create a Sphynx API key with evals:read and evals:write in the same organization and set it as SPHYNX_API_KEY in your shell or CI secrets. Your eval file and GitHub workflow hold no Anthropic key. Hosted E2B sandboxes need no key of their own.

​
Check one trial

Install sphynx-sh, then create claude.eval.ts:
This starts a batch with one run: the one case on the one variant, with one trial. A passing trial confirms the key, the Claude install, its file tools and your check. Open the printed batch link to see the commands and output. One trial says nothing about how reliably the case passes.

​
Compare models

Keep the cases and add variants:
The next batch holds one run per case per variant, so the models appear side by side. Use exact model IDs your Anthropic account can call. Calls are billed to that account. To test tool use, add MCP, CLI or HTTP mocks. Sphynx sets them up in each sandbox for Claude the same way as for other harnesses. A batch is capped at 100 trials (cases x variants x trials). Split a larger grid into separate suites. See API limits, and CI for trial counts and the GitHub Action.

​
If setup fails

  • Set ANTHROPIC_API_KEY in Settings > Environment to run claude: check the organization, and that the key is set for everyone in it.
  • Authentication rejected: set ANTHROPIC_API_KEY again with a working key.
  • Model unavailable: check the exact ID and your account’s access.
  • Works locally but fails in CI: an API key uses the organization’s values, not your personal ones or your local login.
Compiling the suite or passing local mock tests does not prove a hosted Claude trial works. Check one trial’s result before growing the grid.