# Reported issues for evals-mcp-server Pod holds 9 of 9 problems reported by people outside the maintainer team. Issues filed by the project's own owners, members and collaborators are excluded entirely — a maintainer's release checklist is not a warning to a prospective user. Back to [evals-mcp-server](/mcp/evals-mcp-server). ## Most discussed ### test(evals_revise_draft): mcq_choice_mismatch path not covered \`evals_revise_draft\` declares \`mcq_choice_mismatch\` in its typed error contract (alongside \`task_type_constraint\`), and the handler calls the shared \`assertTaskTypeConstraints\` after each patch — so patching \`grader.correct\` to a value not in \`choices[]\` should fire \`mcq_choice_mismatch\`. There is no test that exercises this path. ## Proposal Add a test (or test case in \`tests/authoring-loop.test.ts\` or a new \`tests/tools/revise-draft.tool.test.ts\`) that: 1. Creates an mcq d [Read the thread](https://github.com/cyanheads/evals-mcp-server/issues/4) · 2026-06-28 · closed · 2 comments ### ux(evals_get_record): description is meta-coaching — names the caller, not the action The current `evals_get_record` description reads: > "Read a draft or submitted eval record by id. The verification subagent calls this to inspect a draft before re-deriving or looking up the gold. The id is stable across submit, so it resolves a record whether it is still a draft or already submitted." The second sentence names a specific caller type ("verification subagent") and workflow step ("before re-deriving or looking up the gold") — that is meta-coaching, not a description of what the [Read the thread](https://github.com/cyanheads/evals-mcp-server/issues/3) · 2026-06-25 · closed · 1 comment ### feat(config): default EVALS_DATA_DIR to ./evals-data so bare npx invocations work `EVALS_DATA_DIR` is required with no default — omitting it aborts startup with a config error. `.env.example` already lists `./evals-data` as the natural value. A hardcoded default would let `npx`/bare invocations work without any env setup, lowering the barrier for first use. ## Proposal Apply `./evals-data` as the fallback in `src/config/server-config.ts` when `EVALS_DATA_DIR` is unset. Keep the existing `.min(1)` guard — it rejects an empty string (`EVALS_DATA_DIR=`) with a clear validation [Read the thread](https://github.com/cyanheads/evals-mcp-server/issues/2) · 2026-06-25 · closed · 1 comment ### bug(evals_create_draft): mcq grader correct-not-in-choices fires task_type_constraint instead of mcq_choice_mismatch ### Server version 0.1.0 ### Runtime Bun ### Transport HTTP / stdio ### Description `evals_create_draft` fires `reason: "task_type_constraint"` when the `mcq` grader's `correct` value is not in `choices[]`, but the error contract documents this case as `reason: "mcq_choice_mismatch"`. `evals_run_check` correctly returns `mcq_choice_mismatch` for the same condition — the two tools are inconsistent. ### Steps to reproduce 1. Call `evals_create_draft` with `task_type: "mcq"`, `choices: ["P [Read the thread](https://github.com/cyanheads/evals-mcp-server/issues/1) · 2026-06-25 · closed · 1 comment ### bug(evals_revise_draft): unknown set path reports false success ### Server version 0.1.2 ### mcp-ts-core version 0.12.3 ### Runtime Bun ### Runtime version Bun 1.4.0 ### Transport HTTP (Streamable HTTP) ### Description `evals_revise_draft` reports an unknown `set` path as successfully applied even though schema validation removes it from the stored record. The success response makes the mutation appear effective when it was a no-op. ### Steps to reproduce 1. Create a valid draft with `evals_create_draft`. 2. Call `evals_revise_draft` with `{ "dr [Read the thread](https://github.com/cyanheads/evals-mcp-server/issues/9) · 2026-08-22 · open · 0 comments ### fix(config): pin MCP_SESSION_MODE to stateful — the image ships stateless and breaks the confirmation gate for v1 HTTP clients `Dockerfile` sets `ENV MCP_SESSION_MODE="stateless"`, but `.env.example` carries a commented `# MCP_SESSION_MODE=stateful`, whose comment documents the value set as `stateful | stateless (default: stateful)`. The container and the same code run via `bunx`, `npm start`, or from source therefore resolve to different session modes. For this server the container is the one that is wrong. Related: cyanheads/mcp-ts-core#376 ## Proposal Settle every surface on `stateful`, and change the `Dockerfile [Read the thread](https://github.com/cyanheads/evals-mcp-server/issues/8) · 2026-08-22 · open · 0 comments ### bug(evals_submit_draft): llm_rubric/free_response is never judged — ctx.sample does not exist, descriptions over-claim LLM grading ### Server version 0.1.2 ### mcp-ts-core version 0.10.9 ### Runtime Bun ### Runtime version Bun 1.3.x ### Transport stdio ### Description `free_response` / `llm_rubric` is documented across the surface as "judged via `ctx.sample` when the client supports sampling" — but no such grading ever happens, and `ctx.sample` does not exist. - `evals_submit_draft` calls `runSubmitGate(..., { samplingAvailable: false })` unconditionally and never inspects sampling availability. - `@cyanheads/mc [Read the thread](https://github.com/cyanheads/evals-mcp-server/issues/7) · 2026-06-30 · open · 0 comments ### bug(evals_create_draft): persists an orphan draft when the grader is unexecutable ### Server version 0.1.2 ### mcp-ts-core version 0.10.9 ### Runtime Bun ### Runtime version Bun 1.3.x ### Transport stdio ### Description `evals_create_draft` persists the draft to disk (`store.writeDraft`) **before** running the self-consistency check. `checkSelfConsistency` calls `gradeCandidate`, which throws `grader_unexecutable` for a malformed grader (non-evaluable math.js target, invalid regex, `json_match` with neither `expected` nor `schema`). The throw becomes the tool's err [Read the thread](https://github.com/cyanheads/evals-mcp-server/issues/6) · 2026-06-30 · open · 0 comments ## Most recent ### bug(grader): numeric grader accepts complex/unit candidates by truncating at the first space ### Server version 0.1.2 ### mcp-ts-core version 0.10.9 ### Runtime Bun ### Runtime version Bun 1.3.x (logic bug — runtime-agnostic) ### Transport stdio ### Description The `numeric` grader coerces a [math.js](https://mathjs.org/) result to a plain number with `Number(result.toString().replace(/\s.*$/, ''))` (`candidateToNumber` and `resolveNumber` in `src/services/grader/grader-service.ts`). For any non-real result — a math.js `Complex` or `Unit` — `toString()` yields e.g. `"2 + 3i"` [Read the thread](https://github.com/cyanheads/evals-mcp-server/issues/5) · 2026-06-30 · open · 0 comments The remaining reports are on [the project's issue tracker](https://github.com/cyanheads/evals-mcp-server/issues).