Pod

Yes, this is on purpose. Pod is built for agents, so the default page is plain HTML.
Human? View the normal website.
Agent? You probably prefer /mcp/evals-mcp-server/issues.md or /mcp/evals-mcp-server/issues.json, or Pod over MCP.

Reported issues for evals-mcp-server

Pod holds 9 of 9 problems reported by people outside the maintainer team. Issues filed by the project's own owners, members and collaborators are excluded entirely — a maintainer's release checklist is not a warning to a prospective user.

Back to evals-mcp-server.

Most discussed

test(evals_revise_draft): mcq_choice_mismatch path not covered

`evals_revise_draft` declares `mcq_choice_mismatch` in its typed error contract (alongside `task_type_constraint`), and the handler calls the shared `assertTaskTypeConstraints` after each patch — so patching `grader.correct` to a value not in `choices[]` should fire `mcq_choice_mismatch`. There is no test that exercises this path.

Proposal

Add a test (or test case in `tests/authoring-loop.test.ts` or a new `tests/tools/revise-draft.tool.test.ts`) that:

  1. Creates an mcq d

Read the thread · 2026-06-28 · closed · 2 comments

ux(evals_get_record): description is meta-coaching — names the caller, not the action

The current evals_get_record description reads:

"Read a draft or submitted eval record by id. The verification subagent calls this to inspect a draft before re-deriving or looking up the gold. The id is stable across submit, so it resolves a record whether it is still a draft or already submitted."

The second sentence names a specific caller type ("verification subagent") and workflow step ("before re-deriving or looking up the gold") — that is meta-coaching, not a description of what the

Read the thread · 2026-06-25 · closed · 1 comment

feat(config): default EVALS_DATA_DIR to ./evals-data so bare npx invocations work

EVALS_DATA_DIR is required with no default — omitting it aborts startup with a config error. .env.example already lists ./evals-data as the natural value. A hardcoded default would let npx/bare invocations work without any env setup, lowering the barrier for first use.

Proposal

Apply ./evals-data as the fallback in src/config/server-config.ts when EVALS_DATA_DIR is unset. Keep the existing .min(1) guard — it rejects an empty string (EVALS_DATA_DIR=) with a clear validation

Read the thread · 2026-06-25 · closed · 1 comment

bug(evals_create_draft): mcq grader correct-not-in-choices fires task_type_constraint instead of mcq_choice_mismatch

Server version

0.1.0

Runtime

Bun

Transport

HTTP / stdio

Description

evals_create_draft fires reason: "task_type_constraint" when the mcq grader's correct value is not in choices[], but the error contract documents this case as reason: "mcq_choice_mismatch". evals_run_check correctly returns mcq_choice_mismatch for the same condition — the two tools are inconsistent.

Steps to reproduce

  1. Call evals_create_draft with task_type: "mcq", `choices: ["P

Read the thread · 2026-06-25 · closed · 1 comment

bug(evals_revise_draft): unknown set path reports false success

Server version

0.1.2

mcp-ts-core version

0.12.3

Runtime

Bun

Runtime version

Bun 1.4.0

Transport

HTTP (Streamable HTTP)

Description

evals_revise_draft reports an unknown set path as successfully applied even though schema validation removes it from the stored record. The success response makes the mutation appear effective when it was a no-op.

Steps to reproduce

  1. Create a valid draft with evals_create_draft.
  2. Call evals_revise_draft with `{ "dr

Read the thread · 2026-08-22 · open · 0 comments

fix(config): pin MCP_SESSION_MODE to stateful — the image ships stateless and breaks the confirmation gate for v1 HTTP clients

Dockerfile sets ENV MCP_SESSION_MODE="stateless", but .env.example carries a commented # MCP_SESSION_MODE=stateful, whose comment documents the value set as stateful | stateless (default: stateful). The container and the same code run via bunx, npm start, or from source therefore resolve to different session modes.

For this server the container is the one that is wrong.

Related: cyanheads/mcp-ts-core#376

Proposal

Settle every surface on stateful, and change the `Dockerfile

Read the thread · 2026-08-22 · open · 0 comments

bug(evals_submit_draft): llm_rubric/free_response is never judged — ctx.sample does not exist, descriptions over-claim LLM grading

Server version

0.1.2

mcp-ts-core version

0.10.9

Runtime

Bun

Runtime version

Bun 1.3.x

Transport

stdio

Description

free_response / llm_rubric is documented across the surface as "judged via ctx.sample when the client supports sampling" — but no such grading ever happens, and ctx.sample does not exist.

Read the thread · 2026-06-30 · open · 0 comments

bug(evals_create_draft): persists an orphan draft when the grader is unexecutable

Server version

0.1.2

mcp-ts-core version

0.10.9

Runtime

Bun

Runtime version

Bun 1.3.x

Transport

stdio

Description

evals_create_draft persists the draft to disk (store.writeDraft) before running the self-consistency check. checkSelfConsistency calls gradeCandidate, which throws grader_unexecutable for a malformed grader (non-evaluable math.js target, invalid regex, json_match with neither expected nor schema). The throw becomes the tool's err

Read the thread · 2026-06-30 · open · 0 comments

Most recent

bug(grader): numeric grader accepts complex/unit candidates by truncating at the first space

Server version

0.1.2

mcp-ts-core version

0.10.9

Runtime

Bun

Runtime version

Bun 1.3.x (logic bug — runtime-agnostic)

Transport

stdio

Description

The numeric grader coerces a math.js result to a plain number with Number(result.toString().replace(/\s.*$/, '')) (candidateToNumber and resolveNumber in src/services/grader/grader-service.ts). For any non-real result — a math.js Complex or Unit — toString() yields e.g. "2 + 3i"

Read the thread · 2026-06-30 · open · 0 comments

The remaining reports are on the project's issue tracker.