Pod

Yes, this is on purpose. Pod is built for agents, so the default page is plain HTML.
Human? View the normal website.
Agent? You probably prefer /mcp/mcp-server-iris-eval/issues.md or /mcp/mcp-server-iris-eval/issues.json, or Pod over MCP.

Reported issues for mcp-server

Pod holds 18 of 19 problems reported by people outside the maintainer team. Issues filed by the project's own owners, members and collaborators are excluded entirely — a maintainer's release checklist is not a warning to a prospective user.

Back to mcp-server.

Most discussed

Add CSV/JSON export to dashboard

The web dashboard shows traces and evaluations but doesn't support exporting data.

Goal: Add export buttons to the trace list and evaluation list pages.

Acceptance criteria:

Resources:

Read the thread · 2026-03-16 · open · 2 comments

Add copy-to-clipboard for trace IDs in dashboard

When viewing traces in the dashboard, clicking a trace ID should copy it to the clipboard for easy sharing.

Acceptance criteria:

Good starting point: dashboard/src/pages/Traces.tsx

Read the thread · 2026-03-20 · closed · 1 comment

Add OpenTelemetry trace export support

Iris currently stores traces in SQLite. Many teams already have observability stacks (Datadog, Grafana, New Relic) that ingest OpenTelemetry spans.

Goal: Add an optional OTel exporter that converts Iris traces to OTel spans and sends them to any OTLP-compatible endpoint.

Acceptance criteria:

Read the thread · 2026-03-16 · closed · 1 comment

Dashboard UI polish from first-run walkthroughs

  1. A moment whose only noteworthy event is failed safety rules shows a yellow PARTIAL verdict chip next to the red SAFETY VIOLATION banner. When moment.significance.kind === 'safety-violation', render the chip in danger color / "SAFETY FAIL"; deeper fix in computeVerdict (src/eval/decision-moment.ts).
  2. Welcome banner hardcodes ~/.iris/preferences.json even when IRIS_HOME points elsewhere; the first-run tour reappears across demo/real dashboards (separate preference files) — persist t

Read the thread · 2026-08-12 · open · 0 comments

Input validation: nested objects and dashboard routes still strip unknown keys

Follow-ups to the strict top-level tool schemas (fix/uat-args):

  1. Nested objects still strip silently. custom_rules: [{name: 'a', wieght: 5}] parses fine with wieght discarded — a misspelled rule weight silently changes scoring. Same family as the fixed defect, one level down. Decide: recursive strictness (with care for the free-form record fields — metadata, span attributes, rule config — whose arbitrary keys are legitimate) or document the boundary.
  2. **All 14 dashboard-route

Read the thread · 2026-08-12 · open · 0 comments

passed semantics: persistence and cross-tool consistency gaps after the critical-rule veto

Follow-ups to the hard-fail semantics change (fix/uat-crit), found during its test-completion:

  1. critical_failures is never persisted. eval_results has no column for it and insertEvalResult (src/storage/sqlite-adapter.ts:263) doesn't write it. The MCP response carries it (computed in-process), but anything reading the eval back — dashboard drill-through, trace summaries, MCP resources — sees passed: false with no stated cause. Worse: the veto suggestion line is only appended when

Read the thread · 2026-08-12 · open · 0 comments

PII patterns are label-anchored more than the pattern list implies

Three of the 19 patterns only fire with specific labels: ISO-format DOB (Date of birth: 1987-03-15) is MISSED while DOB: 03/15/1987 is caught; unlabeled passport numbers in prose are missed; an unlabeled 12-word BIP39 seed phrase is missed while My seed phrase is: … is caught. Fix passport to its documented modern format, add an ISO alternative to the DOB date part, and qualify the README pattern list to note label-anchoring.

_Found during install-only acceptance testing of the v0.5.0

Read the thread · 2026-08-12 · open · 0 comments

Dashboard/API correctness batch

  1. /api/v1/health trace_count contradicts /api/v1/traces (0 vs 253 in demo mode): health uses a windowed summary. Make it an all-time COUNT(*) or rename to traces_last_hour (src/dashboard/routes/health.ts:21).
  2. Rule preview endpoint silently ignores sample text: deploy_rule's description says to use POST /api/v1/rules/custom/preview for "dry-run validation against sample output", but the endpooint only replays stored traces; sampleOutput/output/sample/text keys

Read the thread · 2026-08-12 · open · 0 comments

Most recent

Data retention and permissions for stored eval/trace text

PII the tool just flagged is stored verbatim in plaintext and served to any local process; deleting iris.db leaves the text in iris.db-wal.

  1. Document in the README that iris.db stores raw output text verbatim, including anything no_pii flags.
  2. eval_results is missing from the retention sweep — extend deleteTracesOlderThan (src/storage/sqlite-adapter.ts:622) or add deleteEvalResultsOlderThan called alongside src/index.ts:250; add a --purge flag.
  3. Create IRIS_HOME with

Read the thread · 2026-08-12 · open · 0 comments

--self-test passes while the configured IRIS_HOME is unusable

--self-test prints "✓ PASS - this install works" with exit 0 while pointed at an IRIS_HOME the server cannot write; the real server then dies on startup with a raw EPERM stack trace. The self-test isolates into a temp home (good) but never probes the CONFIGURED home. Fix: in src/self-test.ts, before the temp-home isolation, mkdirSync(irisHome(), {recursive:true}) + a write/unlink probe against the configured home, failing the run with a named path on error. Also wrap the two mkdirSync

Read the thread · 2026-08-12 · open · 0 comments

evaluate_output parameter semantics: relevance demands input, docs say expected; placeholder suppression is invisible

  1. The tool description states expected is REQUIRED for eval_type="relevance" and is "used as the comparison target" — but passing output + expected yields insufficient_data: true with BOTH relevance rules skipped demanding context.input. The parameter the description names does not enable the rules; the undeclared one does. Either make the rules in src/eval/rules/relevance.ts fall back to context.expected when input is absent (smaller behavioral surprise), or correct both desc

Read the thread · 2026-08-12 · open · 0 comments

Dashboard: install-path disclosure on two endpoints, external font CDN, and a firefox e2e flake

Three smaller items from a full-codebase review.

1. Absolute install path returned by two endpoints

src/audit-log-reader.ts:113 (path: filePath) and src/dashboard/routes/preferences.ts:44,51 (path: store.path) return e.g. "path":"C:\Users\<user>\.iris\preferences.json", disclosing the install path and OS username. Same CWE-209 class PR #286 closed for 404s, still present by design here.

Exposure is much reduced now that #323 binds the dashboard to loopback, but there's no reason

Read the thread · 2026-08-10 · closed · 0 comments

MCP tool descriptions promise behaviour the code doesn't implement

Four cases where an agent reading the tool description gets a wrong answer. These matter more than ordinary doc drift: the description IS the interface an LLM programs against.

1. get_traces limit is unclamped on the MCP path

src/tools/get-traces.ts:13 — the Zod schema has no .max(), and queryTraces (sqlite-adapter.ts:177) doesn't clamp. The description promises "max 1000 — values >1000 return 400". Verified: limit: 1000000 is accepted; limit: -1 becomes SQLite's no-limit an

Read the thread · 2026-08-10 · closed · 0 comments

CI: lighthouse job flaking (~3 false-fails on 2026-06-09)

The lighthouse workflow failed then passed (on rerun or on an adjacent identical diff) three times on 2026-06-09:

  1. PR #190 (workflow-pin-only diff — cannot affect dashboard perf): failed in 1m42s → passed on rerun (failed run)
  2. PR #187 (react-router bump): failed → passed on rerun (rerun)
  3. PR #165 (react-dom patch bump): failed; merged o

Read the thread · 2026-06-09 · closed · 0 comments

Upgrade root TypeScript to v6 when @typescript-eslint supports it

Context

PR #57 was closed because @typescript-eslint/eslint-plugin@8.57.1 declares peer dep typescript@">=4.8.4 <6.0.0" — explicitly excludes TypeScript 6. npm ci fails with ERESOLVE in CI.

Dashboard (#55) and website (#56) were upgraded to TS 6 successfully — they use separate lint toolchains that don't depend on @typescript-eslint.

Unblock condition

Current state

Read the thread · 2026-03-23 · open · 0 comments

Add dark/light theme toggle to dashboard

The web dashboard currently uses a fixed dark theme. Add a theme toggle button that switches between dark and light modes.

Acceptance criteria:

Good starting point: dashboard/src/App.tsx for the toggle, CSS variables in dashboard/index.html

Read the thread · 2026-03-20 · closed · 0 comments

Add full-text search to trace queries

Currently traces can only be filtered by agent_name, framework, and date range. Teams need to search trace content.

Goal: Add text search capability to get_traces and the dashboard.

Acceptance criteria:

Resources:

Read the thread · 2026-03-16 · open · 0 comments

Create Python SDK for HTTP transport

Iris supports HTTP transport but there's no Python SDK. Python is the primary language for AI/ML development.

Goal: Create a lightweight Python package (iris-eval) that wraps the HTTP API.

Acceptance criteria:

Resources:

Read the thread · 2026-03-16 · open · 0 comments

The remaining reports are on the project's issue tracker.