Reported issues for io.github.Judgment-Pack/judgment-pack

Pod holds 11 of 11 problems reported by people outside the maintainer team. Issues filed by the project's own owners, members and collaborators are excluded entirely — a maintainer's release checklist is not a warning to a prospective user.

Back to io.github.Judgment-Pack/judgment-pack.

Most discussed

Serve a declared graph document over MCP, the way get_pack serves a pack

A client rendering a project's graph has get_pack for every node's pack and nothing for the graph document itself: no tool returns the declared graph (ADR-0015/0017), so the one artifact that states the composition is the one a wire-only client cannot read. Symmetry suggests get_graph { graph_id } returning the document as JSON text with digest and byte size, read through the same rooted reader, alongside a cheap inventory (either a list_graphs or graph entries on an existing listing) so a

Read the thread · 2026-08-24 · closed · outside contributor · 0 comments

Let a caller mark an evaluation as a rehearsal, so a what-if session does not append decisions to the audit trail

The gap

A client exploring counterfactuals over MCP — edit the facts, re-evaluate, compare dispositions — calls experimental_evaluate repeatedly. In a project that declares an audit directory (ADR-0018), every completed call appends an audit record: a what-if session of N runs leaves N records saying the project decided N times, when it decided zero times.

The matrix runner already embodies the needed principle: ADR-0021 keeps the rehearsal outside the record, and `experimental_test_packs

Read the thread · 2026-08-24 · closed · outside contributor · 0 comments

Cover the MCP transport's oversized-line bound

Goal

Test what the stdio MCP server does with a JSON-RPC line larger than maxMessageBytes.

Why

internal/mcp/server.go bounds one line at 16 MiB and sizes the scanner buffer accordingly:

const maxMessageBytes = 16 * 1024 * 1024
scanner.Buffer(make([]byte, 0, 64*1024), maxMessageBytes)

maxMessageBytes appears nowhere in internal/mcp/server_test.go. A client that sends an oversized line gets whatever bufio.Scanner does at its limit, and nothing pins whether that i

Read the thread · 2026-08-15 · open · outside contributor · 0 comments

packs suggest: deterministically enumerate candidate test inputs a pack's own conditions imply — never their expectations

PLAN — ADR-0024: a deterministic test-row input generator (jpack packs suggest)

0. Ground-truth corrections to the brief (read these first)

• ADR-0023 IS merged — PR #100, runtime main. My local checkout was stale (git fetch --dry-run does not move refs), so I read the merged bytes via the GitHub API. internal/project/coverage.go on main is 864 lines and does carry comparisonSite (:384), boundaryGroup (:407), comparisonSites (:694), collectComparisons (:723), `bo

Read the thread · 2026-08-07 · closed · outside contributor · 0 comments

Run declared graph matrices over MCP with experimental_test_graphs

Goal

Add the deferred experimental_test_graphs MCP tool so agents can run declared graph matrices without falling back to a terminal.

Why

ADR-0021 added experimental_test_packs and explicitly recorded graph-matrix support as a reopening condition. The graph CLI and shared graph.TestProject / result.GraphSuite model now provide a path to close that MCP gap without inventing evaluator or graph semantics.

Scope

• Record and obtain the required cross-vendor review before implem

Read the thread · 2026-08-07 · closed · outside contributor · 0 comments

Add MCP transport edge-case tests

Goal

Add table-driven tests for JSON-RPC and line-transport edge cases handled by the stdio MCP server.

Why

The server is expected to keep processing independent input lines and to distinguish requests from notifications. These transport guarantees are easy to regress and can be tested without making any evaluator or specification decision.

Scope

• Change internal/mcp/server_test.go only.
• Use a table-driven test where practical.
• Cover:
  • malformed JSON returning JSON-RPC pa

Read the thread · 2026-08-06 · closed · outside contributor · 0 comments

Bound the work a matrix run does, and decode the pack once across its rows

Found in the cross-vendor review of #77 (finding 2): a carrier-valid near-10 MiB pack with a 10,000-row matrix makes every row re-decode and revalidate the same pack through RunCase — on the order of 100 GiB of parsing for one run — and the evaluator's work budget begins only after preflight, so nothing bounds it. On the CLI this is the operator's own machine and Ctrl-C; over MCP (experimental_test_packs, ADR-0021) the synchronous stdio server is monopolized until the call completes, with no

Read the thread · 2026-08-05 · closed · outside contributor · 0 comments

Run a project's matrix over MCP, so the authoring loop can close where the method is served

The runtime serves the matrix authoring method over MCP and runs matrices only on the CLI. The test_pack prompt walks the client's model through building the instance matrix row by row, and closes on the discipline:

Re-run the whole matrix after any change. Keep the matrix with the pack; it is the pack's regression suite.

(internal/mcp/prompts.go)

No MCP tool runs one. The nine tools validate documents, serve schemas and examples, list packs, and evaluate single cases; list_packs repo

Read the thread · 2026-08-05 · closed · outside contributor · 0 comments

Most recent

Report each node's trace in the graph matrix payload over MCP

GraphNodeEvaluation carries the same trace a standalone evaluation reports (and ADR-0027 pins its contract), but the graph matrix payload a client receives over experimental_test_graphs does not surface per-node traces, so a client showing why one node of a row went unknown has nothing to render. The record exists; only the wire omits it. Size is the design question — a budget or opt-in may be the honest shape, since ADR-0026 already carries a report budget on this surface.

Found building a

Read the thread · 2026-08-24 · closed · outside contributor · 0 comments

Hold every MCP tool's arguments to its exact member names

Goal

Apply the exact member-name check experimental_test_graphs already uses to the other MCP tools that decode an arguments object.

Why

Every tool advertises additionalProperties: false, which means the exact spelling. encoding/json matches struct field names case-insensitively, so DisallowUnknownFields alone is not enough: a differently-cased member binds to the real field and the call succeeds.

internal/mcp/tools.go has an exactMembers helper for this, added with `ex

Read the thread · 2026-08-15 · open · outside contributor · 0 comments

Test the read-only MCP metadata tools end to end

Goal

Add end-to-end stdio tests for the three read-only MCP metadata tools: test_conformance, get_schema, and describe_runtime.

Why

These handlers expose stable metadata that MCP clients depend on, but they currently have no direct coverage through the server's stdio test harness. Focused tests will protect their structured payloads and error handling without changing any runtime or specification behavior.

Scope

• Change internal/mcp/server_test.go only.
• Use the existing

Read the thread · 2026-08-06 · closed · outside contributor · 0 comments

The remaining reports are on the project's issue tracker.