# Reported issues for CompletionKit

Pod holds 18 of 23 problems reported by people outside the maintainer team. Issues filed by the project's own owners, members and collaborators are excluded entirely — a maintainer's release checklist is not a warning to a prospective user.

Back to [CompletionKit](/mcp/completionkit).

## Most discussed

### MCP: expose generation max_tokens (and temperature) per prompt/run — truncation can't be fixed otherwise

## Summary

There is no way to set **generation `max_tokens`** (or other generation params) for a prompt/run over the MCP, and it directly blocked a real optimization.

## Where it bit

Evaluating a `recommend-products` alternatives prompt whose output is long (one section has ~10 primaries × 2 alternatives × 2 reasons). The judge's `alts-format-validity` metric was ~2.86, and the low-score critiques were dominated by **truncation**: *"the JSON is malformed/truncated, several primaries missing."

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/159) · 2026-07-29 · closed · outside contributor · 1 comment

### Plan run-limit is not surfaced via MCP or API — runs stall silently with error_message: null (indistinguishable from 502 flakiness)

## Summary

When an org hits its **plan run limit**, nothing in the MCP or API surfaces that fact. Runs silently fail to progress, and every introspection point reports a clean/empty state. The hard quota is indistinguishable from transient infra flakiness (502s), which caused a long, wrong debugging path (attributed stalls to an outage and then to a config bug, when a quota ceiling was also in play).

## What was observed at the limit

- `runs_create` → returns a normal `pending` run object, **

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/152) · 2026-07-28 · closed · outside contributor · 1 comment

### MCP: runs_generate returns a timeout/-32603 error even though the generation job started (false failure)

## Summary

The MCP `runs_generate` tool frequently returns an **error to the caller** — `-32603 Internal error` or `"The operation timed out."` — **even though the generation job actually started** server-side. The run proceeds to `running` and completes normally; the only way to know is to poll `runs_get` afterward.

So a successful, mutating operation surfaces as a failure. In an agent/automation loop this is a real trap: the natural response to an error is to retry or treat it as failed, whe

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/150) · 2026-07-28 · closed · outside contributor · 3 comments

### First-class run-lifecycle callback so hosts don't monkey-patch to observe run creation

Host apps have no supported way to observe run creation. completion-kit-cloud currently patches three seams to meter/gate runs: `class_eval` on `RunsController` and `Api::V1::RunsController` (before/after action hooks) and a prepend on `McpDispatcher` (`McpRunGate`). Every new create path added to the engine silently bypasses all three until the host notices.

Proposal: a config-level callback, e.g.

```ruby
CompletionKit.configure do |c|
  c.on_run_created = ->(run) { Billing::UsageMeter.record

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/101) · 2026-07-06 · closed · outside contributor · 1 comment

### Import a promptfooconfig.yaml into a prompt + dataset + metrics

## Why
OpenAI's acquisition of Promptfoo (March 2026) means a stream of users are evaluating a switch. The single biggest thing that turns 'I should look around' into 'I'm already set up' is a one-step import of their existing `promptfooconfig.yaml`. This rides a time-limited intent window and is a product feature, so it belongs in the engine (every deployment gets it: Cloud, standalone, engine).

## What
An importer (REST endpoint + a small UI entry point, and ideally an MCP tool) that takes a 

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/76) · 2026-06-15 · closed · outside contributor · 2 comments

### runs_generate raises org_slug UrlGenerationError on remounted hosts after the run has already started (warm_routes! bare helper call)

## Summary

Calling the `runs_generate` MCP tool (and anything else that goes through `Run#start!` outside a web request) raises `ActionController::UrlGenerationError: missing required keys: [:org_slug]` on a host that remounts the engine under a param scope, even though the run has already been enqueued. The caller gets an error for a run that is in fact running.

## Trace

`Run#start!` enqueues the row jobs, then calls `broadcast_ui` -> `broadcast_progress` -> `render_engine_partial` -> `Compl

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/68) · 2026-06-12 · closed · outside contributor · 1 comment

### ModelDiscoveryJob: provider index card stuck on "Looking up models…" because broadcast partial render fails in worker context

## Summary

`ModelDiscoveryJob`'s live UI updates render controller partials from a background-job context (no request) via `CompletionKit::ApplicationController.render`. That render is fragile in a worker: when it raises, the provider **index** card never updates and stays stuck on "Looking up models…", even though discovery actually completes and the DB is correct. The provider **detail** page is fine because it server-renders fresh from the persisted `discovery_status`.

## Symptom

Add a pro

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/59) · 2026-06-08 · closed · outside contributor · 1 comment

### Judge calibration · Phase 4 — Compare and publish judge versions

> Part of the **Judge calibration** epic — #31. Ships to: **early-access flag**. Depends on Phases 1–3.

Judges get versioned the way prompts already are. Editing a metric forks a draft judge. You can test that draft against everything humans have already calibrated, see the two versions side by side, and get an advisory recommendation next to Publish.

### Done when
- [ ] Editing a metric forks a draft judge version; publishing works like publishing a prompt
- [ ] "Test draft against calibratio

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/35) · 2026-05-16 · closed · outside contributor · 1 comment

## Most recent

### MCP: no way to regrade/rerun existing run responses (rerun/retry_failures/regrade not exposed as tools)

## Summary

There is no way, over the MCP surface, to **re-judge the existing responses of a completed run** — e.g. after attaching/changing metrics post-generation, or after editing a metric. `runs_generate` refuses on a completed run with:

> Cannot start a run in state "completed". Use rerun to create a fresh copy, or retry_failures / regrade to work with the existing responses.

…but `rerun`, `retry_failures`, and `regrade` are **not exposed as MCP tools**. The full MCP run surface is only: 

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/148) · 2026-07-28 · closed · outside contributor · 1 comment

### MCP: runs_create/runs_update silently ignore metric_group_id → run generates but judges nothing (false success)

## Summary

`runs_create` and `runs_update` (MCP) accept only `metric_ids`. If you instead pass `metric_group_id` — the intuitive param, since metric groups are first-class MCP objects (`metric_groups_create` / `metric_groups_get` / `metric_groups_list`) — it is **silently dropped**: no validation error, the run is created with `metric_ids: []`, generation runs to completion, and the run ends `status: "completed"` with `judged: 0/0` and `avg_score: null`. It looks like a successful run but nothi

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/147) · 2026-07-28 · closed · outside contributor · 1 comment

### Allow unauthenticated initialize/tools/list on the MCP endpoint; gate tools/call behind auth

The MCP endpoint authenticates the **whole request** in the controller, so an unauthenticated `initialize` or `tools/list` gets 401. That blinds every MCP directory and every curious human: registries (Glama, the Official MCP Registry clients) that introspect a remote server without a token cannot see the toolbox, so a hosted CompletionKit server shows as "requires authentication" / unhealthy rather than surfacing its tools.

## The ask
Make auth **method-aware** instead of endpoint-wide:

- **U

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/131) · 2026-07-21 · closed · outside contributor · 1 comment

### MCP streamable-HTTP endpoint returns 404 on GET (should be 405 or an SSE stream), failing directory health checks

The MCP endpoint only routes `POST` and `DELETE`, so a `GET` gets no route and Rails returns **404**. Per the MCP streamable-HTTP transport, a `GET` to the MCP endpoint should either open an SSE stream (`200 text/event-stream`, for server→client messages) or return **405 Method Not Allowed** when the server does not offer server-initiated streams. A bare 404 is non-compliant and reads to tooling as "endpoint does not exist."

## Where
`config/routes.rb`:
```ruby
post   "mcp", to: "mcp#handle"
de

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/129) · 2026-07-21 · closed · outside contributor · 1 comment

### Model-discovery failure messaging leaks 'ollama' and surfaces opaque provider 404s

## Summary

When model discovery fails for a custom OpenAI-compatible endpoint, the user-facing message leaks the internal provider slug `ollama` and surfaces a raw provider 404 with no next step.

## What the user sees

`_discovery_status.html.erb` renders `discovery_error` verbatim:

```
Model discovery failed: ollama model list request failed (404): Resource not found
```

For someone configuring an Azure AI Foundry endpoint through the generic custom-endpoint option (see #104), this is confu

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/105) · 2026-07-07 · closed · outside contributor · 0 comments

### Blank cell in the graded column crashes score-only runs (stuck pending, 500)

A score-only run over a dataset where the graded column (e.g. `actual_output`) has a blank cell raises `RecordInvalid: Response text can't be blank` inside `Run#start!`'s transaction. The rollback leaves the run stuck in `pending` with 0 responses and no `failure_summary`, and the caller gets a 500/502.

Blank cells are exactly what answer-key datasets contain when the system under test returned nothing; that row should fail its check, not crash the run.

Related paper cut: the `runs_generate` M

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/99) · 2026-07-06 · closed · outside contributor · 0 comments

### Per-row expected/answer-key column is a hidden magic convention: surface it and let users choose the column

## Problem

The per-row answer key that checks (and the judge) grade against is a **hidden magic-column convention** with no surfacing and no configuration.

When a run builds responses from a dataset, it maps only a CSV column named literally `expected_output` into each response's `expected_output` (`app/models/completion_kit/run.rb:216`: `expected_output: row["expected_output"]`); every other column is folded into `input_data` (`input = row.to_json`). Judge-only runs have a parallel magic colu

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/98) · 2026-07-06 · closed · outside contributor · 1 comment

### Check metric: authoring form + strong params + REST/MCP + starters

Let people author and consume check metrics: a type-aware form, strong params in every entry point, check-config versioning, starter checks, and the REST/MCP surface. Depends on the model foundation.

## Form
`metrics/_form.html.erb` (single partial): a type chooser radio on **new** (default "LLM judge (1-5)") that toggles the rubric editor vs a check builder (`check_kind` select, `target` select, kind-dependent inputs). **Lock the type on edit** — absolutely once any run references the metric (

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/89) · 2026-06-29 · closed · outside contributor · 1 comment

### Prompt suggestions present (and one-click apply) a rewrite with no measured before/after delta, unlike validated metric improvements

## Summary

`prompts_suggest_improvement` / "Suggest improvements" can return a plausible-but-regressive prompt rewrite, and nothing in the flow measures whether the rewrite is actually better. The suggestion is presented as a reasoning blurb plus a text word-diff, with an unconditional one-click **Apply** that publishes a new prompt version immediately. The only score on the page is the *original* run's average, never a score for the suggested template.

This is the gap NOTE 2 calls out: surfac

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/67) · 2026-06-10 · closed · outside contributor · 0 comments

### MCP server: run stateless so clients never need to reconnect (session-id loss)

## Summary

The CompletionKit MCP server requires an `initialize` handshake that mints an `Mcp-Session-Id`, and every later request must present that id. When the id is no longer accepted, clients get `-32000 "Session not initialized. Send initialize first."` and must reconnect. This surfaces to users (and a client reported it) as the MCP connection "keeps needing to be reconnected."

## Root cause (grounded in code)

- `app/controllers/completion_kit/mcp_controller.rb:19-21` enforces a live ses

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/61) · 2026-06-08 · closed · outside contributor · 0 comments

The remaining reports are on [the project's issue tracker](https://github.com/homemade-software-inc/completion-kit/issues).