pod
Other formats agents might prefer:
markdownjsonllms.txt

Agent? You probably want markdown or json, or Pod over MCP.

Reported issues for CompletionKit

Pod holds 18 of 23 problems reported by people outside the maintainer team. Issues filed by the project's own owners, members and collaborators are excluded entirely — a maintainer's release checklist is not a warning to a prospective user.

Back to CompletionKit.

Most discussed

MCP: expose generation max_tokens (and temperature) per prompt/run — truncation can't be fixed otherwise

Summary

There is no way to set generation max_tokens (or other generation params) for a prompt/run over the MCP, and it directly blocked a real optimization.

Where it bit

Evaluating a recommend-products alternatives prompt whose output is long (one section has ~10 primaries × 2 alternatives × 2 reasons). The judge's alts-format-validity metric was ~2.86, and the low-score critiques were dominated by truncation: *"the JSON is malformed/truncated, several primaries missing."

Read the thread · 2026-07-29 · closed · outside contributor · 1 comment

Plan run-limit is not surfaced via MCP or API — runs stall silently with error_message: null (indistinguishable from 502 flakiness)

Summary

When an org hits its plan run limit, nothing in the MCP or API surfaces that fact. Runs silently fail to progress, and every introspection point reports a clean/empty state. The hard quota is indistinguishable from transient infra flakiness (502s), which caused a long, wrong debugging path (attributed stalls to an outage and then to a config bug, when a quota ceiling was also in play).

What was observed at the limit

  • runs_create → returns a normal pending run object, **

Read the thread · 2026-07-28 · closed · outside contributor · 1 comment

MCP: runs_generate returns a timeout/-32603 error even though the generation job started (false failure)

Summary

The MCP runs_generate tool frequently returns an error to the caller — -32603 Internal error or "The operation timed out." — even though the generation job actually started server-side. The run proceeds to running and completes normally; the only way to know is to poll runs_get afterward.

So a successful, mutating operation surfaces as a failure. In an agent/automation loop this is a real trap: the natural response to an error is to retry or treat it as failed, whe

Read the thread · 2026-07-28 · closed · outside contributor · 3 comments

First-class run-lifecycle callback so hosts don't monkey-patch to observe run creation

Host apps have no supported way to observe run creation. completion-kit-cloud currently patches three seams to meter/gate runs: class_eval on RunsController and Api::V1::RunsController (before/after action hooks) and a prepend on McpDispatcher (McpRunGate). Every new create path added to the engine silently bypasses all three until the host notices.

Proposal: a config-level callback, e.g.

CompletionKit.configure do |c|
  c.on_run_created = ->(run) { Billing::UsageMeter.record

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/101) · 2026-07-06 · closed · outside contributor · 1 comment

### Import a promptfooconfig.yaml into a prompt + dataset + metrics

## Why
OpenAI's acquisition of Promptfoo (March 2026) means a stream of users are evaluating a switch. The single biggest thing that turns 'I should look around' into 'I'm already set up' is a one-step import of their existing `promptfooconfig.yaml`. This rides a time-limited intent window and is a product feature, so it belongs in the engine (every deployment gets it: Cloud, standalone, engine).

## What
An importer (REST endpoint + a small UI entry point, and ideally an MCP tool) that takes a 

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/76) · 2026-06-15 · closed · outside contributor · 2 comments

### runs_generate raises org_slug UrlGenerationError on remounted hosts after the run has already started (warm_routes! bare helper call)

## Summary

Calling the `runs_generate` MCP tool (and anything else that goes through `Run#start!` outside a web request) raises `ActionController::UrlGenerationError: missing required keys: [:org_slug]` on a host that remounts the engine under a param scope, even though the run has already been enqueued. The caller gets an error for a run that is in fact running.

## Trace

`Run#start!` enqueues the row jobs, then calls `broadcast_ui` -> `broadcast_progress` -> `render_engine_partial` -> `Compl

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/68) · 2026-06-12 · closed · outside contributor · 1 comment

### ModelDiscoveryJob: provider index card stuck on "Looking up models…" because broadcast partial render fails in worker context

## Summary

`ModelDiscoveryJob`'s live UI updates render controller partials from a background-job context (no request) via `CompletionKit::ApplicationController.render`. That render is fragile in a worker: when it raises, the provider **index** card never updates and stays stuck on "Looking up models…", even though discovery actually completes and the DB is correct. The provider **detail** page is fine because it server-renders fresh from the persisted `discovery_status`.

## Symptom

Add a pro

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/59) · 2026-06-08 · closed · outside contributor · 1 comment

### Judge calibration · Phase 4 — Compare and publish judge versions

> Part of the **Judge calibration** epic — #31. Ships to: **early-access flag**. Depends on Phases 1–3.

Judges get versioned the way prompts already are. Editing a metric forks a draft judge. You can test that draft against everything humans have already calibrated, see the two versions side by side, and get an advisory recommendation next to Publish.

### Done when
- [ ] Editing a metric forks a draft judge version; publishing works like publishing a prompt
- [ ] "Test draft against calibratio

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/35) · 2026-05-16 · closed · outside contributor · 1 comment

## Most recent

### MCP: no way to regrade/rerun existing run responses (rerun/retry_failures/regrade not exposed as tools)

## Summary

There is no way, over the MCP surface, to **re-judge the existing responses of a completed run** — e.g. after attaching/changing metrics post-generation, or after editing a metric. `runs_generate` refuses on a completed run with:

> Cannot start a run in state "completed". Use rerun to create a fresh copy, or retry_failures / regrade to work with the existing responses.

…but `rerun`, `retry_failures`, and `regrade` are **not exposed as MCP tools**. The full MCP run surface is only: 

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/148) · 2026-07-28 · closed · outside contributor · 1 comment

### MCP: runs_create/runs_update silently ignore metric_group_id → run generates but judges nothing (false success)

## Summary

`runs_create` and `runs_update` (MCP) accept only `metric_ids`. If you instead pass `metric_group_id` — the intuitive param, since metric groups are first-class MCP objects (`metric_groups_create` / `metric_groups_get` / `metric_groups_list`) — it is **silently dropped**: no validation error, the run is created with `metric_ids: []`, generation runs to completion, and the run ends `status: "completed"` with `judged: 0/0` and `avg_score: null`. It looks like a successful run but nothi

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/147) · 2026-07-28 · closed · outside contributor · 1 comment

### Allow unauthenticated initialize/tools/list on the MCP endpoint; gate tools/call behind auth

The MCP endpoint authenticates the **whole request** in the controller, so an unauthenticated `initialize` or `tools/list` gets 401. That blinds every MCP directory and every curious human: registries (Glama, the Official MCP Registry clients) that introspect a remote server without a token cannot see the toolbox, so a hosted CompletionKit server shows as "requires authentication" / unhealthy rather than surfacing its tools.

## The ask
Make auth **method-aware** instead of endpoint-wide:

- **U

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/131) · 2026-07-21 · closed · outside contributor · 1 comment

### MCP streamable-HTTP endpoint returns 404 on GET (should be 405 or an SSE stream), failing directory health checks

The MCP endpoint only routes `POST` and `DELETE`, so a `GET` gets no route and Rails returns **404**. Per the MCP streamable-HTTP transport, a `GET` to the MCP endpoint should either open an SSE stream (`200 text/event-stream`, for server→client messages) or return **405 Method Not Allowed** when the server does not offer server-initiated streams. A bare 404 is non-compliant and reads to tooling as "endpoint does not exist."

## Where
`config/routes.rb`:
```ruby
post   "mcp", to: "mcp#handle"
de

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/129) · 2026-07-21 · closed · outside contributor · 1 comment

### Model-discovery failure messaging leaks 'ollama' and surfaces opaque provider 404s

## Summary

When model discovery fails for a custom OpenAI-compatible endpoint, the user-facing message leaks the internal provider slug `ollama` and surfaces a raw provider 404 with no next step.

## What the user sees

`_discovery_status.html.erb` renders `discovery_error` verbatim:

Model discovery failed: ollama model list request failed (404): Resource not found


For someone configuring an Azure AI Foundry endpoint through the generic custom-endpoint option (see #104), this is confu

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/105) · 2026-07-07 · closed · outside contributor · 0 comments

### Blank cell in the graded column crashes score-only runs (stuck pending, 500)

A score-only run over a dataset where the graded column (e.g. `actual_output`) has a blank cell raises `RecordInvalid: Response text can't be blank` inside `Run#start!`'s transaction. The rollback leaves the run stuck in `pending` with 0 responses and no `failure_summary`, and the caller gets a 500/502.

Blank cells are exactly what answer-key datasets contain when the system under test returned nothing; that row should fail its check, not crash the run.

Related paper cut: the `runs_generate` M

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/99) · 2026-07-06 · closed · outside contributor · 0 comments

### Per-row expected/answer-key column is a hidden magic convention: surface it and let users choose the column

## Problem

The per-row answer key that checks (and the judge) grade against is a **hidden magic-column convention** with no surfacing and no configuration.

When a run builds responses from a dataset, it maps only a CSV column named literally `expected_output` into each response's `expected_output` (`app/models/completion_kit/run.rb:216`: `expected_output: row["expected_output"]`); every other column is folded into `input_data` (`input = row.to_json`). Judge-only runs have a parallel magic colu

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/98) · 2026-07-06 · closed · outside contributor · 1 comment

### Check metric: authoring form + strong params + REST/MCP + starters

Let people author and consume check metrics: a type-aware form, strong params in every entry point, check-config versioning, starter checks, and the REST/MCP surface. Depends on the model foundation.

## Form
`metrics/_form.html.erb` (single partial): a type chooser radio on **new** (default "LLM judge (1-5)") that toggles the rubric editor vs a check builder (`check_kind` select, `target` select, kind-dependent inputs). **Lock the type on edit** — absolutely once any run references the metric (

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/89) · 2026-06-29 · closed · outside contributor · 1 comment

### Prompt suggestions present (and one-click apply) a rewrite with no measured before/after delta, unlike validated metric improvements

## Summary

`prompts_suggest_improvement` / "Suggest improvements" can return a plausible-but-regressive prompt rewrite, and nothing in the flow measures whether the rewrite is actually better. The suggestion is presented as a reasoning blurb plus a text word-diff, with an unconditional one-click **Apply** that publishes a new prompt version immediately. The only score on the page is the *original* run's average, never a score for the suggested template.

This is the gap NOTE 2 calls out: surfac

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/67) · 2026-06-10 · closed · outside contributor · 0 comments

### MCP server: run stateless so clients never need to reconnect (session-id loss)

## Summary

The CompletionKit MCP server requires an `initialize` handshake that mints an `Mcp-Session-Id`, and every later request must present that id. When the id is no longer accepted, clients get `-32000 "Session not initialized. Send initialize first."` and must reconnect. This surfaces to users (and a client reported it) as the MCP connection "keeps needing to be reconnected."

## Root cause (grounded in code)

- `app/controllers/completion_kit/mcp_controller.rb:19-21` enforces a live ses

[Read the thread](https://github.com/homemade-software-inc/completion-kit/issues/61) · 2026-06-08 · closed · outside contributor · 0 comments

The remaining reports are on [the project's issue tracker](https://github.com/homemade-software-inc/completion-kit/issues).