Reported issues for AgentLens

Pod holds 16 of 16 problems reported by people outside the maintainer team. Issues filed by the project's own owners, members and collaborators are excluded entirely — a maintainer's release checklist is not a warning to a prospective user.

Back to AgentLens.

Most discussed

Roadmap: agent control plane (Tier 2–3)

This is the cross-repo roadmap for agentkitai, derived from the competitive analysis (memory · observability · guardrails). It tracks Tier 2–3 work across agentlens (observe), agentgate (enforce/identify), and lore (remember).

Strategic thesis

The three products map onto the emerging "agent control plane" category. The defensible, underserved wedge is governance / audit / compliance, unified by agent identity — exactly where agentkitai already holds two rare assets (Agent

Read the thread · 2026-06-22 · open · 9 comments

Eval & prompt stack — compliance evals on the tamper-evident audit trail

Roadmap: Tier 3 — Close the eval/prompt quality gap vs Langfuse/LangSmith/Phoenix/Braintrust by leading with a defensible, compliance-first slice: LLM-as-judge scoring anchored to the hash-chained audit trail for tamper-evident evidence. From the agentkitai competitive analysis. Borrowed from: Langfuse (evals, LLM judges, datasets), LangSmith (prompt management, versioning), Phoenix (evaluators), Braintrust (experiments + guardrails gates).

Why

AgentLens today captures a pristine audi

Read the thread · 2026-06-22 · closed · 4 comments

LiveEval: compliance/guardrail scorer over live traffic

Narrow follow-up from #267. The registry scorers (regex/contains/exact_match/llm_judge) are now wired for live eval. Compliance is scored via a separate path (lib/eval/compliance.ts — tool allow/deny lists over the trace's tool_call events), not a registry IScorer. No milestone.

• Adapt the compliance scoring into an IScorer (or a live-specific runner) that reads the sampled session's tool_call events and evaluates against a configured policy (builtin-evaluators' compliance configs).
• Regi

Read the thread · 2026-07-01 · closed · 0 comments

RFC 3161: offline CMS/cert-chain verification of timestamp tokens

Narrow follow-up from #99. Requesting + storing the signed TSA token shipped (PR #268: DER TimeStampReq, POST to TSA, store token + PKIStatus + genTime). This adds offline cryptographic verification of a stored token. No milestone.

• Parse the timeStampToken (CMS SignedData → TSTInfo) and verify: the messageImprint matches the subject hash, the TSA signature is valid, and the signing cert chains to a trusted TSA root.
• A trusted-cert store (bundled roots + configurable) and CMS/ASN.1 p

Read the thread · 2026-07-01 · closed · 0 comments

Evaluator sampling (online eval)

From #224 (d). No milestone. Run evaluators on a sampled fraction of live traffic. Needs a LiveEvalEngine first (sampler + async evaluator runner over the ingest stream, writing scores back as annotations). Prereq-gated → graduate as its own effort once the online-eval engine exists.

Read the thread · 2026-06-30 · closed · 0 comments

Integrations stretch: JS LlamaIndex + Haystack (follow-up from #152)

Follow-up from #152. The two STRETCH integration targets, explicitly out of #152's acceptance. No milestone (does not gate Phase 7).

#152 delivered the must-haves: Python LlamaIndex (#210) + JS LangChain/LangGraph (#209), plus the existing Python LangChain. Acceptance met.

Remaining (stretch)

• JS LlamaIndex instrumentation in packages/sdk (mirror the JS LangChain handler src/langchain.ts → Instrumentation.capture; LlamaIndex.TS uses a callback/instrumentation handler).

Read the thread · 2026-06-30 · closed · 0 comments

Epic: Full Postgres runtime parity — port SQLite-only features to the Postgres path

Problem

AgentLens presents a dual-dialect storage layer (SQLite default, Postgres for scale), but runtime parity is far from complete. Authoritative audit (2026-06-30):

• 53 tables exist in SQLite (db/migrate.sqlite.ts), but only 21 are ever created in Postgres (db/drizzle/*.sql).
• PostgresEventStore (db/postgres-store.ts) implements only the core observability layer: events, sessions, agents, alert rules/history, analytics, retention, stats. It has zero methods

Read the thread · 2026-06-30 · closed · 0 comments

Scheduled blob-storage exports (S3/GCS/Azure) + NDJSON/JSONL

Gap (tier-2, should-have): Only on-demand signed exports; no scheduled exports to blob storage, and no NDJSON/JSONL.

Evidence (audit): ❌ scheduled blob-storage exports, ⚠️ NDJSON. Langfuse offers hourly/daily/weekly exports to S3/GCS/Azure (incl. JSONL). Files: routes/compliance.ts, lib/export-signing.ts.

Scope

• Cron-driven export job → S3/GCS/Azure sink (config per org/project).
• NDJSON / JSONL format (fine-tuning friendly) alongside JSON/CSV.
• Reuse the exi

Read the thread · 2026-06-29 · closed · 0 comments

Most recent

End-user (userId) attribution + per-user analytics

Gap (tier-2, should-have): No first-class end-user (userId) dimension. Analytics breaks down by agent/model/session/tool but not by user.

Evidence (audit): routes/analytics.ts — "No user-level breakdown"; only feedback subjectId HMAC attribution exists. Langfuse treats users as first-class (data-model).

Scope

• Thread a userId attribute through ingest (SDK + OTLP), stored on events/traces (verified where possible, else supplied).
• Per-user analytics break

Read the thread · 2026-06-29 · closed · 0 comments

Prompt runtime primitives: variables, chat type, config, placeholders

Gap (tier-1, must-have): Prompt management is versioning-only. Missing the runtime primitives that make prompts usable and drive a playground.

Evidence (audit): ❌ variables ({{var}}), ❌ message placeholders, ❌ config, ❌ composability, ⚠️ chat prompt type — none found in packages/server/src/db/prompt-store.ts. Langfuse features/variables|message-placeholders|config|composability|folders.

Scope

• {{variable}} compilation (SDK + server) with typed/declared variables.

Read the thread · 2026-06-29 · closed · 0 comments

LLM Playground (interactive prompt run + side-by-side compare)

Gap (tier-1, must-have): No LLM Playground — the single most-cited missing feature vs Langfuse. No interactive prompt run/compare UI exists.

Evidence (audit): inventory "Absent entirely"; Langfuse features/playground, side-by-side compare, tool-calling, open-in-playground.

Scope

• Playground UI: edit a prompt, fill variables, pick model/params, run, see output + tokens/cost/latency.
• Server execution endpoint using the LLM-connection store.
• Side-by-side compari

Read the thread · 2026-06-29 · closed · 0 comments

LLM connections / bring-your-own model-key store

Gap (tier-1, must-have): AgentLens has no server-side LLM-connection / bring-your-own model-key store. The SDKs only instrument the user's own in-app provider calls — the server can't itself call a provider. This blocks the Playground and server-side prompt/evaluator execution.

Evidence (audit): Langfuse administration/llm-connection has no AgentLens equivalent; no connection/key store found in packages/server.

Scope

• Encrypted per-(org/project) provider connection s

Read the thread · 2026-06-29 · closed · 0 comments

Phase 6 — Teams, RBAC, SSO & exports

Epic — AgentLens feature roadmap. Tracked under its phase milestone; see #56 for the cross-repo roadmap and #55 for the eval/prompt stack.

Goal

Turn AgentLens's two divergent identity stacks (the OSS tenant_id model and the Postgres cloud/ org model) into one first-class org → project → member hierarchy with unified role-based access, enterprise SSO/SAML login, and asymmetrically signed, third-party-verifiable data exports plus signed outbound webhooks — every record

Read the thread · 2026-06-28 · closed · 0 comments

Phase 3 — Scores & human feedback

Epic — AgentLens feature roadmap. Tracked under its phase milestone; see #56 for the cross-repo roadmap and #55 for the eval/prompt stack.

Goal

Add a first-class human-and-end-user scoring layer to AgentLens: a public, read-friendly scores API that unifies automated and human scores, annotation queues for human review of sessions/traces, and end-user feedback ingestion attached to sessions. Every human score and piece of feedback is stamped with a verified annotator/agent identity a

Read the thread · 2026-06-28 · closed · 0 comments

Phase 0 — Nested execution traces

Epic — AgentLens feature roadmap. Tracked under its phase milestone; see #56 for the cross-repo roadmap and #55 for the eval/prompt stack.

Goal

Preserve the parent/child span structure AgentLens already receives over OTLP — instead of flattening every span to a timestamp-ordered event list — and render it as a collapsible execution tree with a latency waterfall, where each node is agent-attributed and audit-grade.

Why it matters

Today an agent run that arrives as a clean `tra

Read the thread · 2026-06-28 · closed · 0 comments

Billing-grade spend (Slice B): verify agent identity on the OTLP ingest path

Parent epic: agentkitai/agentgate#24. Slice B of 3. Depends on Slice A (verified-id column + billing-grade mode). Design doc: `docs/billing-grade-spend.md`.

Problem

The OTLP ingest path (/v1/traces, routes/otlp.ts::extractAgentId) reads agentlens.agentId / service.name straight from untrusted span attributes — no token, no signature, no server-authoritative stamp. Anyone POSTing OTLP can attribute cost to any agent. This is the spoofability called out in agentgate#24 an

Read the thread · 2026-06-25 · closed · 0 comments

The remaining reports are on the project's issue tracker.