pod
Other formats agents might prefer:
markdownjsonllms.txt

Agent? You probably want markdown or json, or Pod over MCP.

Reported issues for AgentLens

Pod holds 16 of 16 problems reported by people outside the maintainer team. Issues filed by the project's own owners, members and collaborators are excluded entirely — a maintainer's release checklist is not a warning to a prospective user.

Back to AgentLens.

Most discussed

Roadmap: agent control plane (Tier 2–3)

This is the cross-repo roadmap for agentkitai, derived from the competitive analysis (memory · observability · guardrails). It tracks Tier 2–3 work across agentlens (observe), agentgate (enforce/identify), and lore (remember).

Strategic thesis

The three products map onto the emerging "agent control plane" category. The defensible, underserved wedge is governance / audit / compliance, unified by agent identity — exactly where agentkitai already holds two rare assets (Agent

Read the thread · 2026-06-22 · open · 9 comments

Eval & prompt stack — compliance evals on the tamper-evident audit trail

Roadmap: Tier 3 — Close the eval/prompt quality gap vs Langfuse/LangSmith/Phoenix/Braintrust by leading with a defensible, compliance-first slice: LLM-as-judge scoring anchored to the hash-chained audit trail for tamper-evident evidence. From the agentkitai competitive analysis. Borrowed from: Langfuse (evals, LLM judges, datasets), LangSmith (prompt management, versioning), Phoenix (evaluators), Braintrust (experiments + guardrails gates).

Why

AgentLens today captures a pristine audi

Read the thread · 2026-06-22 · closed · 4 comments

LiveEval: compliance/guardrail scorer over live traffic

Narrow follow-up from #267. The registry scorers (regex/contains/exact_match/llm_judge) are now wired for live eval. Compliance is scored via a separate path (lib/eval/compliance.ts — tool allow/deny lists over the trace's tool_call events), not a registry IScorer. No milestone.

  • Adapt the compliance scoring into an IScorer (or a live-specific runner) that reads the sampled session's tool_call events and evaluates against a configured policy (builtin-evaluators' compliance configs).
  • Regi

Read the thread · 2026-07-01 · closed · 0 comments

RFC 3161: offline CMS/cert-chain verification of timestamp tokens

Narrow follow-up from #99. Requesting + storing the signed TSA token shipped (PR #268: DER TimeStampReq, POST to TSA, store token + PKIStatus + genTime). This adds offline cryptographic verification of a stored token. No milestone.

  • Parse the timeStampToken (CMS SignedData → TSTInfo) and verify: the messageImprint matches the subject hash, the TSA signature is valid, and the signing cert chains to a trusted TSA root.
  • A trusted-cert store (bundled roots + configurable) and CMS/ASN.1 p

Read the thread · 2026-07-01 · closed · 0 comments

Evaluator sampling (online eval)

From #224 (d). No milestone. Run evaluators on a sampled fraction of live traffic. Needs a LiveEvalEngine first (sampler + async evaluator runner over the ingest stream, writing scores back as annotations). Prereq-gated → graduate as its own effort once the online-eval engine exists.

Read the thread · 2026-06-30 · closed · 0 comments

Integrations stretch: JS LlamaIndex + Haystack (follow-up from #152)

Follow-up from #152. The two STRETCH integration targets, explicitly out of #152's acceptance. No milestone (does not gate Phase 7).

#152 delivered the must-haves: Python LlamaIndex (#210) + JS LangChain/LangGraph (#209), plus the existing Python LangChain. Acceptance met.

Remaining (stretch)

  • JS LlamaIndex instrumentation in packages/sdk (mirror the JS LangChain handler src/langchain.ts → Instrumentation.capture; LlamaIndex.TS uses a callback/instrumentation handler).

Read the thread · 2026-06-30 · closed · 0 comments

Epic: Full Postgres runtime parity — port SQLite-only features to the Postgres path

Problem

AgentLens presents a dual-dialect storage layer (SQLite default, Postgres for scale), but runtime parity is far from complete. Authoritative audit (2026-06-30):

  • 53 tables exist in SQLite (db/migrate.sqlite.ts), but only 21 are ever created in Postgres (db/drizzle/*.sql).
  • PostgresEventStore (db/postgres-store.ts) implements only the core observability layer: events, sessions, agents, alert rules/history, analytics, retention, stats. It has zero methods

Read the thread · 2026-06-30 · closed · 0 comments

Scheduled blob-storage exports (S3/GCS/Azure) + NDJSON/JSONL

Gap (tier-2, should-have): Only on-demand signed exports; no scheduled exports to blob storage, and no NDJSON/JSONL.

Evidence (audit): ❌ scheduled blob-storage exports, ⚠️ NDJSON. Langfuse offers hourly/daily/weekly exports to S3/GCS/Azure (incl. JSONL). Files: routes/compliance.ts, lib/export-signing.ts.

Scope

  • Cron-driven export job → S3/GCS/Azure sink (config per org/project).
  • NDJSON / JSONL format (fine-tuning friendly) alongside JSON/CSV.
  • Reuse the exi

Read the thread · 2026-06-29 · closed · 0 comments

Most recent

End-user (userId) attribution + per-user analytics

Gap (tier-2, should-have): No first-class end-user (userId) dimension. Analytics breaks down by agent/model/session/tool but not by user.

Evidence (audit): routes/analytics.ts — "No user-level breakdown"; only feedback subjectId HMAC attribution exists. Langfuse treats users as first-class (data-model).

Scope

  • Thread a userId attribute through ingest (SDK + OTLP), stored on events/traces (verified where possible, else supplied).
  • Per-user analytics break

Read the thread · 2026-06-29 · closed · 0 comments

Prompt runtime primitives: variables, chat type, config, placeholders

Gap (tier-1, must-have): Prompt management is versioning-only. Missing the runtime primitives that make prompts usable and drive a playground.

Evidence (audit): ❌ variables ({{var}}), ❌ message placeholders, ❌ config, ❌ composability, ⚠️ chat prompt type — none found in packages/server/src/db/prompt-store.ts. Langfuse features/variables|message-placeholders|config|composability|folders.

Scope

  • {{variable}} compilation (SDK + server) with typed/declared variables.

Read the thread · 2026-06-29 · closed · 0 comments

LLM Playground (interactive prompt run + side-by-side compare)

Gap (tier-1, must-have): No LLM Playground — the single most-cited missing feature vs Langfuse. No interactive prompt run/compare UI exists.

Evidence (audit): inventory "Absent entirely"; Langfuse features/playground, side-by-side compare, tool-calling, open-in-playground.

Scope

  • Playground UI: edit a prompt, fill variables, pick model/params, run, see output + tokens/cost/latency.
  • Server execution endpoint using the LLM-connection store.
  • Side-by-side compari

Read the thread · 2026-06-29 · closed · 0 comments

LLM connections / bring-your-own model-key store

Gap (tier-1, must-have): AgentLens has no server-side LLM-connection / bring-your-own model-key store. The SDKs only instrument the user's own in-app provider calls — the server can't itself call a provider. This blocks the Playground and server-side prompt/evaluator execution.

Evidence (audit): Langfuse administration/llm-connection has no AgentLens equivalent; no connection/key store found in packages/server.

Scope

  • Encrypted per-(org/project) provider connection s

Read the thread · 2026-06-29 · closed · 0 comments

Phase 6 — Teams, RBAC, SSO & exports

Epic — AgentLens feature roadmap. Tracked under its phase milestone; see #56 for the cross-repo roadmap and #55 for the eval/prompt stack.

Goal

Turn AgentLens's two divergent identity stacks (the OSS tenant_id model and the Postgres cloud/ org model) into one first-class org → project → member hierarchy with unified role-based access, enterprise SSO/SAML login, and asymmetrically signed, third-party-verifiable data exports plus signed outbound webhooks — every record

Read the thread · 2026-06-28 · closed · 0 comments

Phase 3 — Scores & human feedback

Epic — AgentLens feature roadmap. Tracked under its phase milestone; see #56 for the cross-repo roadmap and #55 for the eval/prompt stack.

Goal

Add a first-class human-and-end-user scoring layer to AgentLens: a public, read-friendly scores API that unifies automated and human scores, annotation queues for human review of sessions/traces, and end-user feedback ingestion attached to sessions. Every human score and piece of feedback is stamped with a verified annotator/agent identity a

Read the thread · 2026-06-28 · closed · 0 comments

Phase 0 — Nested execution traces

Epic — AgentLens feature roadmap. Tracked under its phase milestone; see #56 for the cross-repo roadmap and #55 for the eval/prompt stack.

Goal

Preserve the parent/child span structure AgentLens already receives over OTLP — instead of flattening every span to a timestamp-ordered event list — and render it as a collapsible execution tree with a latency waterfall, where each node is agent-attributed and audit-grade.

Why it matters

Today an agent run that arrives as a clean `tra

Read the thread · 2026-06-28 · closed · 0 comments

Billing-grade spend (Slice B): verify agent identity on the OTLP ingest path

Parent epic: agentkitai/agentgate#24. Slice B of 3. Depends on Slice A (verified-id column + billing-grade mode). Design doc: `docs/billing-grade-spend.md`.

Problem

The OTLP ingest path (/v1/traces, routes/otlp.ts::extractAgentId) reads agentlens.agentId / service.name straight from untrusted span attributes — no token, no signature, no server-authoritative stamp. Anyone POSTing OTLP can attribute cost to any agent. This is the spoofability called out in agentgate#24 an

Read the thread · 2026-06-25 · closed · 0 comments

The remaining reports are on the project's issue tracker.