Skip to content

🦭 PromptSeal — Product Roadmap

Mission: make LLM behavior regressions impossible to ship. Vision: the default safety net under every prompt, agent, and model swap — the way pytest is the default safety net under Python code.

North-star metrics

Metric 3 months 6 months 12 months
GitHub stars 1,000 5,000 15,000
PyPI downloads / month 10k 100k 500k
Teams gating CI on PromptSeal 50 500 3,000
Community-contributed suites 10 50 200

Phase 0 — Core Loop ✅ (shipped in v0.1)

The one-command regression loop for individual developers.

  • [x] init — scaffold promptseal.yaml + starter suite
  • [x] YAML case format with variables, system prompts, tags
  • [x] 10 built-in assertions (text, JSON, regex, judge, latency, cost, length)
  • [x] Providers: any OpenAI-compatible endpoint (OpenAI, OpenRouter, Ollama, vLLM)
    • deterministic mock provider for offline demos/tests
  • [x] Baseline sealing (run --save-baseline) and diff (regression / improvement / stable)
  • [x] ci command: exit codes, GitHub step summaries, min_pass_rate
  • [x] Self-contained dark-theme HTML reports (no external assets)
  • [x] LLM-as-judge with configurable judge provider
  • [x] Multi-model matrix runs: run -p a -p b -p c → scorecard + recommendation + HTML
  • [x] seal command (run + baseline in one), --json output on run/diff
  • [x] 18 built-in assertions (text, JSON, regex, shape, judge, latency, cost, length, agent tools)

Phase 1 — Capture Reality (v0.2–0.3, weeks 2–6)

Theme: stop hand-writing cases. Seal what your app actually does.

  • [x] promptseal record — local OpenAI-compatible reverse proxy that captures real app traffic and turns it into draft cases (PII redaction built in)
  • [x] Official GitHub Action — ArsinShaabani/promptseal-action (composite action: provider/min-pass-rate/working-directory inputs, step summary, exit codes)
  • [x] Hero demo GIF (assets/demo.gif, generated by scripts/make_demo_gif.py)
  • [x] Python SDK (promptseal.capture) for in-process recording — v0.3
  • [ ] PR delta comments (inline review comments, beyond the step summary)
  • [x] pytest plugin — pytest --promptseal runs suites as tests; the session is saved as a PromptSeal run so seal/diff/ci keep working — v0.3
  • [x] Flaky-case detection — run --repeat N + statistical flaky_pass_rate — v0.3
  • [x] Committed baselines — baseline.json embeds the full run snapshot and is git-whitelisted, so CI gates work with zero extra wiring — v0.3
  • [x] Docs site (mkdocs-material) — docs/ + auto-deploy workflow to GitHub Pages (enable Pages once → deploy from gh-pages) — v0.3

Exit criteria: a developer can go from zero → CI-gated evals of real traffic in under 10 minutes.

Phase 2 — Agent-Native (v0.4–0.5, months 2–3)

Theme: evaluate what agents DO, not just what they say.

  • [x] Tool-call trace assertions — tools_called / tools_not_called / call_order / tool_args on any OpenAI-compatible tool-calling model (mock:tools offline) — v0.4
  • [x] Multi-turn conversation cases — case-level messages: chat history with {{vars}} rendering, plus scripted-user simulation (script:) — v0.4/v0.6
  • [x] Adapters — promptseal.adapters: evaluate traces from LangChain-style messages (dicts or duck-typed objects), OpenAI Agents SDK function_call items, or any OpenAI-format trace (no SDK needed); Claude Code hook recipe in the tutorial — v0.6
  • [x] driftwatch long-term view — promptseal driftwatch: self-contained HTML dashboard charting pass-rate/cost/latency across every saved run (reads .promptseal/runs JSON — boring storage beats SQLite) — v0.4
  • [x] Assertion plugins (entry-point group promptseal.assertions) + promptseal doctor self-check (config / providers / judge / registry / storage) — v0.4
  • [x] Cost observability — per-case cost/latency percentiles (p50/p95/max) across providers in the driftwatch dashboard + driftwatch.per_case_stats() API — v0.6

Phase 3 — Teams & Scale (v0.6–1.0, months 3–6)

Theme: from personal tool to team platform — still local-first, now multi-tenant.

  • [ ] promptseal server (optional, self-hosted): web UI over shared run history, team dashboards, drift alerts (Slack/Discord/email webhooks)
  • [ ] Suite inheritance for monorepos (extends:), org-wide shared baselines
  • [ ] Storage backends: S3 / GCS / git-LFS for run artifacts
  • [ ] RBAC via git provider permissions; audit log of every seal/diff
  • [ ] Scheduled shadow runs against production traffic samples
  • [ ] SOC2-friendly posture: no data egress, self-hosting guide, signed releases

Phase 4 — Ecosystem (months 6–12)

Theme: become the registry where the community shares behavior specs.

  • [ ] Suite Registry: promptseal install security-basics — curated, versioned, community suites (PII, jailbreak resistance, tone, JSON contracts)
  • [ ] "Sealed" badge for open-source projects that gate their LLM features on PromptSeal
  • [ ] TypeScript/JS SDK + MCP server ("ask my seal history")
  • [ ] Model leaderboard per vertical: community-voted suites scored across popular models
  • [ ] Enterprise: SSO, audit exports, on-prem runner

Launch & distribution playbook

Status: v0.6.0 is LIVE on PyPI — pip install promptseal works globally. GitHub Release published, Discussions enabled, bilingual tutorials shipped. v0.3.0: committed self-contained baselines, flaky-repeat runs, promptseal.capture SDK, pytest plugin (pytest --promptseal), mkdocs-material docs site. v0.4.0: agent tool-call assertions + multi-turn cases, promptseal driftwatch, assertion plugins + promptseal doctor, run-id collision fix. v0.5.0: case filtering (--tags/--case/--skip-case/--list), execution control (--fail-fast, --concurrency + defaults.concurrency), runs --json, diff --fail-on-regression, init --provider/--suite, global --version flag. v0.6.0: scripted-user simulation (script:), framework adapters (promptseal.adapters), per-case cost percentiles in driftwatch — Phase 2 complete. v0.7.0 — Phase 3 features: extends: suite inheritance, PROMPTSEAL_HOME shared storage, promptseal audit, promptseal server (read-only local dashboard/API), ci.alert_webhook drift alerts. v0.8.0 — Reliability & trust: retries + jitter on 429/5xx, case-level params: passthrough, deterministic judge.params, persisted judge-answer cache, XSS-safe reports, honest webhooks, recorder GET /v1/models, deep-merged provider config, filtered CI diffs, CHANGELOG, version-pinned action + comment-on-failure.

Live now: PyPI 0.8.0 (verified with a fresh-venv install) · GitHub Release v0.8.0 · docs site https://arsinshaabani.github.io/promptseal/ (Pages: legacy/gh-pages, changelog/ page published) · promptseal-action v1.2.0 released (comment-on-failure via always(), version: pin, python_version validation, e2e smoke CI) and the v1 tag moved to it.

Live now: PyPI 0.6.0 (verified with a fresh-venv install) · GitHub Release v0.6.0 · docs site https://arsinshaabani.github.io/promptseal/ (Pages: legacy/gh-pages, build green) · promptseal-action v1.1.0 released and the v1 tag moved to it.

  1. Assets first: demo GIF (vhs/asciinema), landing page (GitHub Pages), docs site. ✅ GIF shipped (assets/demo.gif) · ✅ docs site shipped (docs/, mkdocs-material, auto-deploys on push — enable GitHub Pages once, source: gh-pages branch).
  2. Launch week: Show HN, r/LocalLLaMA + r/LLMDevs, X thread with the diff GIF, dev.to deep-dive (feeds daily.dev), Persian dev communities. Drafts ready in marketing/posts/ (local).
  3. SEO: own the phrases prompt regression testing, LLM regression CI, prompt drift with comparison + tutorial content.
  4. Distribution: PRs to awesome-llm / awesome-ai-agents lists, integration announcements with Ollama/OpenRouter communities, "Sealed" badge backlinks.
  5. Cadence: weekly changelogs, one deep-dive post per phase, every release ships a demo.

Non-goals

  • No hosted SaaS for your prompt data. (Self-hosted server only — your traces stay yours.)
  • No generic "AI platform" sprawl. We do regression testing; we do it perfectly.
  • No vendor-exclusive formats. YAML in, JSON out, OpenAI-compatible everywhere.