Overhaul docs retrieval and web search quality

Replace the abandoned llms-txt-mcp/Chroma docs backend with an in-repo
MCP service: SQLite WAL + FTS5 + sentence-transformer embeddings,
transactional source replacement, persisted state across restarts,
singleflight refresh with conditional requests, hybrid lexical/semantic
ranking with exact-duplicate collapse, source/host filters, and
explicit-by-default content retrieval. Add docs_rebuild and a
docs-rebuild CLI command.

Add deterministic llms-full.txt snapshot generation for machine-local
menus with hash-validated provenance manifests; lifecycle commands
promote a local menu to its snapshot only when the manifest validates.
Switch public source profiles to content-bearing llms-full.txt feeds.

Improve web search: bounded provider fallback with per-attempt
diagnostics and cancellation, an optional Brave Search API provider,
strict SearXNG engine selection, capped link/media extraction, and a
real engine=browser renderer that routes every request through the
existing SSRF vetting while blocking WebSockets, non-GET traffic, and
private destinations.

Extend release checks with offline unit suites and isolated candidate
container tests for both images.
This commit is contained in:
2026-07-25 08:49:26 -07:00
parent 29bcb123fa
commit 51dceee224
60 changed files with 3207 additions and 107 deletions

View File

@@ -10,6 +10,11 @@ Context Kit is designed to be safe by default for local development.
- The web-search MCP image runs as the non-root `node` user.
- Web-search MCP sessions are stateless. Its HTTP front end permits only
loopback/internal Host values and rejects every supplied Origin with 403.
- Browser fetch intercepts each network GET, resolves it outside Chromium, and
blocks private/localhost addresses, non-GET requests, request-count overflow,
and byte-budget overflow. Redirect targets are checked independently.
- Search diagnostics contain bounded categorized error messages and never emit
the optional Brave credential.
- Repomix mounts only the current project read-only.
- Docs indexing stores data under `$HOME/.local/share/context-kit` unless you
override it.
@@ -27,6 +32,11 @@ Only index sources you trust enough to retrieve into an agent conversation. More
sources are not always better. Large or noisy docs can make retrieval slower and
less precise.
Docs source replacement is transactional. SQLite WAL state persists on the docs
volume, removed source profiles become inactive immediately, and full content is
not returned by default. Local snapshot provenance is stored separately from the
retrieval text so metadata does not pollute ranking.
## Code-Editing MCP Servers
Context Kit's default MCP servers either read remote content or mount the