Overhaul docs retrieval and web search quality

Replace the abandoned llms-txt-mcp/Chroma docs backend with an in-repo
MCP service: SQLite WAL + FTS5 + sentence-transformer embeddings,
transactional source replacement, persisted state across restarts,
singleflight refresh with conditional requests, hybrid lexical/semantic
ranking with exact-duplicate collapse, source/host filters, and
explicit-by-default content retrieval. Add docs_rebuild and a
docs-rebuild CLI command.

Add deterministic llms-full.txt snapshot generation for machine-local
menus with hash-validated provenance manifests; lifecycle commands
promote a local menu to its snapshot only when the manifest validates.
Switch public source profiles to content-bearing llms-full.txt feeds.

Improve web search: bounded provider fallback with per-attempt
diagnostics and cancellation, an optional Brave Search API provider,
strict SearXNG engine selection, capped link/media extraction, and a
real engine=browser renderer that routes every request through the
existing SSRF vetting while blocking WebSockets, non-GET traffic, and
private destinations.

Extend release checks with offline unit suites and isolated candidate
container tests for both images.
This commit is contained in:
2026-07-25 08:49:26 -07:00
parent 29bcb123fa
commit 51dceee224
60 changed files with 3207 additions and 107 deletions

View File

@@ -36,6 +36,9 @@ services:
HTTP_TIMEOUT: "${CONTEXT_KIT_WEB_SEARCH_HTTP_TIMEOUT:-15000}"
MAX_BYTES: "${CONTEXT_KIT_WEB_SEARCH_MAX_BYTES:-52428800}"
MAX_RESULTS: "${CONTEXT_KIT_WEB_SEARCH_MAX_RESULTS:-10}"
MAX_PROVIDER_ATTEMPTS: "${CONTEXT_KIT_WEB_SEARCH_MAX_PROVIDER_ATTEMPTS:-4}"
SEARCH_PROVIDER_TIMEOUT_MS: "${CONTEXT_KIT_WEB_SEARCH_PROVIDER_TIMEOUT:-15000}"
BRAVE_SEARCH_API_KEY: "${CONTEXT_KIT_BRAVE_SEARCH_API_KEY:-}"
BROWSER_SEARCH_USER_AGENT: "${CONTEXT_KIT_WEB_SEARCH_BROWSER_USER_AGENT:-}"
MCP_COMPAT_MODE: "${CONTEXT_KIT_WEB_SEARCH_MCP_COMPAT_MODE:-}"
healthcheck:
@@ -54,8 +57,8 @@ services:
build:
context: ./docker/docs
image: ${CONTEXT_KIT_DOCS_IMAGE:-context-kit/docs-mcp:latest}
# Long-lived shared docs MCP. One container = one Chroma writer; clients
# connect over Streamable HTTP (mcp-proxy bridges llms-txt-mcp's stdio).
# Long-lived shared docs MCP. One container owns the transactional SQLite
# index and embedding model; clients connect over Streamable HTTP.
restart: unless-stopped
ports:
- "127.0.0.1:${CONTEXT_KIT_DOCS_PORT:-8776}:8000"