Overhaul docs retrieval and web search quality
Replace the abandoned llms-txt-mcp/Chroma docs backend with an in-repo MCP service: SQLite WAL + FTS5 + sentence-transformer embeddings, transactional source replacement, persisted state across restarts, singleflight refresh with conditional requests, hybrid lexical/semantic ranking with exact-duplicate collapse, source/host filters, and explicit-by-default content retrieval. Add docs_rebuild and a docs-rebuild CLI command. Add deterministic llms-full.txt snapshot generation for machine-local menus with hash-validated provenance manifests; lifecycle commands promote a local menu to its snapshot only when the manifest validates. Switch public source profiles to content-bearing llms-full.txt feeds. Improve web search: bounded provider fallback with per-attempt diagnostics and cancellation, an optional Brave Search API provider, strict SearXNG engine selection, capped link/media extraction, and a real engine=browser renderer that routes every request through the existing SSRF vetting while blocking WebSockets, non-GET traffic, and private destinations. Extend release checks with offline unit suites and isolated candidate container tests for both images.
This commit is contained in:
@@ -36,6 +36,9 @@ services:
|
||||
HTTP_TIMEOUT: "${CONTEXT_KIT_WEB_SEARCH_HTTP_TIMEOUT:-15000}"
|
||||
MAX_BYTES: "${CONTEXT_KIT_WEB_SEARCH_MAX_BYTES:-52428800}"
|
||||
MAX_RESULTS: "${CONTEXT_KIT_WEB_SEARCH_MAX_RESULTS:-10}"
|
||||
MAX_PROVIDER_ATTEMPTS: "${CONTEXT_KIT_WEB_SEARCH_MAX_PROVIDER_ATTEMPTS:-4}"
|
||||
SEARCH_PROVIDER_TIMEOUT_MS: "${CONTEXT_KIT_WEB_SEARCH_PROVIDER_TIMEOUT:-15000}"
|
||||
BRAVE_SEARCH_API_KEY: "${CONTEXT_KIT_BRAVE_SEARCH_API_KEY:-}"
|
||||
BROWSER_SEARCH_USER_AGENT: "${CONTEXT_KIT_WEB_SEARCH_BROWSER_USER_AGENT:-}"
|
||||
MCP_COMPAT_MODE: "${CONTEXT_KIT_WEB_SEARCH_MCP_COMPAT_MODE:-}"
|
||||
healthcheck:
|
||||
@@ -54,8 +57,8 @@ services:
|
||||
build:
|
||||
context: ./docker/docs
|
||||
image: ${CONTEXT_KIT_DOCS_IMAGE:-context-kit/docs-mcp:latest}
|
||||
# Long-lived shared docs MCP. One container = one Chroma writer; clients
|
||||
# connect over Streamable HTTP (mcp-proxy bridges llms-txt-mcp's stdio).
|
||||
# Long-lived shared docs MCP. One container owns the transactional SQLite
|
||||
# index and embedding model; clients connect over Streamable HTTP.
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:${CONTEXT_KIT_DOCS_PORT:-8776}:8000"
|
||||
|
||||
Reference in New Issue
Block a user