Overhaul docs retrieval and web search quality
Replace the abandoned llms-txt-mcp/Chroma docs backend with an in-repo MCP service: SQLite WAL + FTS5 + sentence-transformer embeddings, transactional source replacement, persisted state across restarts, singleflight refresh with conditional requests, hybrid lexical/semantic ranking with exact-duplicate collapse, source/host filters, and explicit-by-default content retrieval. Add docs_rebuild and a docs-rebuild CLI command. Add deterministic llms-full.txt snapshot generation for machine-local menus with hash-validated provenance manifests; lifecycle commands promote a local menu to its snapshot only when the manifest validates. Switch public source profiles to content-bearing llms-full.txt feeds. Improve web search: bounded provider fallback with per-attempt diagnostics and cancellation, an optional Brave Search API provider, strict SearXNG engine selection, capped link/media extraction, and a real engine=browser renderer that routes every request through the existing SSRF vetting while blocking WebSockets, non-GET traffic, and private destinations. Extend release checks with offline unit suites and isolated candidate container tests for both images.
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
import { requireToolSuccess, runSmoke } from "./mcp-smoke-client.mjs";
|
||||
import { requireToolSuccess, runSmoke, textFrom } from "./mcp-smoke-client.mjs";
|
||||
|
||||
const live = process.env.CONTEXT_KIT_LIVE_CHECKS === "1";
|
||||
const localSourceSmokeUrl = process.env.CONTEXT_KIT_LOCAL_SOURCE_SMOKE_URL;
|
||||
@@ -12,7 +12,8 @@ runSmoke({
|
||||
const toolNames = await client.requireTools(["docs_query", "docs_sources"]);
|
||||
|
||||
const sources = requireToolSuccess("docs_sources", await client.callTool("docs_sources"));
|
||||
if (!Array.isArray(sources?.structuredContent?.result)) {
|
||||
const sourcesPayload = sources?.structuredContent || JSON.parse(textFrom(sources) || "null");
|
||||
if (typeof sourcesPayload?.source_count !== "number") {
|
||||
const sourcesText = JSON.stringify(sources);
|
||||
throw new Error(`docs_sources returned unexpected payload: ${sourcesText.slice(0, 500)}`);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user