Overhaul docs retrieval and web search quality
Replace the abandoned llms-txt-mcp/Chroma docs backend with an in-repo MCP service: SQLite WAL + FTS5 + sentence-transformer embeddings, transactional source replacement, persisted state across restarts, singleflight refresh with conditional requests, hybrid lexical/semantic ranking with exact-duplicate collapse, source/host filters, and explicit-by-default content retrieval. Add docs_rebuild and a docs-rebuild CLI command. Add deterministic llms-full.txt snapshot generation for machine-local menus with hash-validated provenance manifests; lifecycle commands promote a local menu to its snapshot only when the manifest validates. Switch public source profiles to content-bearing llms-full.txt feeds. Improve web search: bounded provider fallback with per-attempt diagnostics and cancellation, an optional Brave Search API provider, strict SearXNG engine selection, capped link/media extraction, and a real engine=browser renderer that routes every request through the existing SSRF vetting while blocking WebSockets, non-GET traffic, and private destinations. Extend release checks with offline unit suites and isolated candidate container tests for both images.
This commit is contained in:
22
scripts/docs-rebuild.mjs
Normal file
22
scripts/docs-rebuild.mjs
Normal file
@@ -0,0 +1,22 @@
|
||||
import { rpc } from "../docker/web-search/mcp-probe.mjs";
|
||||
|
||||
const [url, ...sources] = process.argv.slice(2);
|
||||
if (!url) throw new Error("usage: node scripts/docs-rebuild.mjs <mcp-url> [source ...]");
|
||||
|
||||
await rpc(url, 1, "initialize", {
|
||||
protocolVersion: "2024-11-05",
|
||||
capabilities: {},
|
||||
clientInfo: { name: "context-kit-docs-rebuild", version: "1" }
|
||||
}, 10_000);
|
||||
const result = await rpc(url, 2, "tools/call", {
|
||||
name: "docs_rebuild",
|
||||
arguments: sources.length ? { sources } : {}
|
||||
}, 600_000);
|
||||
if (result.isError) {
|
||||
const text = (result.content || []).map(part => part.text || "").join("\n");
|
||||
throw new Error(text || "docs_rebuild failed");
|
||||
}
|
||||
const structured = result.structuredContent || JSON.parse(
|
||||
(result.content || []).find(part => part.type === "text")?.text || "{}"
|
||||
);
|
||||
console.log(JSON.stringify(structured, null, 2));
|
||||
Reference in New Issue
Block a user