ajaynomics/context-kit
1
0
Fork 0
Code Issues Pull Requests Releases Activity

Overhaul docs retrieval and web search quality

Browse Source 
Replace the abandoned llms-txt-mcp/Chroma docs backend with an in-repo
MCP service: SQLite WAL + FTS5 + sentence-transformer embeddings,
transactional source replacement, persisted state across restarts,
singleflight refresh with conditional requests, hybrid lexical/semantic
ranking with exact-duplicate collapse, source/host filters, and
explicit-by-default content retrieval. Add docs_rebuild and a
docs-rebuild CLI command.

Add deterministic llms-full.txt snapshot generation for machine-local
menus with hash-validated provenance manifests; lifecycle commands
promote a local menu to its snapshot only when the manifest validates.
Switch public source profiles to content-bearing llms-full.txt feeds.

Improve web search: bounded provider fallback with per-attempt
diagnostics and cancellation, an optional Brave Search API provider,
strict SearXNG engine selection, capped link/media extraction, and a
real engine=browser renderer that routes every request through the
existing SSRF vetting while blocking WebSockets, non-GET traffic, and
private destinations.

Extend release checks with offline unit suites and isolated candidate
container tests for both images.
This commit is contained in:
Ajay Krishnan
2026-07-25 08:49:26 -07:00
parent 29bcb123fa
commit 51dceee224
60 changed files with 3207 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
.env.example
View File

@@ -14,14 +14,18 @@ CONTEXT_KIT_SEARXNG_PORT=8099
# Keep this aligned with agent tool-call defaults to avoid schema rejections.
CONTEXT_KIT_WEB_SEARCH_MAX_BYTES=52428800
# Web-search defaults. Search uses SearXNG first, then falls back to
# DuckDuckGo and Bing. Bing requires Chromium inside the web-search image.
# Web-search defaults. Search uses SearXNG first, then bounded fallbacks with
# per-attempt diagnostics. Bing and engine=browser use Chromium.
CONTEXT_KIT_WEB_SEARCH_PORT=8777
# Override only for a loopback proxy that preserves the Host/Origin policy.
# CONTEXT_KIT_WEB_SEARCH_HTTP_URL=http://127.0.0.1:8777/mcp
CONTEXT_KIT_WEB_SEARCH_PROVIDER=searxng
CONTEXT_KIT_WEB_SEARCH_HTTP_TIMEOUT=15000
CONTEXT_KIT_WEB_SEARCH_MAX_RESULTS=10
CONTEXT_KIT_WEB_SEARCH_MAX_PROVIDER_ATTEMPTS=4
CONTEXT_KIT_WEB_SEARCH_PROVIDER_TIMEOUT=15000
# Optional hosted fallback. Context Kit remains fully usable without it.
# CONTEXT_KIT_BRAVE_SEARCH_API_KEY=
CONTEXT_KIT_WEB_SEARCH_CHROME_PATH=/usr/bin/chromium
# User agent used by the Chromium-backed Bing search fallback.
# CONTEXT_KIT_WEB_SEARCH_BROWSER_USER_AGENT="Mozilla/5.0 ..."
@@ -41,7 +45,7 @@ CONTEXT_KIT_DOCS_MAX_GET_BYTES=75000
CONTEXT_KIT_DOCS_EMBED_MODEL=BAAI/bge-small-en-v1.5
# Eagerly index every source on container start. Off by default so startup is
# fast; call the docs_refresh MCP tool when you want to populate the index.
# fast; call docs_refresh or `bin/context-kit docs-rebuild` to populate it.
# CONTEXT_KIT_DOCS_PREINDEX=1
# One or more source files, separated by spaces. Keep committed profiles generic.