Overhaul docs retrieval and web search quality
Replace the abandoned llms-txt-mcp/Chroma docs backend with an in-repo MCP service: SQLite WAL + FTS5 + sentence-transformer embeddings, transactional source replacement, persisted state across restarts, singleflight refresh with conditional requests, hybrid lexical/semantic ranking with exact-duplicate collapse, source/host filters, and explicit-by-default content retrieval. Add docs_rebuild and a docs-rebuild CLI command. Add deterministic llms-full.txt snapshot generation for machine-local menus with hash-validated provenance manifests; lifecycle commands promote a local menu to its snapshot only when the manifest validates. Switch public source profiles to content-bearing llms-full.txt feeds. Improve web search: bounded provider fallback with per-attempt diagnostics and cancellation, an optional Brave Search API provider, strict SearXNG engine selection, capped link/media extraction, and a real engine=browser renderer that routes every request through the existing SSRF vetting while blocking WebSockets, non-GET traffic, and private destinations. Extend release checks with offline unit suites and isolated candidate container tests for both images.
This commit is contained in:
@@ -64,6 +64,9 @@ Only the variables below are part of the public configuration surface. Other
|
||||
| `CONTEXT_KIT_WEB_SEARCH_PROVIDER` | `searxng` | Default `search_web` provider; fallback order depends on this provider |
|
||||
| `CONTEXT_KIT_WEB_SEARCH_HTTP_TIMEOUT` | `15000` | HTTP timeout in milliseconds for search providers |
|
||||
| `CONTEXT_KIT_WEB_SEARCH_MAX_RESULTS` | `10` | Default search result count when clients omit `limit` |
|
||||
| `CONTEXT_KIT_WEB_SEARCH_MAX_PROVIDER_ATTEMPTS` | `4` | Maximum providers attempted for one search |
|
||||
| `CONTEXT_KIT_WEB_SEARCH_PROVIDER_TIMEOUT` | `15000` | Per-provider diagnostic timeout in milliseconds |
|
||||
| `CONTEXT_KIT_BRAVE_SEARCH_API_KEY` | unset | Optional Brave Search API fallback credential |
|
||||
| `CONTEXT_KIT_WEB_SEARCH_CHROME_PATH` | `/usr/bin/chromium` | Chromium path inside the web-search image for Bing fallback |
|
||||
| `CONTEXT_KIT_WEB_SEARCH_BROWSER_USER_AGENT` | bundled Chrome/Linux UA | User agent for the Chromium-backed Bing fallback |
|
||||
| `CONTEXT_KIT_WEB_SEARCH_MCP_COMPAT_MODE` | unset | Set to `legacy` for MCP clients with weak tool-schema parsers |
|
||||
@@ -74,7 +77,7 @@ Only the variables below are part of the public configuration surface. Other
|
||||
| `CONTEXT_KIT_DOCS_SOURCES` | `config/sources.default.txt` | Space-separated source profile files |
|
||||
| `CONTEXT_KIT_DOCS_MAX_GET_BYTES` | `75000` | Max bytes returned by docs retrieval |
|
||||
| `CONTEXT_KIT_DOCS_EMBED_MODEL` | `BAAI/bge-small-en-v1.5` | SentenceTransformers embedding model |
|
||||
| `CONTEXT_KIT_DOCS_PREINDEX` | `0` | Set to `1` to re-embed every source on container start |
|
||||
| `CONTEXT_KIT_DOCS_PREINDEX` | `0` | Set to `1` to refresh stale/missing sources in the background on startup |
|
||||
| `CONTEXT_KIT_DOCS_LOCAL_SOURCES_DIR` | `${CONTEXT_KIT_DATA_DIR}/local-sources` | Machine-local llms.txt tree mounted read-only into docs-mcp |
|
||||
| `CONTEXT_KIT_DOCS_LOCAL_SOURCES_PORT` | `8769` | Loopback port inside docs-mcp for serving local source files |
|
||||
|
||||
@@ -136,6 +139,10 @@ same-ID `restart` does not apply a changed TTL; it takes effect only when a new
|
||||
container is explicitly provisioned. When freshness matters for one task,
|
||||
prefer `docs_refresh` instead of replacing the shared container.
|
||||
|
||||
Use `bin/context-kit docs-rebuild [SOURCE_URL ...]` after parser/model changes or
|
||||
to force an atomic rebuild. Existing searchable generations remain available if
|
||||
a source fetch, parse, or embedding step fails.
|
||||
|
||||
## Browser CORS
|
||||
|
||||
`context-docs` disables browser CORS by default. CLI assistants and server-side
|
||||
@@ -172,3 +179,10 @@ For local llms.txt files, place content under
|
||||
`http://127.0.0.1:8769/path/inside/local-sources/llms.txt` or another URL that
|
||||
ends in `/llms.txt` or `/llms-full.txt`; that loopback URL is inside the docs-mcp
|
||||
container, not exposed on the host.
|
||||
|
||||
Run `bin/context-kit docs-snapshot [--only DIRECTORY]` to materialize linked
|
||||
local menus. Each successful directory gets `llms-full.txt` and
|
||||
`llms-full.provenance.json`; cache validators live under
|
||||
`${CONTEXT_KIT_DATA_DIR}/snapshot-cache`. `--offline` rebuilds only from that
|
||||
cache. During `start`/`restart`, a local `/llms.txt` URL is automatically changed
|
||||
to its sibling `/llms-full.txt` when that file exists.
|
||||
|
||||
@@ -10,6 +10,11 @@ Context Kit is designed to be safe by default for local development.
|
||||
- The web-search MCP image runs as the non-root `node` user.
|
||||
- Web-search MCP sessions are stateless. Its HTTP front end permits only
|
||||
loopback/internal Host values and rejects every supplied Origin with 403.
|
||||
- Browser fetch intercepts each network GET, resolves it outside Chromium, and
|
||||
blocks private/localhost addresses, non-GET requests, request-count overflow,
|
||||
and byte-budget overflow. Redirect targets are checked independently.
|
||||
- Search diagnostics contain bounded categorized error messages and never emit
|
||||
the optional Brave credential.
|
||||
- Repomix mounts only the current project read-only.
|
||||
- Docs indexing stores data under `$HOME/.local/share/context-kit` unless you
|
||||
override it.
|
||||
@@ -27,6 +32,11 @@ Only index sources you trust enough to retrieve into an agent conversation. More
|
||||
sources are not always better. Large or noisy docs can make retrieval slower and
|
||||
less precise.
|
||||
|
||||
Docs source replacement is transactional. SQLite WAL state persists on the docs
|
||||
volume, removed source profiles become inactive immediately, and full content is
|
||||
not returned by default. Local snapshot provenance is stored separately from the
|
||||
retrieval text so metadata does not pollute ranking.
|
||||
|
||||
## Code-Editing MCP Servers
|
||||
|
||||
Context Kit's default MCP servers either read remote content or mount the
|
||||
|
||||
@@ -133,9 +133,15 @@ race result rendering and return no items even when Chromium sees Bing result
|
||||
cards. The override waits for result cards and decodes current Bing redirect
|
||||
URLs before handing results back to the upstream fallback registry.
|
||||
|
||||
`fetch_url` is different: in upstream `mcp-web-search` 1.3.0, `engine=browser` is
|
||||
accepted but reserved for future support. It does not currently invoke Chromium;
|
||||
URL fetching uses the HTTP extractor path.
|
||||
`search_web` now returns bounded `diagnostics.attempts` entries. Check each
|
||||
provider's `status`, `duration_ms`, `result_count`, and categorized error before
|
||||
changing provider order. An optional Brave API fallback is enabled only when
|
||||
`CONTEXT_KIT_BRAVE_SEARCH_API_KEY` is set.
|
||||
|
||||
`fetch_url engine=browser` invokes Chromium for JavaScript-rendered pages. Every
|
||||
HTTP(S) GET is intercepted and fetched through vetted DNS addresses; non-GET
|
||||
requests, private/localhost destinations, more than 100 requests, and more than
|
||||
20 MiB total browser traffic are blocked. Use `engine=http` for ordinary pages.
|
||||
|
||||
## Docs Indexing Is Slow
|
||||
|
||||
@@ -147,11 +153,10 @@ Cloudflare and other large docs sets can take significantly longer than the
|
||||
default source profile. Set `CONTEXT_KIT_DOCS_PREINDEX=1` only if you want
|
||||
startup to eagerly embed every configured source.
|
||||
|
||||
## Docs Tools Say Index Manager Not Initialized
|
||||
## Docs Sources Report Refresh Errors
|
||||
|
||||
If `docs_query` or `docs_refresh` returns `Index manager not initialized` while
|
||||
`/status` still responds, the HTTP wrapper is up but `llms-txt-mcp` failed to
|
||||
initialize its embedding model or Chroma database. Check the container logs:
|
||||
If `docs_sources` reports `last_error`, the service keeps the previous generation
|
||||
searchable and records the failed check. Check the container logs:
|
||||
|
||||
```sh
|
||||
docker compose -p "${CONTEXT_KIT_COMPOSE_PROJECT:-context-kit}" -f compose.yml logs docs-mcp
|
||||
@@ -173,7 +178,7 @@ sudo chown -R "$(id -u):$(id -g)" "$DATA_DIR/docs" "$DATA_DIR/models"
|
||||
bin/context-kit restart
|
||||
```
|
||||
|
||||
`bin/context-kit start` now pre-creates these directories and `doctor` reports
|
||||
existing directories that are not writable by the current user. If an assistant
|
||||
client reports `Session not found` after restarting `docs-mcp`, restart the
|
||||
assistant so it opens a fresh Streamable HTTP MCP session.
|
||||
`bin/context-kit start` pre-creates these directories and `doctor` reports
|
||||
existing directories that are not writable by the current user. The docs MCP
|
||||
uses stateless HTTP sessions, so clients do not retain a session ID across calls.
|
||||
Use `bin/context-kit docs-rebuild` after fixing the underlying error.
|
||||
|
||||
Reference in New Issue
Block a user