ajaynomics/context-kit
1
0
Fork 0
Code Issues Pull Requests Releases Activity

Overhaul docs retrieval and web search quality

Browse Source 
Replace the abandoned llms-txt-mcp/Chroma docs backend with an in-repo
MCP service: SQLite WAL + FTS5 + sentence-transformer embeddings,
transactional source replacement, persisted state across restarts,
singleflight refresh with conditional requests, hybrid lexical/semantic
ranking with exact-duplicate collapse, source/host filters, and
explicit-by-default content retrieval. Add docs_rebuild and a
docs-rebuild CLI command.

Add deterministic llms-full.txt snapshot generation for machine-local
menus with hash-validated provenance manifests; lifecycle commands
promote a local menu to its snapshot only when the manifest validates.
Switch public source profiles to content-bearing llms-full.txt feeds.

Improve web search: bounded provider fallback with per-attempt
diagnostics and cancellation, an optional Brave Search API provider,
strict SearXNG engine selection, capped link/media extraction, and a
real engine=browser renderer that routes every request through the
existing SSRF vetting while blocking WebSockets, non-GET traffic, and
private destinations.

Extend release checks with offline unit suites and isolated candidate
container tests for both images.
This commit is contained in:
Ajay Krishnan
2026-07-25 08:49:26 -07:00
parent 29bcb123fa
commit 51dceee224
60 changed files with 3207 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
docker/docs/Dockerfile
View File

@@ -1,7 +1,8 @@
FROM python:3.12-slim@sha256:6c4dd321d176d61ea848dc8c73a4f7dbae8f70e0ee48bb411ea2f045b599fa8e
ARG LLMS_TXT_MCP_VERSION=0.2.0
ARG MCP_VERSION=1.28.0
ARG MCP_PROXY_VERSION=0.12.0
ARG SENTENCE_TRANSFORMERS_VERSION=5.6.0
ARG TORCH_VERSION=2.12.1+cpu
COPY constraints.txt /tmp/context-kit-docs-constraints.txt
@@ -11,31 +12,35 @@ RUN apt-get update \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# Install CPU-only torch first so llms-txt-mcp does not pull large CUDA wheels.
# Install CPU-only torch first so sentence-transformers does not pull CUDA wheels.
RUN pip install --no-cache-dir \
--index-url https://download.pytorch.org/whl/cpu \
-c /tmp/context-kit-docs-constraints.txt \
"torch==${TORCH_VERSION}"
# llms-txt-mcp does the indexing/search; mcp-proxy fronts its stdio transport
# as Streamable HTTP so multiple MCP clients can share one long-lived process
# (and therefore one Chroma DB writer).
RUN if [ -n "${LLMS_TXT_MCP_VERSION}" ]; then \
pip install --no-cache-dir -c /tmp/context-kit-docs-constraints.txt "llms-txt-mcp==${LLMS_TXT_MCP_VERSION}"; \
else \
pip install --no-cache-dir -c /tmp/context-kit-docs-constraints.txt llms-txt-mcp; \
fi \
&& pip install --no-cache-dir -c /tmp/context-kit-docs-constraints.txt "mcp-proxy==${MCP_PROXY_VERSION}" \
RUN pip install --no-cache-dir -c /tmp/context-kit-docs-constraints.txt \
"mcp==${MCP_VERSION}" \
"mcp-proxy==${MCP_PROXY_VERSION}" \
"sentence-transformers==${SENTENCE_TRANSFORMERS_VERSION}" \
httpx numpy PyYAML \
&& rm /tmp/context-kit-docs-constraints.txt
COPY context_docs /opt/context-kit/context_docs
COPY tests /opt/context-kit/tests
COPY entrypoint.sh /usr/local/bin/docs-mcp-entrypoint
RUN chmod 0555 /usr/local/bin/docs-mcp-entrypoint
RUN chmod -R a+rX /opt/context-kit \
&& chmod 0555 /usr/local/bin/docs-mcp-entrypoint
RUN mkdir -p /data /models /etc/context-kit
ENV HF_HOME=/models \
HOME=/tmp \
USER=context-kit \
LOGNAME=context-kit \
SENTENCE_TRANSFORMERS_HOME=/models \
PYTHONPATH=/opt/context-kit \
DOCS_MCP_HTTP_HOST=0.0.0.0 \
DOCS_MCP_HTTP_PORT=8000 \
DOCS_MCP_STORE_PATH=/data/docs.sqlite3 \
DOCS_MCP_SOURCES_FILE=/etc/context-kit/docs-sources.txt
VOLUME ["/data", "/models"]