ajaynomics/context-kit
1
0
Fork 0
Code Issues Pull Requests Releases Activity

Fix ID collisions for repeated section titles

Browse Source 
Large llms-full.txt feeds repeat section headings, which made document
identity hashes collide within one source and abort indexing on the
documents primary key. Include each document's ordinal in the identity.
This commit is contained in:
Ajay Krishnan
2026-07-25 09:00:59 -07:00
parent 51dceee224
commit ac3465c656
2 changed files with 22 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docker/docs/context_docs/refresh.py
View File

@@ -65,10 +65,12 @@ class RefreshCoordinator:
vectors = await self.embedder.encode_documents(texts) if texts else []
documents: list[PreparedDocument] = []
source_host = (urlparse(response.resolved_url).hostname or "").lower()
for parsed_document, vector in zip(parsed.documents, vectors, strict=True):
for ordinal, (parsed_document, vector) in enumerate(zip(parsed.documents, vectors, strict=True)):
content_hash = hashlib.sha256(parsed_document.content.encode()).hexdigest()
# Include the ordinal so repeated section titles (common in large
# llms-full.txt feeds) cannot collide on the primary key.
identity = "\0".join(
[source, parsed_document.canonical_url, parsed_document.heading_path, str(parsed_document.chunk_index)]
[source, str(ordinal), parsed_document.canonical_url, parsed_document.heading_path, str(parsed_document.chunk_index)]
)
documents.append(
PreparedDocument(