- eval_retrieval.py: --judge MODEL lets an OpenAI model decide whether the
top-5 chunks answer each question, regardless of source and page; reports
Answer@5, answer-level MRR and agreement with the page check; verdicts are
cached in eval/judge_cache.json
- On the curated goldset, gpt-6-luna rates pypdf and Docling equal
(Answer@5 0.93 each, MRR 0.90 vs 0.92), so the chat stays on pypdf for now
- local_rag.py: chunks 2000/400, chunk headers off by default, OpenAI model
gpt-6-luna; server.py loads ./vectorstore_pypdf_2000
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- prepare_corpus.py: split double-page scans (split_spreads.py), re-OCR with
OCRmyPDF/Tesseract only where the existing text layer is worse, Docling
layout extraction with heading hierarchy, QA report per PDF
- docling_loader.py: cleanup (margin noise, split paragraphs, footnote marks,
OCR comma/period repair), page mapping back to the original PDF, chunk
headers with author, title and section
- local_rag.py: bge-m3 embeddings, configurable chunking and loader, index
settings check (index_info.json), hybrid/union BM25 search with Reciprocal
Rank Fusion and bge-reranker-v2-m3 cross-encoder; query instruction text
removed after it lowered recall
- eval_retrieval.py: CLI options, same retrieval path as the chat, tolerant
file name matching; goldsets and evaluation reports for the Kafka corpus
- grobid_test.py, grobid_qa.py, extract_docling.py, compare_extractors.py:
extractor tests and quality checks
- .gitignore: corpus PDFs, vector stores and extractor output stay local
- server.py: uses ./vectorstore_docling_hdr
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Philipp Mock
