Philipp MockandClaude Opus 5.5
d54b3e5b8b
- prepare_corpus.py: split double-page scans (split_spreads.py), re-OCR with OCRmyPDF/Tesseract only where the existing text layer is worse, Docling layout extraction with heading hierarchy, QA report per PDF - docling_loader.py: cleanup (margin noise, split paragraphs, footnote marks, OCR comma/period repair), page mapping back to the original PDF, chunk headers with author, title and section - local_rag.py: bge-m3 embeddings, configurable chunking and loader, index settings check (index_info.json), hybrid/union BM25 search with Reciprocal Rank Fusion and bge-reranker-v2-m3 cross-encoder; query instruction text removed after it lowered recall - eval_retrieval.py: CLI options, same retrieval path as the chat, tolerant file name matching; goldsets and evaluation reports for the Kafka corpus - grobid_test.py, grobid_qa.py, extract_docling.py, compare_extractors.py: extractor tests and quality checks - .gitignore: corpus PDFs, vector stores and extractor output stay local - server.py: uses ./vectorstore_docling_hdr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
15 lines
211 B
Plaintext
15 lines
211 B
Plaintext
langchain
|
|
langchain-community
|
|
langchain-ollama
|
|
langchain-openai
|
|
langchain-text-splitters
|
|
langchain-huggingface
|
|
faiss-cpu
|
|
sentence-transformers
|
|
pypdf
|
|
numpy<2.0
|
|
fastapi
|
|
uvicorn[standard]
|
|
rank_bm25
|
|
snowballstemmer
|