Philipp MockandClaude Opus 5.5
d54b3e5b8b
- prepare_corpus.py: split double-page scans (split_spreads.py), re-OCR with OCRmyPDF/Tesseract only where the existing text layer is worse, Docling layout extraction with heading hierarchy, QA report per PDF - docling_loader.py: cleanup (margin noise, split paragraphs, footnote marks, OCR comma/period repair), page mapping back to the original PDF, chunk headers with author, title and section - local_rag.py: bge-m3 embeddings, configurable chunking and loader, index settings check (index_info.json), hybrid/union BM25 search with Reciprocal Rank Fusion and bge-reranker-v2-m3 cross-encoder; query instruction text removed after it lowered recall - eval_retrieval.py: CLI options, same retrieval path as the chat, tolerant file name matching; goldsets and evaluation reports for the Kafka corpus - grobid_test.py, grobid_qa.py, extract_docling.py, compare_extractors.py: extractor tests and quality checks - .gitignore: corpus PDFs, vector stores and extractor output stay local - server.py: uses ./vectorstore_docling_hdr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
52 lines
604 B
Plaintext
52 lines
604 B
Plaintext
# Python
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
*.so
|
|
.Python
|
|
venv/
|
|
env/
|
|
ENV/
|
|
|
|
# Korpus-PDFs (urheberrechtlich geschützt, nicht ins Repository)
|
|
data/
|
|
data_test/
|
|
|
|
# Vector store
|
|
vectorstore/
|
|
|
|
# Uploads
|
|
uploads/
|
|
|
|
# GROBID-Testausgabe
|
|
grobid_out/
|
|
|
|
# Extraktor-Vergleich (Docling u. a.)
|
|
extract_out/
|
|
|
|
# Tesseract-Sprachmodelle (heruntergeladen)
|
|
tessdata/
|
|
|
|
# Aufbereitetes Korpus (prepare_corpus.py) und Vergleichsindizes
|
|
corpus_processed/
|
|
vectorstore_*/
|
|
|
|
# IDE
|
|
.vscode/
|
|
.idea/
|
|
*.swp
|
|
*.swo
|
|
|
|
# OS
|
|
.DS_Store
|
|
Thumbs.db
|
|
|
|
# Secrets / API keys (never commit these)
|
|
.env
|
|
.env.*
|
|
*.key
|
|
*.pem
|
|
*key*.txt
|
|
*key*.docx
|
|
*key*.zip
|