pmock/localRAG
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add LLM judge to retrieval eval; switch chat to pypdf with 2000/400 chunks

Browse Source 
- eval_retrieval.py: --judge MODEL lets an OpenAI model decide whether the
  top-5 chunks answer each question, regardless of source and page; reports
  Answer@5, answer-level MRR and agreement with the page check; verdicts are
  cached in eval/judge_cache.json
- On the curated goldset, gpt-6-luna rates pypdf and Docling equal
  (Answer@5 0.93 each, MRR 0.90 vs 0.92), so the chat stays on pypdf for now
- local_rag.py: chunks 2000/400, chunk headers off by default, OpenAI model
  gpt-6-luna; server.py loads ./vectorstore_pypdf_2000

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Philipp Mock
2026-10-08 09:07:51 +02:00
co-authored by Claude Opus 5.5
parent 9ebcda1f55
commit e75716714e
7 changed files with 804 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
add_pdfs.py
+1 -1
View File
@@ -13,7 +13,7 @@ from local_rag import CHUNK_OVERLAP, CHUNK_SIZE, PDF_LOADER, PROCESSED_DIR, Loca
# Folder to scan for PDFs (recursively).
DATA_ROOT = Path("/Users/Philipp/Desktop/workspace/python/gpt_publikationen/data_vs")
# The chat (server.py) currently loads ./vectorstore_docling_hdr; pass --index to build that one.
# The chat (server.py) currently loads ./vectorstore_pypdf_2000; pass --index to build that one.
VECTORSTORE_PATH = "./vectorstore"
# If True: delete the existing vector store folder, then index all PDFs from scratch.