pmock/localRAG
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add LLM judge to retrieval eval; switch chat to pypdf with 2000/400 chunks

Browse Source 
- eval_retrieval.py: --judge MODEL lets an OpenAI model decide whether the
  top-5 chunks answer each question, regardless of source and page; reports
  Answer@5, answer-level MRR and agreement with the page check; verdicts are
  cached in eval/judge_cache.json
- On the curated goldset, gpt-6-luna rates pypdf and Docling equal
  (Answer@5 0.93 each, MRR 0.90 vs 0.92), so the chat stays on pypdf for now
- local_rag.py: chunks 2000/400, chunk headers off by default, OpenAI model
  gpt-6-luna; server.py loads ./vectorstore_pypdf_2000

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Philipp Mock
2026-10-08 09:07:51 +02:00
co-authored by Claude Opus 5.5
parent 9ebcda1f55
commit e75716714e
7 changed files with 804 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
server.py
+5 -4
View File
@@ -12,10 +12,11 @@ from pydantic import BaseModel
from local_rag import LocalRAG
OLLAMA_MODEL = "gpt-oss:20b"
OPENAI_MODEL = "gpt-5-mini"
# Kafka corpus: Docling pipeline (split double pages) + chunk headers, bge-m3, hybrid search + reranker
# built with: add_pdfs.py --data data --index ./vectorstore_docling_hdr --loader docling
VECTORSTORE_PATH = "./vectorstore_docling_hdr"
OPENAI_MODEL = "gpt-6-luna"
# Kafka corpus: PyPDFLoader, chunks 2000/400 without headers, bge-m3, hybrid search + reranker.
# Docling was not better at retrieval (eval/kafka_human_judge_*.txt), so pypdf stays for now.
# built with: add_pdfs.py --data data --index ./vectorstore_pypdf_2000 --loader pypdf --no-headers
VECTORSTORE_PATH = "./vectorstore_pypdf_2000"
# Dual RAG instances for on-the-fly provider switching
rag_ollama = LocalRAG(