pmock/localRAG
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e75716714e42b723d62bd0c4b967b136b1cfaf80
localRAG/eval
T
History
Philipp MockandClaude Opus 5.5 e75716714e Add LLM judge to retrieval eval; switch chat to pypdf with 2000/400 chunks 
- eval_retrieval.py: --judge MODEL lets an OpenAI model decide whether the
  top-5 chunks answer each question, regardless of source and page; reports
  Answer@5, answer-level MRR and agreement with the page check; verdicts are
  cached in eval/judge_cache.json
- On the curated goldset, gpt-6-luna rates pypdf and Docling equal
  (Answer@5 0.93 each, MRR 0.90 vs 0.92), so the chat stays on pypdf for now
- local_rag.py: chunks 2000/400, chunk headers off by default, OpenAI model
  gpt-6-luna; server.py loads ./vectorstore_pypdf_2000

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:07:51 +02:00
..
build_docling_2000_hdr.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
build_docling_2000.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
build_docling_hdr.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
build_docling_split.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
build_docling.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
build_pypdf_2000_hdr.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
build_pypdf_2000.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
build_pypdf_hdr.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
build_pypdf.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_docling_hdr_dense_rerank.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_docling_hdr_dense.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_docling_hdr_hybrid_rerank.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_docling_hdr_hybrid.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_docling_hdr.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_docling.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_docling_dense_rerank.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_docling_dense.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_docling_hybrid_rerank.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_docling_hybrid.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_pypdf2000_dense.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_pypdf2000_hybrid_rerank.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_pypdf_dense_rerank.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_pypdf_dense.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_pypdf_hybrid_rerank.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_human_pypdf_hybrid.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_m2000_human_docling_2000_dense.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_docling_2000_hdr_dense.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_docling_2000_hdr_hybrid.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_docling_2000_hdr_union.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_docling_2000_hybrid.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_docling_2000_union.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_pypdf_2000_dense.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_pypdf_2000_hdr_dense.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_pypdf_2000_hdr_hybrid.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_pypdf_2000_hdr_union.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_pypdf_2000_hybrid.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_human_pypdf_2000_union.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_docling_2000_dense.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_docling_2000_hdr_dense.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_docling_2000_hdr_hybrid.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_docling_2000_hdr_union.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_docling_2000_hybrid.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_docling_2000_union.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_pypdf_2000_dense.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_pypdf_2000_hdr_dense.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_pypdf_2000_hdr_hybrid.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_pypdf_2000_hdr_union.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_pypdf_2000_hybrid.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_m2000_synthetic_pypdf_2000_union.log
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
eval_pypdf_hdr_dense_rerank.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_pypdf_hdr_dense.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_pypdf_hdr_hybrid_rerank.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_pypdf_hdr_hybrid.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_pypdf_hdr.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
eval_pypdf.log
Add build and evaluation logs
2026-10-07 16:49:22 +02:00
goldset_kafka_human.csv
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
goldset_kafka.csv
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
goldset.csv
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
judge_cache.json
Add LLM judge to retrieval eval; switch chat to pypdf with 2000/400 chunks
2026-10-08 09:07:51 +02:00
kafka_docling_hdr_dense_rerank.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_docling_hdr_dense.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_docling_hdr_hybrid_rerank.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_docling_hdr_hybrid.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_docling_hdr.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_docling_split.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_docling_zusatztext.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_docling.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_docling_dense_rerank.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_docling_dense.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_docling_hybrid_rerank.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_docling_hybrid.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_judge_docling_2000_hdr.txt
Add LLM judge to retrieval eval; switch chat to pypdf with 2000/400 chunks
2026-10-08 09:07:51 +02:00
kafka_human_judge_pypdf_2000.txt
Add LLM judge to retrieval eval; switch chat to pypdf with 2000/400 chunks
2026-10-08 09:07:51 +02:00
kafka_human_pypdf1500_dense.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_pypdf2000_dense.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_pypdf2000_hybrid_rerank.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_pypdf_dense_rerank.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_pypdf_dense.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_pypdf_hybrid_rerank.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_human_pypdf_hybrid.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_m2000_human_docling_2000_dense.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_docling_2000_hdr_dense.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_docling_2000_hdr_hybrid.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_docling_2000_hdr_union.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_docling_2000_hybrid.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_docling_2000_union.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_pypdf_2000_dense.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_pypdf_2000_hdr_dense.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_pypdf_2000_hdr_hybrid.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_pypdf_2000_hdr_union.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_pypdf_2000_hybrid.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_human_pypdf_2000_union.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_docling_2000_dense.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_docling_2000_hdr_dense.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_docling_2000_hdr_hybrid.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_docling_2000_hdr_union.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_docling_2000_hybrid.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_docling_2000_union.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_pypdf_2000_dense.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_pypdf_2000_hdr_dense.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_pypdf_2000_hdr_hybrid.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_pypdf_2000_hdr_union.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_pypdf_2000_hybrid.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_m2000_synthetic_pypdf_2000_union.txt
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
kafka_pypdf_hdr_dense_rerank.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_pypdf_hdr_dense.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_pypdf_hdr_hybrid_rerank.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_pypdf_hdr_hybrid.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_pypdf_hdr.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_pypdf_minilm.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_pypdf_zusatztext.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
kafka_pypdf.txt
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00
matrix_2000.out
Add evaluation matrix for 2000/400 chunks (loader x headers x search mode)
2026-10-07 18:04:48 +02:00
run_matrix_2000.sh
Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking
2026-10-07 16:48:12 +02:00