Add Docling corpus pipeline, retrieval evaluation and hybrid search with reranking

- prepare_corpus.py: split double-page scans (split_spreads.py), re-OCR with
  OCRmyPDF/Tesseract only where the existing text layer is worse, Docling
  layout extraction with heading hierarchy, QA report per PDF
- docling_loader.py: cleanup (margin noise, split paragraphs, footnote marks,
  OCR comma/period repair), page mapping back to the original PDF, chunk
  headers with author, title and section
- local_rag.py: bge-m3 embeddings, configurable chunking and loader, index
  settings check (index_info.json), hybrid/union BM25 search with Reciprocal
  Rank Fusion and bge-reranker-v2-m3 cross-encoder; query instruction text
  removed after it lowered recall
- eval_retrieval.py: CLI options, same retrieval path as the chat, tolerant
  file name matching; goldsets and evaluation reports for the Kafka corpus
- grobid_test.py, grobid_qa.py, extract_docling.py, compare_extractors.py:
  extractor tests and quality checks
- .gitignore: corpus PDFs, vector stores and extractor output stay local
- server.py: uses ./vectorstore_docling_hdr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -146,6 +146,46 @@ Open [http://127.0.0.1:8000](http://127.0.0.1:8000) or [http://localhost:8000](h
 | `GET /api/providers` | Which providers are available (`openai` false if no API key at startup) |
 | `GET /api/health` | Server and whether a vector store is loaded |
 
+**`POST /api/chat` response fields:**
+
+| Field | Type | Meaning |
+|-------|------|---------|
+| `answer` | string | The generated answer (empty string on error) |
+| `error` | string or `null` | Error message, e.g. empty question or OpenAI not configured |
+| `retrieved` | list or `null` | The chunks used as context for this answer |
+
+Each entry in `retrieved` has:
+
+| Field | Type | Meaning |
+|-------|------|---------|
+| `content` | string | The chunk text handed to the LLM |
+| `source` | string | Path of the file the chunk came from |
+| `page` | int or `null` | Page number (PDFs only) |
+| `score` | float or `null` | FAISS **L2 distance** – **lower = more similar** (it is a distance, not a similarity percentage) |
+
+---
+
+## Inspect what was retrieved (debugging)
+
+Every answer reports which chunks it was based on, in two places:
+
+- **Browser:** open the developer console (Chrome/Edge: `F12` or `Cmd`+`Option`+`I`, Firefox: `F12`) before sending a question. Each answer logs a collapsed group `[RAG] Retrieved N chunk(s)` with the full text, source file and page of every chunk.
+- **Server terminal:** the terminal running `uvicorn` prints the rephrased standalone query, the retrieval query, the L2 distances, a 150-character preview of each chunk, and the raw LLM response.
+
+Use this when an answer looks wrong: if the right chunks were never retrieved, tune `RETRIEVAL_K`, `QUERY_ADDITIONAL_INSTRUCTIONS` or the chunk size; if the chunks were correct but the answer was not, tune `ANSWER_PROMPT` (see [Configuration reference](#configuration-reference)).
+
+---
+
+## Quick check from the command line
+
+[local_rag.py](local_rag.py) can be run directly for a smoke test without the web server:
+
+```bash
+python local_rag.py
+```
+
+It loads the existing vector store, lists the documents in it, and runs **one hardcoded example question** through Ollama. This is only a demo: edit the `question` variable in `main()` to ask something else, and note that it always uses Ollama (the provider is not configurable from the command line). For normal use, run the web app instead.
+
 ---
 
 ## How it works (high level)