# Local RAG – Kafka Chat

Web chat with three conversation partners on Kafka's *Die Verwandlung*, built on LangChain, FAISS and either **Ollama** (local), **OpenAI** (Responses API) or the **OpenAI Realtime API**. The corpus search is a tool the LLM calls when it needs evidence.

A short German quick start is in [EINSTIEG.md](EINSTIEG.md).

---

## Environments

| Conda env | Architecture | Use |
|---|---|---|
| `docling` | arm64 (Apple Silicon), Python 3.12, torch 2.14 | Default. Also needed for Docling (`prepare_corpus.py`). |
| `local_rag` | x86_64 (Intel Mac or Rosetta), Python 3.10, torch 2.2.2 | Alternative for x86. Chat, search and indexing tested; Docling does not run. Search is slower under Rosetta (about 2.3 s instead of about 1 s). |

**bge-m3 on x86**

- **The problem:** macOS x86 is limited to torch 2.2.2. `transformers` refuses to load `.bin` weights below torch 2.6, and the main branch of bge-m3 only ships `pytorch_model.bin`.
- **The fix:** [local_rag.py](local_rag.py) loads the official safetensors conversion (`EMBEDDING_REVISION`) in both environments. The weights are identical and so are the vectors, so the same indexes work everywhere.

**New environment**

[envs/](envs/) holds one file per environment with exact versions, exported from the working environments. The header of each file has the exact commands.

```bash
CONDA_SUBDIR=osx-arm64 conda env create -f envs/environment-docling.yml
```

```bash
CONDA_SUBDIR=osx-64 conda env create -f envs/environment-local_rag.yml
```

`CONDA_SUBDIR` forces the architecture; without it, conda on Apple Silicon may pick the wrong one. [requirements.txt](requirements.txt) lists only the direct dependencies, without versions.

**Ollama**

Ollama must be running and the model pulled once:

```bash
ollama pull gemma4:12b
```

**OpenAI** (ChatGPT and Realtime)

Put the key into a `.env` file in the project folder. It is git-ignored; protect it with `chmod 600`. An exported `OPENAI_API_KEY` also works and takes precedence.

```
OPENAI_API_KEY=sk-...
```

Without a key, both OpenAI entries in the model menu are greyed out.

---

## Run the chat

```bash
cd /Users/Philipp/Desktop/workspace/diverses/local_rag && conda activate docling && uvicorn server:app --port 8010
```

Open http://localhost:8010.

- **Startup takes about a minute.** It loads the embedding model and the reranker and builds the BM25 indexes, so the first question is as fast as any other.
- **`--reload`** restarts the server on code changes. Without it, every change to a `.py` file needs a restart; for `templates/chat.html`, reloading the page is enough.

The header has two menus:
- **Gegenüber:** Forscherin / Zeitgenosse / Autor.
- **Modell:** Ollama / ChatGPT / Realtime Mini. ChatGPT is the default and falls back to Ollama without a key.

---

## How it works

```
Browser (templates/chat.html)
   │  POST /api/chat {message, history, persona, llm_provider}
   ▼
server.py ── picks the persona (personas.py) and the RAG instance (provider × vector store)
   ▼
LocalRAG.query_with_history (local_rag.py)
   │  system prompt + history + question to the LLM, search tool offered
   │  ┌─ LLM calls search_kafka_corpus(query)  (at most MAX_TOOL_ROUNDS = 3 times)
   │  │    retrieve(): FAISS (bge-m3) + BM25 → Reciprocal Rank Fusion → 25 candidates
   │  │                → cross-encoder bge-reranker-v2-m3 → best 5 chunks
   │  └─ LLM answers from the excerpts, citing source and page
   ▼
answer + retrieved chunks + search queries → browser
```

- **Personas:** Each persona has its own system prompt, its own tool description and its own vector store.
  - The researcher searches the whole corpus: the primary text and the secondary literature.
  - The contemporary and Kafka himself search only *Die Verwandlung*.
- **Shared resources:** The embedding model, reranker and BM25 index exist once per process and are shared by all RAG instances, whatever the provider. They are loaded at startup with `LocalRAG.preload()`.
- **Realtime:** [realtime_chat.py](realtime_chat.py) wraps the OpenAI Realtime API (WebSocket, text only) as a LangChain chat model. The tool loop is therefore the same for all providers. It opens one session per LLM call.

### Files

| File | Content |
|---|---|
| [server.py](server.py) | FastAPI app, model names, RAG instances, startup preload, console trace |
| [personas.py](personas.py) | the three personas: UI texts, images, system prompts, tool description, vector store |
| [local_rag.py](local_rag.py) | model loading, retrieval (`retrieve`), tool loop (`query_with_history`), indexing (`add_documents`) |
| [realtime_chat.py](realtime_chat.py) | `ChatOpenAIRealtime`, the Realtime API as a LangChain model |
| [templates/chat.html](templates/chat.html) | UI, persona and model menus |
| `templates/visuals/` | images (`*.png` originals, `*.jpg` web versions, `*_avatar.jpg`) |
| [add_pdfs.py](add_pdfs.py) | build a vector store from a PDF folder |
| [prepare_corpus.py](prepare_corpus.py), [docling_loader.py](docling_loader.py), [split_spreads.py](split_spreads.py) | Docling pipeline: split double-page scans, OCR where needed, structured extraction |
| [eval_retrieval.py](eval_retrieval.py) | retrieval quality on a goldset (Recall@k, MRR, optional LLM judge) |
| [grobid_qa.py](grobid_qa.py) | extraction quality checks (GROBID, Docling, pypdf) |

---

## Configuration reference

### Personas – [personas.py](personas.py)

Each persona is one entry in `PERSONAS`.

| Field | Effect |
|---|---|
| `system_prompt` | role, tone, rules for searching and citing |
| `tool_description` | what the LLM is told about the search tool (default: `SEARCH_TOOL_DESCRIPTION`) |
| `vectorstore` | which index is searched (`VECTORSTORE_PATH` or `PRIMARY_VECTORSTORE_PATH`) |
| `label`, `topic`, `name`, `role`, `place`, `status`, `greeting`, `suggestions` | UI texts |
| `image`, `image_position`, `image_alt`, `avatar` | images |

- The search rules shared by the contemporary and Kafka are in `_TOOL_RULES`.
- The researcher's prompt is `SYSTEM_PROMPT` in `local_rag.py`.
- `PUBLIC_FIELDS` lists what the UI receives; system prompts stay on the server.

### Models – [server.py](server.py) and [local_rag.py](local_rag.py)

| What | Current | Where |
|---|---|---|
| Ollama model | `gemma4:12b` (thinks by default, `reasoning` not set) | `OLLAMA_MODEL` (server.py), `ChatOllama(...)` (local_rag.py) |
| OpenAI model | `gpt-6.1-sol` | `OPENAI_MODEL` (server.py) |
| OpenAI reasoning effort | `low`, via the Responses API | `ChatOpenAI(..., reasoning_effort="low", use_responses_api=True)` (local_rag.py) |
| Realtime model | `gpt-realtime-2.1-mini` | `REALTIME_MODEL` (server.py) |
| Default in the model menu | ChatGPT | `selected` in chat.html |

GPT-6 models accept function tools on Chat Completions only with `reasoning_effort="none"`, and gpt-6.1-sol does not support `none` at all. Tool calls therefore go through the Responses API.

### Retrieval – module constants in [local_rag.py](local_rag.py)

| Constant | Current | Meaning |
|---|---|---|
| `RETRIEVAL_K` | 5 | chunks per search handed to the LLM |
| `RETRIEVAL_MODE` | `hybrid` | `dense` (vectors only), `hybrid` (vectors + BM25, RRF), `union` (BM25 adds candidates only) |
| `CANDIDATES_K` | 25 | candidates before reranking; 25 was faster than 50 and not worse on either goldset |
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | `None` disables reranking; runs in fp16 on the GPU (MPS/CUDA), fp32 on the CPU |
| `MAX_TOOL_ROUNDS` | 3 | searches per message before the LLM must answer |
| `SEARCH_TOOL_NAME` / `SEARCH_TOOL_DESCRIPTION` | `search_kafka_corpus` / text | tool name, default tool description |
| `QUERY_ADDITIONAL_INSTRUCTIONS` | empty | text appended to the search query; it lowered recall, so it is left empty |

### Index – rebuild required after changing these

| Constant | Current |
|---|---|
| `EMBEDDING_MODEL` / `EMBEDDING_REVISION` | `BAAI/bge-m3`, safetensors revision (normalized vectors) |
| `CHUNK_SIZE` / `CHUNK_OVERLAP` | 2000 / 400 characters |
| `PDF_LOADER` | `pypdf`; `docling` was not better in the goldset comparison |
| `CHUNK_HEADERS` | `False` |

Each index stores its settings in `index_info.json`. `LocalRAG` refuses to load an index built with a different embedding model.

---

## Build the vector stores

[add_pdfs.py](add_pdfs.py) indexes every PDF under a folder, including subfolders. With `CLEAR_VECTORSTORE_FIRST = True` (the default in the script), the target folder is deleted and rebuilt; set it to `False` to merge into an existing index.

Whole corpus, used by the researcher:

```bash
python add_pdfs.py --data data --index ./vectorstore_pypdf_2000 --loader pypdf --no-headers
```

Primary text only, used by the contemporary and Kafka. `data_primary/` contains just a link to `data/Kafka_Die_Verwandlung_.pdf`:

```bash
python add_pdfs.py --data data_primary --index ./vectorstore_primary_pypdf_2000 --loader pypdf --no-headers
```

Other options: `--loader docling` (needs `prepare_corpus.py` output, `--processed`), `--chunk-size`, `--chunk-overlap`.

- **Docling pipeline (arm64 env only):** `python prepare_corpus.py [pdf_dir]` writes to `corpus_processed/data_split`. Options: `--only "name1;name2"`, `--no-split`, `--layout`, `--no-headings`.
- **Not in git:** the corpus PDFs (`data/`, `data_primary/`), vector stores (`vectorstore*/`), extraction output and `.env`.

---

## Measure retrieval quality

```bash
python eval_retrieval.py --goldset eval/goldset_kafka_human.csv --index ./vectorstore_pypdf_2000 --report eval/my_report.txt
```

- Options: `--mode dense|hybrid|union`, `--no-rerank`, `--judge MODEL`.
- `--judge` lets an OpenAI model check whether the answer is in the top 5. It needs the key; verdicts are cached in `eval/judge_cache.json`.
- Current numbers (whole corpus, hybrid, 25 candidates, fp16 reranker):

| Goldset | R@5 | MRR |
|---|---|---|
| human (30 questions) | 0.73 | 0.68 |
| synthetic (43 questions) | 0.98 | 0.79 |

---

## API

| Endpoint | Purpose |
|---|---|
| `POST /api/chat` | Body: `message`, `history` (list of `{role, content}`), `persona` (`researcher`, `contemporary`, `kafka`), `llm_provider` (`ollama`, `openai`, `realtime`; API default `ollama`) |
| `GET /api/personas` | default persona and the public fields of all personas |
| `GET /api/providers` | which providers are available (`openai` and `realtime` need a key at startup) |
| `GET /api/health` | server status and whether all vector stores are loaded |

**Response of `POST /api/chat`**

| Field | Meaning |
|---|---|
| `answer` | the answer (empty on error) |
| `error` | error message or `null` |
| `retrieved` | chunks the LLM saw: `content`, `source`, `page`, `score` (reranker relevance, higher is better) |
| `searches` | the queries the LLM sent to the search tool (empty if it did not search) |

---

## Debugging and timing

The server terminal prints, per question:

```
[RAG] User question: '…'
[Zeit] LLM-Aufruf 1: 6.86s (Suche angefordert)
[RAG] Tool search '…': 5 chunk(s)
[Zeit] Suche 1.37s (davon Reranker 1.21s)
[Zeit] LLM-Aufruf 2: 15.31s (Antwort)
[Zeit] Gesamt 23.54s
```

After that it prints the retrieved chunks (source, page, score, preview) and the full answer, with provider, model and persona.

In the browser console, each answer logs a collapsed group `[RAG] Retrieved N chunk(s)` with the full chunk texts.

Where to look when an answer is wrong:
- **Wrong chunks:** tune retrieval, i.e. `CANDIDATES_K`, `RETRIEVAL_MODE` or the reranker, and measure with `eval_retrieval.py`.
- **Right chunks but a bad answer, or searching when it should not (or the reverse):** edit the persona's `system_prompt` or `tool_description`.

To try retrieval without the server:

```python
from local_rag import LocalRAG
rag = LocalRAG(vectorstore_path="./vectorstore_primary_pypdf_2000")
for doc, score in rag.retrieve("Wo versteckt sich Gregor?", k=5):
    print(doc.metadata.get("page"), round(score, 3), doc.page_content[:80])
```

`python local_rag.py` still runs an old demo against `./vectorstore` and is not maintained.