pmock/localRAG
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Load bge-m3 as safetensors for x86; update README, add EINSTIEG.md

Browse Source 
- EMBEDDING_REVISION pins the official safetensors conversion of bge-m3, so the x86
  env (torch 2.2.2) can load it; identical vectors, no index rebuild
- README rewritten for the current state (personas, providers, retrieval, environments)
- EINSTIEG.md: short German guide to running and configuring the chat

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Philipp Mock
2026-10-26 07:26:29 +01:00
co-authored by Claude Opus 5.5
parent e4b7f0ce34
commit ce9a863a87
3 changed files with 381 additions and 209 deletions
Download Patch File Download Diff File Expand all files Collapse all files
EINSTIEG.md
+156
View File
@@ -0,0 +1,156 @@
# Kafka-Chat: Einstieg
Stand: Oktober 2026. Ausführliche englische Referenz: [README.md](README.md).
## Starten
**Conda-Umgebungen**
| Umgebung | Architektur | Einsatz |
|---|---|---|
| `docling` | arm64 (Apple Silicon), Python 3.12, torch 2.14 | Standard. Auch für Docling (`prepare_corpus.py`). |
| `local_rag` | x86_64 (Intel-Mac oder Rosetta), Python 3.10, torch 2.2.2 | Alternative für x86. Chat und Suche getestet; Indexbau sollte gehen (gleiches Embedding), ungetestet. Docling nicht. |
- **Warum bge-m3 auf x86 nur über safetensors lädt:** Für macOS x86 gibt es höchstens torch 2.2.2. `transformers` lädt `.bin`-Gewichte aber erst ab torch 2.6, und bge-m3 hat im Hauptzweig nur `pytorch_model.bin`.
- **Die Lösung im Code:** `local_rag.py` lädt deshalb die offizielle safetensors-Konvertierung (`EMBEDDING_REVISION`). Die Gewichte sind dieselben, die Vektoren in beiden Umgebungen identisch, und dieselben Indizes funktionieren in beiden.
- **Geschwindigkeit unter Rosetta:** Spürbar langsamer. Im Test brauchte die Suche gut 5 statt gut 1 Sekunde, der Serverstart war dagegen nicht langsamer.
**Voraussetzungen (einmalig)**
- Ollama mit dem Modell: `ollama pull gemma4:12b`.
- Für ChatGPT und Realtime eine Datei `.env` im Projektordner mit `OPENAI_API_KEY=sk-...`. Sie ist git-ignoriert, am besten mit `chmod 600` schützen. Ohne Key sind beide Einträge im Menü ausgegraut.
- Die beiden Vektorspeicher `vectorstore_pypdf_2000/` und `vectorstore_primary_pypdf_2000/`. Sie sind nicht im Git; Neubau siehe unten.
**Server starten**
```bash
cd /Users/Philipp/Desktop/workspace/diverses/local_rag
conda activate docling # auf x86: conda activate local_rag
uvicorn server:app --port 8010
```
Dann im Browser http://localhost:8010 öffnen. Der Start dauert etwa eine Minute: Embedding-Modell, Reranker und Stichwortindex werden geladen. Danach ist auch die erste Frage schnell.
- Optionen: `--reload` startet den Server bei Codeänderungen neu, `--log-level debug` zeigt mehr Details.
- Ohne `--reload` braucht jede Änderung an `.py`-Dateien einen Neustart. Bei `templates/chat.html` reicht es, die Seite neu zu laden.
**Was im Terminal steht, pro Frage**
```
[RAG] User question: '…'
[Zeit] LLM-Aufruf 1: 6.86s (Suche angefordert)
[RAG] Tool search '…': 5 chunk(s)
[Zeit] Suche 1.37s (davon Reranker 1.21s)
[Zeit] LLM-Aufruf 2: 15.31s (Antwort)
[Zeit] Gesamt 23.54s
```
Danach folgen die gefundenen Chunks mit Quelle, Seite und Score sowie die vollständige Antwort.
## Aufbau
```
Browser (templates/chat.html)
│ POST /api/chat {message, history, persona, llm_provider}
▼
server.py ── wählt Figur (personas.py) und RAG-Instanz (Anbieter × Vektorspeicher)
▼
local_rag.py · LocalRAG.query_with_history
│ Systemprompt + Verlauf + Frage an das LLM, Such-Werkzeug angeboten
│ ┌─ LLM ruft search_kafka_corpus(query) auf (höchstens 3 Runden)
│ │ retrieve(): FAISS (bge-m3) + BM25 → Fusion (RRF) → 25 Kandidaten
│ │ → Reranker (bge-reranker-v2-m3) → beste 5 Chunks
│ └─ LLM antwortet mit den Auszügen
▼
Antwort + gefundene Chunks + Suchanfragen → Browser
```
**Dateien**
| Datei | Inhalt |
|---|---|
| [server.py](server.py) | FastAPI-Server, Modellwahl, eine RAG-Instanz pro Anbieter und Vektorspeicher, Konsolenausgabe |
| [personas.py](personas.py) | die drei Figuren: Texte, Bilder, Systemprompts, Werkzeugbeschreibung, Vektorspeicher |
| [local_rag.py](local_rag.py) | Laden der Modelle, Suche (`retrieve`), Werkzeugschleife (`query_with_history`), Indexbau |
| [realtime_chat.py](realtime_chat.py) | Anbindung der OpenAI Realtime API (WebSocket, nur Text) als LangChain-Modell |
| [templates/chat.html](templates/chat.html) | Oberfläche, Menüs für Figur und Modell |
| `templates/visuals/` | Bilder (`*.png` Vorlagen, `*.jpg` Webversionen, `*_avatar.jpg`) |
| [add_pdfs.py](add_pdfs.py) | Vektorspeicher aus einem PDF-Ordner bauen |
| [eval_retrieval.py](eval_retrieval.py) | Suchqualität auf einem Goldset messen (Recall@k, MRR, optional LLM-Judge) |
## Stellschrauben
### Verhalten der Figuren: [personas.py](personas.py)
Pro Figur ein Eintrag in `PERSONAS`; eine neue Figur ist ein neuer Eintrag mit eigenem Bild.
| Feld | wirkt auf |
|---|---|
| `system_prompt` | Ton, Rolle, Regeln für Suche und Belege |
| `tool_description` | was das Modell über das Such-Werkzeug erfährt (ohne Feld gilt `SEARCH_TOOL_DESCRIPTION`) |
| `vectorstore` | welcher Index durchsucht wird |
| `label`, `topic`, `name`, `role`, `place`, `status`, `greeting`, `suggestions` | Texte in der Oberfläche |
| `image`, `image_position`, `image_alt`, `avatar` | Bilder |
- Die Regeln zur Suche für Zeitgenosse und Autor stehen gemeinsam in `_TOOL_RULES`.
- Der Prompt der Forscherin ist `SYSTEM_PROMPT` in `local_rag.py`.
### Sprachmodelle: [server.py](server.py) und [local_rag.py](local_rag.py)
| Einstellung | aktuell | wo |
|---|---|---|
| lokales Modell (Ollama) | `gemma4:12b`, denkt standardmäßig (`reasoning` nicht gesetzt) | `OLLAMA_MODEL` in server.py, `ChatOllama(...)` in local_rag.py |
| ChatGPT | `gpt-6.1-sol` | `OPENAI_MODEL` in server.py |
| Denkaufwand ChatGPT | `low`, über die Responses API | `ChatOpenAI(...)` in local_rag.py |
| Realtime | `gpt-realtime-2.1-mini` | `REALTIME_MODEL` in server.py |
| Voreinstellung im Menü | ChatGPT | `selected` in chat.html |
- **gpt-6.1-sol:** kennt keinen Denkaufwand `none`. Werkzeuge funktionieren nur über die Responses API, deshalb steht `use_responses_api=True`.
- **Realtime:** öffnet pro Modellaufruf eine eigene WebSocket-Sitzung.
### Suche: Konstanten oben in [local_rag.py](local_rag.py)
| Konstante | aktuell | Bedeutung |
|---|---|---|
| `RETRIEVAL_K` | 5 | Chunks pro Suche an das LLM |
| `RETRIEVAL_MODE` | `hybrid` | `dense` (nur Vektoren), `hybrid` (Vektoren + BM25, RRF), `union` |
| `CANDIDATES_K` | 25 | Kandidaten vor dem Reranker. 25 war schneller und nicht schlechter als 50. |
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | `None` schaltet das Reranking ab. Läuft auf der GPU in fp16. |
| `MAX_TOOL_ROUNDS` | 3 | Suchen pro Nachricht, danach muss das Modell antworten |
| `SEARCH_TOOL_DESCRIPTION` | Text | Werkzeugbeschreibung für die Forscherin |
Die Suche dauert etwa 1 bis 1,5 Sekunden pro Aufruf, fast nur der Reranker. Den größten Teil der Antwortzeit brauchen die Modellaufrufe.
### Index: Neubau nötig bei Änderung
| Konstante | aktuell |
|---|---|
| `EMBEDDING_MODEL` | `BAAI/bge-m3` (normalisierte Vektoren) |
| `CHUNK_SIZE` / `CHUNK_OVERLAP` | 2000 / 400 Zeichen |
| `PDF_LOADER` | `pypdf`; `docling` war im Goldset-Vergleich nicht besser |
| `CHUNK_HEADERS` | `False` |
Neubau der beiden Indizes. `data_primary/` enthält nur einen Verweis auf `data/Kafka_Die_Verwandlung_.pdf`.
```bash
python add_pdfs.py --data data --index ./vectorstore_pypdf_2000 --loader pypdf --no-headers
```
```bash
python add_pdfs.py --data data_primary --index ./vectorstore_primary_pypdf_2000 --loader pypdf --no-headers
```
Jeder Index speichert seine Einstellungen in `index_info.json`. Passt das Embedding-Modell nicht zum Code, verweigert `LocalRAG` das Laden.
### Suchqualität messen
```bash
python eval_retrieval.py --goldset eval/goldset_kafka_human.csv --report eval/mein_bericht.txt
```
- `--judge gpt-6-luna` lässt zusätzlich ein LLM prüfen, ob die Antwort in den Top 5 steht; das braucht den Key.
- Stand auf dem menschlichen Goldset (30 Fragen): R@5 0,73, MRR 0,68.
## Gut zu wissen
- **Speicher:** Embedding-Modell, Reranker und BM25-Index gibt es pro Prozess nur einmal, auch bei mehreren Anbietern. Ollama braucht zusätzlich etwa 8 bis 10 GB für gemma4:12b.
- **Nicht im Git:** Korpus-PDFs (`data/`), Vektorspeicher, `.env` und Schlüsseldateien.
README.md
+220 -209
View File
@@ -1,245 +1,256 @@
# Local RAG Setup # Local RAG – Kafka Chat
Minimal RAG implementation with LangChain, FAISS, and either **Ollama** (local) or **OpenAI** (API key). A web chat UI is included. Web chat with three conversation partners on Kafka's *Die Verwandlung*, built on LangChain, FAISS and either **Ollama** (local), **OpenAI** (Responses API) or the **OpenAI Realtime API**. The corpus search is a tool the LLM calls when it needs evidence.
A short German quick start is in [EINSTIEG.md](EINSTIEG.md).
--- ---
## What you need (before you start) ## Environments
- **Python 3.10 or newer** ([python.org](https://www.python.org/downloads/)) | Conda env | Architecture | Use |
- **Git** (optional, only if you clone the project) |---|---|---|
- Either: | `docling` | arm64 (Apple Silicon), Python 3.12, torch 2.14 | Default. Also needed for Docling (`prepare_corpus.py`). |
- **Ollama** installed and running ([ollama.com](https://ollama.com)), with at least one model pulled, **or** | `local_rag` | x86_64 (Intel Mac or Rosetta), Python 3.10, torch 2.2.2 | Alternative for x86. Chat and search work; Docling does not. Search is slower under Rosetta (about 5 s instead of about 1 s). |
- An **OpenAI API key** (if you use OpenAI in the chat)
--- **bge-m3 on x86**
## Install dependencies (step by step) - **The problem:** macOS x86 is limited to torch 2.2.2. `transformers` refuses to load `.bin` weights below torch 2.6, and the main branch of bge-m3 only ships `pytorch_model.bin`.
- **The fix:** [local_rag.py](local_rag.py) loads the official safetensors conversion (`EMBEDDING_REVISION`) in both environments. The weights are identical and so are the vectors, so the same indexes work everywhere.
Install [Miniconda](https://docs.conda.io/en/latest/miniconda.html) or [Anaconda](https://www.anaconda.com/) if you do not have Conda yet. **New environment**
All commands below assume your terminal is open **in the project folder** (the folder that contains `requirements.txt`).
```bash ```bash
conda create -n local_rag python=3.10 -y conda create -n local_rag python=3.10 -y
conda activate local_rag
pip install --upgrade pip
pip install -r requirements.txt
``` ```
Use `conda activate local_rag` in every new terminal session before running `python` or `uvicorn` for this project.
### OpenAI (only if you use the OpenAI provider in the chat)
In the same terminal **before** starting the server:
```bash ```bash
export OPENAI_API_KEY="your-key-here" conda activate local_rag && pip install -r requirements.txt
``` ```
On Windows (Command Prompt): `set OPENAI_API_KEY=your-key-here` For the arm64 env on Apple Silicon, create it with `CONDA_SUBDIR=osx-arm64` so that conda does not fall back to x86 packages.
--- **Ollama**
## Run Ollama (only if you use Ollama) Ollama must be running and the model pulled once:
In a **separate** terminal:
```bash
ollama serve
```
In another terminal, pull a model once (example):
```bash ```bash
ollama pull gemma4:12b ollama pull gemma4:12b
``` ```
The model name must match what you configure in `server.py` (see [Configuration reference](#configuration-reference)). **OpenAI** (ChatGPT and Realtime)
--- Put the key into a `.env` file in the project folder. It is git-ignored; protect it with `chmod 600`. An exported `OPENAI_API_KEY` also works and takes precedence.
## Build the vector store from a folder of PDFs ```
OPENAI_API_KEY=sk-...
The project includes [add_pdfs.py](add_pdfs.py). It finds every **`.pdf`** file under a folder you choose (including subfolders), then chunks, embeds, and saves to FAISS.
**Two modes** (set in the script):
| Setting | Behavior |
|---------|----------|
| `CLEAR_VECTORSTORE_FIRST = True` | Deletes the existing vector store folder, then builds a **new** index from the PDFs under `DATA_ROOT`. Use this for a full rebuild. |
| `CLEAR_VECTORSTORE_FIRST = False` | Keeps the current index (if it exists) and **merges** chunks from the PDFs under `DATA_ROOT` into it. Use this to add another batch of PDFs without wiping what you already indexed. |
**Steps:**
1. Open [add_pdfs.py](add_pdfs.py) in a text editor.
2. Set **`DATA_ROOT`** to the folder that contains your PDFs (absolute path or path relative to how you run the script).
3. Set **`CLEAR_VECTORSTORE_FIRST`** to `True` (fresh index) or `False` (append to existing store).
4. Optionally set **`VECTORSTORE_PATH`** (default: `./vectorstore`). It must match **`VECTORSTORE_PATH`** in [server.py](server.py) so the chat loads the same index.
5. From the project folder, with `conda activate local_rag` (or your chosen env name):
```bash
python add_pdfs.py
``` ```
Indexing can take a long time for many large PDFs. When it finishes, you should see `Vector store saved to ...`. Without a key, both OpenAI entries in the model menu are greyed out.
**Note:** This script only indexes **PDF** files. To add `.txt` or `.md` files, use the Python snippet below or call `add_documents` yourself.
--- ---
## Add more documents later (alternative to add_pdfs) ## Run the chat
You can also merge files by hand with a short script (any mix of supported types): ```bash
cd /Users/Philipp/Desktop/workspace/diverses/local_rag && conda activate docling && uvicorn server:app --port 8010
```
Open http://localhost:8010.
- **Startup takes about a minute.** It loads the embedding model and the reranker and builds the BM25 indexes, so the first question is as fast as any other.
- **`--reload`** restarts the server on code changes. Without it, every change to a `.py` file needs a restart; for `templates/chat.html`, reloading the page is enough.
The header has two menus:
- **Gegenüber:** Forscherin / Zeitgenosse / Autor.
- **Modell:** Ollama / ChatGPT / Realtime Mini. ChatGPT is the default and falls back to Ollama without a key.
---
## How it works
```
Browser (templates/chat.html)
│ POST /api/chat {message, history, persona, llm_provider}
▼
server.py ── picks the persona (personas.py) and the RAG instance (provider × vector store)
▼
LocalRAG.query_with_history (local_rag.py)
│ system prompt + history + question to the LLM, search tool offered
│ ┌─ LLM calls search_kafka_corpus(query) (at most MAX_TOOL_ROUNDS = 3 times)
│ │ retrieve(): FAISS (bge-m3) + BM25 → Reciprocal Rank Fusion → 25 candidates
│ │ → cross-encoder bge-reranker-v2-m3 → best 5 chunks
│ └─ LLM answers from the excerpts, citing source and page
▼
answer + retrieved chunks + search queries → browser
```
- **Personas:** Each persona has its own system prompt, its own tool description and its own vector store.
- The researcher searches the whole corpus: the primary text and the secondary literature.
- The contemporary and Kafka himself search only *Die Verwandlung*.
- **Shared resources:** The embedding model, reranker and BM25 index exist once per process and are shared by all RAG instances, whatever the provider. They are loaded at startup with `LocalRAG.preload()`.
- **Realtime:** [realtime_chat.py](realtime_chat.py) wraps the OpenAI Realtime API (WebSocket, text only) as a LangChain chat model. The tool loop is therefore the same for all providers. It opens one session per LLM call.
### Files
| File | Content |
|---|---|
| [server.py](server.py) | FastAPI app, model names, RAG instances, startup preload, console trace |
| [personas.py](personas.py) | the three personas: UI texts, images, system prompts, tool description, vector store |
| [local_rag.py](local_rag.py) | model loading, retrieval (`retrieve`), tool loop (`query_with_history`), indexing (`add_documents`) |
| [realtime_chat.py](realtime_chat.py) | `ChatOpenAIRealtime`, the Realtime API as a LangChain model |
| [templates/chat.html](templates/chat.html) | UI, persona and model menus |
| `templates/visuals/` | images (`*.png` originals, `*.jpg` web versions, `*_avatar.jpg`) |
| [add_pdfs.py](add_pdfs.py) | build a vector store from a PDF folder |
| [prepare_corpus.py](prepare_corpus.py), [docling_loader.py](docling_loader.py), [split_spreads.py](split_spreads.py) | Docling pipeline: split double-page scans, OCR where needed, structured extraction |
| [eval_retrieval.py](eval_retrieval.py) | retrieval quality on a goldset (Recall@k, MRR, optional LLM judge) |
| [grobid_qa.py](grobid_qa.py) | extraction quality checks (GROBID, Docling, pypdf) |
---
## Configuration reference
### Personas – [personas.py](personas.py)
Each persona is one entry in `PERSONAS`.
| Field | Effect |
|---|---|
| `system_prompt` | role, tone, rules for searching and citing |
| `tool_description` | what the LLM is told about the search tool (default: `SEARCH_TOOL_DESCRIPTION`) |
| `vectorstore` | which index is searched (`VECTORSTORE_PATH` or `PRIMARY_VECTORSTORE_PATH`) |
| `label`, `topic`, `name`, `role`, `place`, `status`, `greeting`, `suggestions` | UI texts |
| `image`, `image_position`, `image_alt`, `avatar` | images |
- The search rules shared by the contemporary and Kafka are in `_TOOL_RULES`.
- The researcher's prompt is `SYSTEM_PROMPT` in `local_rag.py`.
- `PUBLIC_FIELDS` lists what the UI receives; system prompts stay on the server.
### Models – [server.py](server.py) and [local_rag.py](local_rag.py)
| What | Current | Where |
|---|---|---|
| Ollama model | `gemma4:12b` (thinks by default, `reasoning` not set) | `OLLAMA_MODEL` (server.py), `ChatOllama(...)` (local_rag.py) |
| OpenAI model | `gpt-6.1-sol` | `OPENAI_MODEL` (server.py) |
| OpenAI reasoning effort | `low`, via the Responses API | `ChatOpenAI(..., reasoning_effort="low", use_responses_api=True)` (local_rag.py) |
| Realtime model | `gpt-realtime-2.1-mini` | `REALTIME_MODEL` (server.py) |
| Default in the model menu | ChatGPT | `selected` in chat.html |
GPT-6 models accept function tools on Chat Completions only with `reasoning_effort="none"`, and gpt-6.1-sol does not support `none` at all. Tool calls therefore go through the Responses API.
### Retrieval – module constants in [local_rag.py](local_rag.py)
| Constant | Current | Meaning |
|---|---|---|
| `RETRIEVAL_K` | 5 | chunks per search handed to the LLM |
| `RETRIEVAL_MODE` | `hybrid` | `dense` (vectors only), `hybrid` (vectors + BM25, RRF), `union` (BM25 adds candidates only) |
| `CANDIDATES_K` | 25 | candidates before reranking; 25 was faster than 50 and not worse on either goldset |
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | `None` disables reranking; runs in fp16 on the GPU (MPS/CUDA), fp32 on the CPU |
| `MAX_TOOL_ROUNDS` | 3 | searches per message before the LLM must answer |
| `SEARCH_TOOL_NAME` / `SEARCH_TOOL_DESCRIPTION` | `search_kafka_corpus` / text | tool name, default tool description |
| `QUERY_ADDITIONAL_INSTRUCTIONS` | empty | text appended to the search query; it lowered recall, so it is left empty |
### Index – rebuild required after changing these
| Constant | Current |
|---|---|
| `EMBEDDING_MODEL` / `EMBEDDING_REVISION` | `BAAI/bge-m3`, safetensors revision (normalized vectors) |
| `CHUNK_SIZE` / `CHUNK_OVERLAP` | 2000 / 400 characters |
| `PDF_LOADER` | `pypdf`; `docling` was not better in the goldset comparison |
| `CHUNK_HEADERS` | `False` |
Each index stores its settings in `index_info.json`. `LocalRAG` refuses to load an index built with a different embedding model.
---
## Build the vector stores
[add_pdfs.py](add_pdfs.py) indexes every PDF under a folder, including subfolders. With `CLEAR_VECTORSTORE_FIRST = True` (the default in the script), the target folder is deleted and rebuilt; set it to `False` to merge into an existing index.
Whole corpus, used by the researcher:
```bash
python add_pdfs.py --data data --index ./vectorstore_pypdf_2000 --loader pypdf --no-headers
```
Primary text only, used by the contemporary and Kafka. `data_primary/` contains just a link to `data/Kafka_Die_Verwandlung_.pdf`:
```bash
python add_pdfs.py --data data_primary --index ./vectorstore_primary_pypdf_2000 --loader pypdf --no-headers
```
Other options: `--loader docling` (needs `prepare_corpus.py` output, `--processed`), `--chunk-size`, `--chunk-overlap`.
- **Docling pipeline (arm64 env only):** `python prepare_corpus.py [pdf_dir]` writes to `corpus_processed/data_split`. Options: `--only "name1;name2"`, `--no-split`, `--layout`, `--no-headings`.
- **Not in git:** the corpus PDFs (`data/`, `data_primary/`), vector stores (`vectorstore*/`), extraction output and `.env`.
---
## Measure retrieval quality
```bash
python eval_retrieval.py --goldset eval/goldset_kafka_human.csv --index ./vectorstore_pypdf_2000 --report eval/my_report.txt
```
- Options: `--mode dense|hybrid|union`, `--no-rerank`, `--judge MODEL`.
- `--judge` lets an OpenAI model check whether the answer is in the top 5. It needs the key; verdicts are cached in `eval/judge_cache.json`.
- Current numbers (whole corpus, hybrid, 25 candidates, fp16 reranker):
| Goldset | R@5 | MRR |
|---|---|---|
| human (30 questions) | 0.73 | 0.68 |
| synthetic (43 questions) | 0.98 | 0.79 |
---
## API
| Endpoint | Purpose |
|---|---|
| `POST /api/chat` | Body: `message`, `history` (list of `{role, content}`), `persona` (`researcher`, `contemporary`, `kafka`), `llm_provider` (`ollama`, `openai`, `realtime`; API default `ollama`) |
| `GET /api/personas` | default persona and the public fields of all personas |
| `GET /api/providers` | which providers are available (`openai` and `realtime` need a key at startup) |
| `GET /api/health` | server status and whether all vector stores are loaded |
**Response of `POST /api/chat`**
| Field | Meaning |
|---|---|
| `answer` | the answer (empty on error) |
| `error` | error message or `null` |
| `retrieved` | chunks the LLM saw: `content`, `source`, `page`, `score` (reranker relevance, higher is better) |
| `searches` | the queries the LLM sent to the search tool (empty if it did not search) |
---
## Debugging and timing
The server terminal prints, per question:
```
[RAG] User question: '…'
[Zeit] LLM-Aufruf 1: 6.86s (Suche angefordert)
[RAG] Tool search '…': 5 chunk(s)
[Zeit] Suche 1.37s (davon Reranker 1.21s)
[Zeit] LLM-Aufruf 2: 15.31s (Antwort)
[Zeit] Gesamt 23.54s
```
After that it prints the retrieved chunks (source, page, score, preview) and the full answer, with provider, model and persona.
In the browser console, each answer logs a collapsed group `[RAG] Retrieved N chunk(s)` with the full chunk texts.
Where to look when an answer is wrong:
- **Wrong chunks:** tune retrieval, i.e. `CANDIDATES_K`, `RETRIEVAL_MODE` or the reranker, and measure with `eval_retrieval.py`.
- **Right chunks but a bad answer, or searching when it should not (or the reverse):** edit the persona's `system_prompt` or `tool_description`.
To try retrieval without the server:
```python ```python
from local_rag import LocalRAG from local_rag import LocalRAG
rag = LocalRAG(vectorstore_path="./vectorstore_primary_pypdf_2000")
rag = LocalRAG(vectorstore_path="./vectorstore") # same path as server.py for doc, score in rag.retrieve("Wo versteckt sich Gregor?", k=5):
rag.add_documents([ print(doc.metadata.get("page"), round(score, 3), doc.page_content[:80])
"path/to/new1.pdf",
"path/to/notes.txt",
])
``` ```
`add_documents` merges new chunks into the existing FAISS store and saves it again—the same behavior as [add_pdfs.py](add_pdfs.py) with `CLEAR_VECTORSTORE_FIRST = False`. `python local_rag.py` still runs an old demo against `./vectorstore` and is not maintained.
---
## Swap or experiment with different vector stores
The vector index is stored on disk under the folder given by **`VECTORSTORE_PATH`** (default `./vectorstore`). That folder contains files such as `index.faiss` and `index.pkl`.
**To use a different index:**
1. Set **`VECTORSTORE_PATH`** in both [server.py](server.py) and any script you use to build the index (e.g. [add_pdfs.py](add_pdfs.py)) to the **same** path, e.g. `./vectorstore_experiment`.
2. Rebuild the index (run `add_pdfs.py` or `add_documents`) so that folder is created.
3. **Restart** the web server so it loads the new path at startup.
**Tips:**
- Keep multiple copies of the folder (e.g. `vectorstore_backup`, `vectorstore_papers_only`) and swap `VECTORSTORE_PATH` to switch between them.
- If you change **chunk size**, **embedding model**, or **FAISS** usage in code, treat the old index as incompatible: use a new `VECTORSTORE_PATH` or delete the old folder and rebuild.
---
## Run the chat web app
With the Conda environment activated (`conda activate local_rag`) and (if needed) `OPENAI_API_KEY` set:
```bash
uvicorn server:app --reload
```
Open [http://127.0.0.1:8000](http://127.0.0.1:8000) or [http://localhost:8000](http://localhost:8000).
- Use the **LLM provider** dropdown: **Ollama** or **OpenAI** (OpenAI only works if the server was started with a valid `OPENAI_API_KEY`).
- You need a **non-empty vector store** (see above) for answers to work.
---
## API (short reference)
| Endpoint | Purpose |
|----------|---------|
| `POST /api/chat` | Body: `message`, optional `history`, optional `llm_provider` (`ollama` or `openai`) |
| `GET /api/providers` | Which providers are available (`openai` false if no API key at startup) |
| `GET /api/health` | Server and whether a vector store is loaded |
**`POST /api/chat` response fields:**
| Field | Type | Meaning |
|-------|------|---------|
| `answer` | string | The generated answer (empty string on error) |
| `error` | string or `null` | Error message, e.g. empty question or OpenAI not configured |
| `retrieved` | list or `null` | The chunks used as context for this answer |
Each entry in `retrieved` has:
| Field | Type | Meaning |
|-------|------|---------|
| `content` | string | The chunk text handed to the LLM |
| `source` | string | Path of the file the chunk came from |
| `page` | int or `null` | Page number (PDFs only) |
| `score` | float or `null` | FAISS **L2 distance** – **lower = more similar** (it is a distance, not a similarity percentage) |
---
## Inspect what was retrieved (debugging)
Every answer reports which chunks it was based on, in two places:
- **Browser:** open the developer console (Chrome/Edge: `F12` or `Cmd`+`Option`+`I`, Firefox: `F12`) before sending a question. Each answer logs a collapsed group `[RAG] Retrieved N chunk(s)` with the full text, source file and page of every chunk.
- **Server terminal:** the terminal running `uvicorn` prints the search queries the LLM sent to the corpus search tool (or that it did not search), the scores, a 150-character preview of each chunk, and the raw LLM response.
Use this when an answer looks wrong: if the right chunks were never retrieved, tune `RETRIEVAL_K`, `QUERY_ADDITIONAL_INSTRUCTIONS` or the chunk size; if the chunks were correct but the answer was not, or the LLM searched when it should not (or vice versa), tune `SYSTEM_PROMPT` (see [Configuration reference](#configuration-reference)).
---
## Quick check from the command line
[local_rag.py](local_rag.py) can be run directly for a smoke test without the web server:
```bash
python local_rag.py
```
It loads the existing vector store, lists the documents in it, and runs **one hardcoded example question** through Ollama. This is only a demo: edit the `question` variable in `main()` to ask something else, and note that it always uses Ollama (the provider is not configurable from the command line). For normal use, run the web app instead.
---
## How it works (high level)
1. **Load documents** – PDFs via `PyPDFLoader`, text via `TextLoader`.
2. **Chunk** – `RecursiveCharacterTextSplitter` (defaults in [local_rag.py](local_rag.py)).
3. **Embed** – Hugging Face `sentence-transformers/all-MiniLM-L6-v2`.
4. **Store** – FAISS; retrieval uses `similarity_search_with_score`.
5. **Query** – The corpus search is a tool the LLM may call (`search_kafka_corpus`). It decides per message whether to search, writes the query itself from the chat history, and answers from the excerpts it found; greetings or thanks are answered without a search.
---
## Configuration reference (what to edit)
These are the main places to change behavior without restructuring the app.
### [server.py](server.py)
| What | Where |
|------|--------|
| Ollama model name | `OLLAMA_MODEL = "..."` |
| OpenAI model name | `OPENAI_MODEL = "..."` |
| Where the FAISS index is loaded from | `VECTORSTORE_PATH = "./vectorstore"` (must match your indexing script) |
### [local_rag.py](local_rag.py) – `LocalRAG.__init__`
| What | Where (approx.) |
|------|------------------|
| Default vector store folder | Parameter `vectorstore_path="./vectorstore"` |
| Embedding model | `HuggingFaceEmbeddings(model_name="sentence-transformers/...")` |
| Chunk size and overlap | Module-level `CHUNK_SIZE` and `CHUNK_OVERLAP` (used by `RecursiveCharacterTextSplitter` when adding documents) |
| Default Ollama / OpenAI model strings | Parameters `ollama_model`, `openai_model`, `ollama_base_url` |
Changing the embedding model or chunk settings requires **rebuilding** the vector store (old index is not compatible).
### [local_rag.py](local_rag.py) – `query_with_history`
| What | Where |
|------|--------|
| Default number of chunks retrieved (`k`) | Module-level `RETRIEVAL_K` (overrides: pass `k=` to `query` / `query_with_history`) |
| Extra text appended only to the **FAISS query** (biases retrieval, not the final answer phrasing) | `QUERY_ADDITIONAL_INSTRUCTIONS` (concatenated to the search query before embedding) |
| **System** prompt – when to search, how to answer | Module-level `SYSTEM_PROMPT` |
| Search tool name, description, searches per message | `SEARCH_TOOL_NAME`, `SEARCH_TOOL_DESCRIPTION`, `MAX_TOOL_ROUNDS` |
### [add_pdfs.py](add_pdfs.py)
| What | Where |
|------|--------|
| Folder to scan for PDFs | `DATA_ROOT = Path("...")` |
| Output vector store folder | `VECTORSTORE_PATH = "./vectorstore"` (keep in sync with `server.py`) |
| Wipe index vs merge | `CLEAR_VECTORSTORE_FIRST = True` (delete and rebuild) or `False` (append to existing index) |
---
## Dependencies (for developers)
See [requirements.txt](requirements.txt) for the full list (LangChain, FAISS, sentence-transformers, FastAPI, uvicorn, etc.).
local_rag.py
+5
View File
@@ -28,6 +28,10 @@ CHUNK_OVERLAP = 400
# Multilingual embedding model with a long input window (8192 tokens), so chunks # Multilingual embedding model with a long input window (8192 tokens), so chunks
# are never truncated. Changing it requires rebuilding the index. # are never truncated. Changing it requires rebuilding the index.
EMBEDDING_MODEL = "BAAI/bge-m3" EMBEDDING_MODEL = "BAAI/bge-m3"
# Same weights as safetensors (the official Hugging Face conversion, refs/pr/130). The main
# branch only has pytorch_model.bin, which transformers refuses to load with torch < 2.6;
# macOS x86 (conda env local_rag) is stuck at torch 2.2.2. Identical vectors, no index rebuild.
EMBEDDING_REVISION = "9a0624b896d81da7492a910ffa53731274b6cf3d"
# How PDFs are read: "pypdf" (PyPDFLoader, one unit per page) or "docling" # How PDFs are read: "pypdf" (PyPDFLoader, one unit per page) or "docling"
# (structured output from prepare_corpus.py, cleaned by docling_loader.py). # (structured output from prepare_corpus.py, cleaned by docling_loader.py).
@@ -105,6 +109,7 @@ def load_embeddings(model_name):
# Normalized vectors: FAISS's L2 distance then ranks exactly like cosine similarity. # Normalized vectors: FAISS's L2 distance then ranks exactly like cosine similarity.
_EMBEDDINGS[model_name] = HuggingFaceEmbeddings( _EMBEDDINGS[model_name] = HuggingFaceEmbeddings(
model_name=model_name, model_name=model_name,
model_kwargs={"revision": EMBEDDING_REVISION} if model_name == EMBEDDING_MODEL else {},
encode_kwargs={"normalize_embeddings": True}, encode_kwargs={"normalize_embeddings": True},
query_encode_kwargs={"normalize_embeddings": True}, query_encode_kwargs={"normalize_embeddings": True},
) )