From ce9a863a87edb352dfce4f3e76e87c1cd3ac957a Mon Sep 17 00:00:00 2001 From: Philipp Mock Date: Mon, 26 Oct 2026 07:26:29 +0100 Subject: [PATCH] Load bge-m3 as safetensors for x86; update README, add EINSTIEG.md - EMBEDDING_REVISION pins the official safetensors conversion of bge-m3, so the x86 env (torch 2.2.2) can load it; identical vectors, no index rebuild - README rewritten for the current state (personas, providers, retrieval, environments) - EINSTIEG.md: short German guide to running and configuring the chat Co-Authored-By: Claude Opus 5.5 --- EINSTIEG.md | 156 +++++++++++++++++++ README.md | 429 ++++++++++++++++++++++++++------------------------- local_rag.py | 5 + 3 files changed, 381 insertions(+), 209 deletions(-) create mode 100644 EINSTIEG.md diff --git a/EINSTIEG.md b/EINSTIEG.md new file mode 100644 index 0000000..424b479 --- /dev/null +++ b/EINSTIEG.md @@ -0,0 +1,156 @@ +# Kafka-Chat: Einstieg + +Stand: Oktober 2026. Ausführliche englische Referenz: [README.md](README.md). + +## Starten + +**Conda-Umgebungen** + +| Umgebung | Architektur | Einsatz | +|---|---|---| +| `docling` | arm64 (Apple Silicon), Python 3.12, torch 2.14 | Standard. Auch für Docling (`prepare_corpus.py`). | +| `local_rag` | x86_64 (Intel-Mac oder Rosetta), Python 3.10, torch 2.2.2 | Alternative für x86. Chat und Suche getestet; Indexbau sollte gehen (gleiches Embedding), ungetestet. Docling nicht. | + +- **Warum bge-m3 auf x86 nur über safetensors lädt:** Für macOS x86 gibt es höchstens torch 2.2.2. `transformers` lädt `.bin`-Gewichte aber erst ab torch 2.6, und bge-m3 hat im Hauptzweig nur `pytorch_model.bin`. +- **Die Lösung im Code:** `local_rag.py` lädt deshalb die offizielle safetensors-Konvertierung (`EMBEDDING_REVISION`). Die Gewichte sind dieselben, die Vektoren in beiden Umgebungen identisch, und dieselben Indizes funktionieren in beiden. +- **Geschwindigkeit unter Rosetta:** Spürbar langsamer. Im Test brauchte die Suche gut 5 statt gut 1 Sekunde, der Serverstart war dagegen nicht langsamer. + +**Voraussetzungen (einmalig)** +- Ollama mit dem Modell: `ollama pull gemma4:12b`. +- Für ChatGPT und Realtime eine Datei `.env` im Projektordner mit `OPENAI_API_KEY=sk-...`. Sie ist git-ignoriert, am besten mit `chmod 600` schützen. Ohne Key sind beide Einträge im Menü ausgegraut. +- Die beiden Vektorspeicher `vectorstore_pypdf_2000/` und `vectorstore_primary_pypdf_2000/`. Sie sind nicht im Git; Neubau siehe unten. + +**Server starten** + +```bash +cd /Users/Philipp/Desktop/workspace/diverses/local_rag +conda activate docling # auf x86: conda activate local_rag +uvicorn server:app --port 8010 +``` + +Dann im Browser http://localhost:8010 öffnen. Der Start dauert etwa eine Minute: Embedding-Modell, Reranker und Stichwortindex werden geladen. Danach ist auch die erste Frage schnell. + +- Optionen: `--reload` startet den Server bei Codeänderungen neu, `--log-level debug` zeigt mehr Details. +- Ohne `--reload` braucht jede Änderung an `.py`-Dateien einen Neustart. Bei `templates/chat.html` reicht es, die Seite neu zu laden. + +**Was im Terminal steht, pro Frage** + +``` +[RAG] User question: '…' +[Zeit] LLM-Aufruf 1: 6.86s (Suche angefordert) +[RAG] Tool search '…': 5 chunk(s) +[Zeit] Suche 1.37s (davon Reranker 1.21s) +[Zeit] LLM-Aufruf 2: 15.31s (Antwort) +[Zeit] Gesamt 23.54s +``` + +Danach folgen die gefundenen Chunks mit Quelle, Seite und Score sowie die vollständige Antwort. + +## Aufbau + +``` +Browser (templates/chat.html) + │ POST /api/chat {message, history, persona, llm_provider} + ▼ +server.py ── wählt Figur (personas.py) und RAG-Instanz (Anbieter × Vektorspeicher) + ▼ +local_rag.py · LocalRAG.query_with_history + │ Systemprompt + Verlauf + Frage an das LLM, Such-Werkzeug angeboten + │ ┌─ LLM ruft search_kafka_corpus(query) auf (höchstens 3 Runden) + │ │ retrieve(): FAISS (bge-m3) + BM25 → Fusion (RRF) → 25 Kandidaten + │ │ → Reranker (bge-reranker-v2-m3) → beste 5 Chunks + │ └─ LLM antwortet mit den Auszügen + ▼ +Antwort + gefundene Chunks + Suchanfragen → Browser +``` + +**Dateien** + +| Datei | Inhalt | +|---|---| +| [server.py](server.py) | FastAPI-Server, Modellwahl, eine RAG-Instanz pro Anbieter und Vektorspeicher, Konsolenausgabe | +| [personas.py](personas.py) | die drei Figuren: Texte, Bilder, Systemprompts, Werkzeugbeschreibung, Vektorspeicher | +| [local_rag.py](local_rag.py) | Laden der Modelle, Suche (`retrieve`), Werkzeugschleife (`query_with_history`), Indexbau | +| [realtime_chat.py](realtime_chat.py) | Anbindung der OpenAI Realtime API (WebSocket, nur Text) als LangChain-Modell | +| [templates/chat.html](templates/chat.html) | Oberfläche, Menüs für Figur und Modell | +| `templates/visuals/` | Bilder (`*.png` Vorlagen, `*.jpg` Webversionen, `*_avatar.jpg`) | +| [add_pdfs.py](add_pdfs.py) | Vektorspeicher aus einem PDF-Ordner bauen | +| [eval_retrieval.py](eval_retrieval.py) | Suchqualität auf einem Goldset messen (Recall@k, MRR, optional LLM-Judge) | + +## Stellschrauben + +### Verhalten der Figuren: [personas.py](personas.py) + +Pro Figur ein Eintrag in `PERSONAS`; eine neue Figur ist ein neuer Eintrag mit eigenem Bild. + +| Feld | wirkt auf | +|---|---| +| `system_prompt` | Ton, Rolle, Regeln für Suche und Belege | +| `tool_description` | was das Modell über das Such-Werkzeug erfährt (ohne Feld gilt `SEARCH_TOOL_DESCRIPTION`) | +| `vectorstore` | welcher Index durchsucht wird | +| `label`, `topic`, `name`, `role`, `place`, `status`, `greeting`, `suggestions` | Texte in der Oberfläche | +| `image`, `image_position`, `image_alt`, `avatar` | Bilder | + +- Die Regeln zur Suche für Zeitgenosse und Autor stehen gemeinsam in `_TOOL_RULES`. +- Der Prompt der Forscherin ist `SYSTEM_PROMPT` in `local_rag.py`. + +### Sprachmodelle: [server.py](server.py) und [local_rag.py](local_rag.py) + +| Einstellung | aktuell | wo | +|---|---|---| +| lokales Modell (Ollama) | `gemma4:12b`, denkt standardmäßig (`reasoning` nicht gesetzt) | `OLLAMA_MODEL` in server.py, `ChatOllama(...)` in local_rag.py | +| ChatGPT | `gpt-6.1-sol` | `OPENAI_MODEL` in server.py | +| Denkaufwand ChatGPT | `low`, über die Responses API | `ChatOpenAI(...)` in local_rag.py | +| Realtime | `gpt-realtime-2.1-mini` | `REALTIME_MODEL` in server.py | +| Voreinstellung im Menü | ChatGPT | `selected` in chat.html | + +- **gpt-6.1-sol:** kennt keinen Denkaufwand `none`. Werkzeuge funktionieren nur über die Responses API, deshalb steht `use_responses_api=True`. +- **Realtime:** öffnet pro Modellaufruf eine eigene WebSocket-Sitzung. + +### Suche: Konstanten oben in [local_rag.py](local_rag.py) + +| Konstante | aktuell | Bedeutung | +|---|---|---| +| `RETRIEVAL_K` | 5 | Chunks pro Suche an das LLM | +| `RETRIEVAL_MODE` | `hybrid` | `dense` (nur Vektoren), `hybrid` (Vektoren + BM25, RRF), `union` | +| `CANDIDATES_K` | 25 | Kandidaten vor dem Reranker. 25 war schneller und nicht schlechter als 50. | +| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | `None` schaltet das Reranking ab. Läuft auf der GPU in fp16. | +| `MAX_TOOL_ROUNDS` | 3 | Suchen pro Nachricht, danach muss das Modell antworten | +| `SEARCH_TOOL_DESCRIPTION` | Text | Werkzeugbeschreibung für die Forscherin | + +Die Suche dauert etwa 1 bis 1,5 Sekunden pro Aufruf, fast nur der Reranker. Den größten Teil der Antwortzeit brauchen die Modellaufrufe. + +### Index: Neubau nötig bei Änderung + +| Konstante | aktuell | +|---|---| +| `EMBEDDING_MODEL` | `BAAI/bge-m3` (normalisierte Vektoren) | +| `CHUNK_SIZE` / `CHUNK_OVERLAP` | 2000 / 400 Zeichen | +| `PDF_LOADER` | `pypdf`; `docling` war im Goldset-Vergleich nicht besser | +| `CHUNK_HEADERS` | `False` | + +Neubau der beiden Indizes. `data_primary/` enthält nur einen Verweis auf `data/Kafka_Die_Verwandlung_.pdf`. + +```bash +python add_pdfs.py --data data --index ./vectorstore_pypdf_2000 --loader pypdf --no-headers +``` + +```bash +python add_pdfs.py --data data_primary --index ./vectorstore_primary_pypdf_2000 --loader pypdf --no-headers +``` + +Jeder Index speichert seine Einstellungen in `index_info.json`. Passt das Embedding-Modell nicht zum Code, verweigert `LocalRAG` das Laden. + +### Suchqualität messen + +```bash +python eval_retrieval.py --goldset eval/goldset_kafka_human.csv --report eval/mein_bericht.txt +``` + +- `--judge gpt-6-luna` lässt zusätzlich ein LLM prüfen, ob die Antwort in den Top 5 steht; das braucht den Key. +- Stand auf dem menschlichen Goldset (30 Fragen): R@5 0,73, MRR 0,68. + +## Gut zu wissen + +- **Speicher:** Embedding-Modell, Reranker und BM25-Index gibt es pro Prozess nur einmal, auch bei mehreren Anbietern. Ollama braucht zusätzlich etwa 8 bis 10 GB für gemma4:12b. +- **Nicht im Git:** Korpus-PDFs (`data/`), Vektorspeicher, `.env` und Schlüsseldateien. diff --git a/README.md b/README.md index 58c5700..883f838 100644 --- a/README.md +++ b/README.md @@ -1,245 +1,256 @@ -# Local RAG Setup +# Local RAG – Kafka Chat -Minimal RAG implementation with LangChain, FAISS, and either **Ollama** (local) or **OpenAI** (API key). A web chat UI is included. +Web chat with three conversation partners on Kafka's *Die Verwandlung*, built on LangChain, FAISS and either **Ollama** (local), **OpenAI** (Responses API) or the **OpenAI Realtime API**. The corpus search is a tool the LLM calls when it needs evidence. + +A short German quick start is in [EINSTIEG.md](EINSTIEG.md). --- -## What you need (before you start) +## Environments -- **Python 3.10 or newer** ([python.org](https://www.python.org/downloads/)) -- **Git** (optional, only if you clone the project) -- Either: - - **Ollama** installed and running ([ollama.com](https://ollama.com)), with at least one model pulled, **or** - - An **OpenAI API key** (if you use OpenAI in the chat) +| Conda env | Architecture | Use | +|---|---|---| +| `docling` | arm64 (Apple Silicon), Python 3.12, torch 2.14 | Default. Also needed for Docling (`prepare_corpus.py`). | +| `local_rag` | x86_64 (Intel Mac or Rosetta), Python 3.10, torch 2.2.2 | Alternative for x86. Chat and search work; Docling does not. Search is slower under Rosetta (about 5 s instead of about 1 s). | ---- +**bge-m3 on x86** -## Install dependencies (step by step) +- **The problem:** macOS x86 is limited to torch 2.2.2. `transformers` refuses to load `.bin` weights below torch 2.6, and the main branch of bge-m3 only ships `pytorch_model.bin`. +- **The fix:** [local_rag.py](local_rag.py) loads the official safetensors conversion (`EMBEDDING_REVISION`) in both environments. The weights are identical and so are the vectors, so the same indexes work everywhere. -Install [Miniconda](https://docs.conda.io/en/latest/miniconda.html) or [Anaconda](https://www.anaconda.com/) if you do not have Conda yet. - -All commands below assume your terminal is open **in the project folder** (the folder that contains `requirements.txt`). +**New environment** ```bash conda create -n local_rag python=3.10 -y -conda activate local_rag -pip install --upgrade pip -pip install -r requirements.txt ``` -Use `conda activate local_rag` in every new terminal session before running `python` or `uvicorn` for this project. - -### OpenAI (only if you use the OpenAI provider in the chat) - -In the same terminal **before** starting the server: - ```bash -export OPENAI_API_KEY="your-key-here" +conda activate local_rag && pip install -r requirements.txt ``` -On Windows (Command Prompt): `set OPENAI_API_KEY=your-key-here` +For the arm64 env on Apple Silicon, create it with `CONDA_SUBDIR=osx-arm64` so that conda does not fall back to x86 packages. ---- +**Ollama** -## Run Ollama (only if you use Ollama) - -In a **separate** terminal: - -```bash -ollama serve -``` - -In another terminal, pull a model once (example): +Ollama must be running and the model pulled once: ```bash ollama pull gemma4:12b ``` -The model name must match what you configure in `server.py` (see [Configuration reference](#configuration-reference)). +**OpenAI** (ChatGPT and Realtime) ---- +Put the key into a `.env` file in the project folder. It is git-ignored; protect it with `chmod 600`. An exported `OPENAI_API_KEY` also works and takes precedence. -## Build the vector store from a folder of PDFs - -The project includes [add_pdfs.py](add_pdfs.py). It finds every **`.pdf`** file under a folder you choose (including subfolders), then chunks, embeds, and saves to FAISS. - -**Two modes** (set in the script): - -| Setting | Behavior | -|---------|----------| -| `CLEAR_VECTORSTORE_FIRST = True` | Deletes the existing vector store folder, then builds a **new** index from the PDFs under `DATA_ROOT`. Use this for a full rebuild. | -| `CLEAR_VECTORSTORE_FIRST = False` | Keeps the current index (if it exists) and **merges** chunks from the PDFs under `DATA_ROOT` into it. Use this to add another batch of PDFs without wiping what you already indexed. | - -**Steps:** - -1. Open [add_pdfs.py](add_pdfs.py) in a text editor. -2. Set **`DATA_ROOT`** to the folder that contains your PDFs (absolute path or path relative to how you run the script). -3. Set **`CLEAR_VECTORSTORE_FIRST`** to `True` (fresh index) or `False` (append to existing store). -4. Optionally set **`VECTORSTORE_PATH`** (default: `./vectorstore`). It must match **`VECTORSTORE_PATH`** in [server.py](server.py) so the chat loads the same index. -5. From the project folder, with `conda activate local_rag` (or your chosen env name): - -```bash -python add_pdfs.py +``` +OPENAI_API_KEY=sk-... ``` -Indexing can take a long time for many large PDFs. When it finishes, you should see `Vector store saved to ...`. - -**Note:** This script only indexes **PDF** files. To add `.txt` or `.md` files, use the Python snippet below or call `add_documents` yourself. +Without a key, both OpenAI entries in the model menu are greyed out. --- -## Add more documents later (alternative to add_pdfs) +## Run the chat -You can also merge files by hand with a short script (any mix of supported types): +```bash +cd /Users/Philipp/Desktop/workspace/diverses/local_rag && conda activate docling && uvicorn server:app --port 8010 +``` + +Open http://localhost:8010. + +- **Startup takes about a minute.** It loads the embedding model and the reranker and builds the BM25 indexes, so the first question is as fast as any other. +- **`--reload`** restarts the server on code changes. Without it, every change to a `.py` file needs a restart; for `templates/chat.html`, reloading the page is enough. + +The header has two menus: +- **Gegenüber:** Forscherin / Zeitgenosse / Autor. +- **Modell:** Ollama / ChatGPT / Realtime Mini. ChatGPT is the default and falls back to Ollama without a key. + +--- + +## How it works + +``` +Browser (templates/chat.html) + │ POST /api/chat {message, history, persona, llm_provider} + ▼ +server.py ── picks the persona (personas.py) and the RAG instance (provider × vector store) + ▼ +LocalRAG.query_with_history (local_rag.py) + │ system prompt + history + question to the LLM, search tool offered + │ ┌─ LLM calls search_kafka_corpus(query) (at most MAX_TOOL_ROUNDS = 3 times) + │ │ retrieve(): FAISS (bge-m3) + BM25 → Reciprocal Rank Fusion → 25 candidates + │ │ → cross-encoder bge-reranker-v2-m3 → best 5 chunks + │ └─ LLM answers from the excerpts, citing source and page + ▼ +answer + retrieved chunks + search queries → browser +``` + +- **Personas:** Each persona has its own system prompt, its own tool description and its own vector store. + - The researcher searches the whole corpus: the primary text and the secondary literature. + - The contemporary and Kafka himself search only *Die Verwandlung*. +- **Shared resources:** The embedding model, reranker and BM25 index exist once per process and are shared by all RAG instances, whatever the provider. They are loaded at startup with `LocalRAG.preload()`. +- **Realtime:** [realtime_chat.py](realtime_chat.py) wraps the OpenAI Realtime API (WebSocket, text only) as a LangChain chat model. The tool loop is therefore the same for all providers. It opens one session per LLM call. + +### Files + +| File | Content | +|---|---| +| [server.py](server.py) | FastAPI app, model names, RAG instances, startup preload, console trace | +| [personas.py](personas.py) | the three personas: UI texts, images, system prompts, tool description, vector store | +| [local_rag.py](local_rag.py) | model loading, retrieval (`retrieve`), tool loop (`query_with_history`), indexing (`add_documents`) | +| [realtime_chat.py](realtime_chat.py) | `ChatOpenAIRealtime`, the Realtime API as a LangChain model | +| [templates/chat.html](templates/chat.html) | UI, persona and model menus | +| `templates/visuals/` | images (`*.png` originals, `*.jpg` web versions, `*_avatar.jpg`) | +| [add_pdfs.py](add_pdfs.py) | build a vector store from a PDF folder | +| [prepare_corpus.py](prepare_corpus.py), [docling_loader.py](docling_loader.py), [split_spreads.py](split_spreads.py) | Docling pipeline: split double-page scans, OCR where needed, structured extraction | +| [eval_retrieval.py](eval_retrieval.py) | retrieval quality on a goldset (Recall@k, MRR, optional LLM judge) | +| [grobid_qa.py](grobid_qa.py) | extraction quality checks (GROBID, Docling, pypdf) | + +--- + +## Configuration reference + +### Personas – [personas.py](personas.py) + +Each persona is one entry in `PERSONAS`. + +| Field | Effect | +|---|---| +| `system_prompt` | role, tone, rules for searching and citing | +| `tool_description` | what the LLM is told about the search tool (default: `SEARCH_TOOL_DESCRIPTION`) | +| `vectorstore` | which index is searched (`VECTORSTORE_PATH` or `PRIMARY_VECTORSTORE_PATH`) | +| `label`, `topic`, `name`, `role`, `place`, `status`, `greeting`, `suggestions` | UI texts | +| `image`, `image_position`, `image_alt`, `avatar` | images | + +- The search rules shared by the contemporary and Kafka are in `_TOOL_RULES`. +- The researcher's prompt is `SYSTEM_PROMPT` in `local_rag.py`. +- `PUBLIC_FIELDS` lists what the UI receives; system prompts stay on the server. + +### Models – [server.py](server.py) and [local_rag.py](local_rag.py) + +| What | Current | Where | +|---|---|---| +| Ollama model | `gemma4:12b` (thinks by default, `reasoning` not set) | `OLLAMA_MODEL` (server.py), `ChatOllama(...)` (local_rag.py) | +| OpenAI model | `gpt-6.1-sol` | `OPENAI_MODEL` (server.py) | +| OpenAI reasoning effort | `low`, via the Responses API | `ChatOpenAI(..., reasoning_effort="low", use_responses_api=True)` (local_rag.py) | +| Realtime model | `gpt-realtime-2.1-mini` | `REALTIME_MODEL` (server.py) | +| Default in the model menu | ChatGPT | `selected` in chat.html | + +GPT-6 models accept function tools on Chat Completions only with `reasoning_effort="none"`, and gpt-6.1-sol does not support `none` at all. Tool calls therefore go through the Responses API. + +### Retrieval – module constants in [local_rag.py](local_rag.py) + +| Constant | Current | Meaning | +|---|---|---| +| `RETRIEVAL_K` | 5 | chunks per search handed to the LLM | +| `RETRIEVAL_MODE` | `hybrid` | `dense` (vectors only), `hybrid` (vectors + BM25, RRF), `union` (BM25 adds candidates only) | +| `CANDIDATES_K` | 25 | candidates before reranking; 25 was faster than 50 and not worse on either goldset | +| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | `None` disables reranking; runs in fp16 on the GPU (MPS/CUDA), fp32 on the CPU | +| `MAX_TOOL_ROUNDS` | 3 | searches per message before the LLM must answer | +| `SEARCH_TOOL_NAME` / `SEARCH_TOOL_DESCRIPTION` | `search_kafka_corpus` / text | tool name, default tool description | +| `QUERY_ADDITIONAL_INSTRUCTIONS` | empty | text appended to the search query; it lowered recall, so it is left empty | + +### Index – rebuild required after changing these + +| Constant | Current | +|---|---| +| `EMBEDDING_MODEL` / `EMBEDDING_REVISION` | `BAAI/bge-m3`, safetensors revision (normalized vectors) | +| `CHUNK_SIZE` / `CHUNK_OVERLAP` | 2000 / 400 characters | +| `PDF_LOADER` | `pypdf`; `docling` was not better in the goldset comparison | +| `CHUNK_HEADERS` | `False` | + +Each index stores its settings in `index_info.json`. `LocalRAG` refuses to load an index built with a different embedding model. + +--- + +## Build the vector stores + +[add_pdfs.py](add_pdfs.py) indexes every PDF under a folder, including subfolders. With `CLEAR_VECTORSTORE_FIRST = True` (the default in the script), the target folder is deleted and rebuilt; set it to `False` to merge into an existing index. + +Whole corpus, used by the researcher: + +```bash +python add_pdfs.py --data data --index ./vectorstore_pypdf_2000 --loader pypdf --no-headers +``` + +Primary text only, used by the contemporary and Kafka. `data_primary/` contains just a link to `data/Kafka_Die_Verwandlung_.pdf`: + +```bash +python add_pdfs.py --data data_primary --index ./vectorstore_primary_pypdf_2000 --loader pypdf --no-headers +``` + +Other options: `--loader docling` (needs `prepare_corpus.py` output, `--processed`), `--chunk-size`, `--chunk-overlap`. + +- **Docling pipeline (arm64 env only):** `python prepare_corpus.py [pdf_dir]` writes to `corpus_processed/data_split`. Options: `--only "name1;name2"`, `--no-split`, `--layout`, `--no-headings`. +- **Not in git:** the corpus PDFs (`data/`, `data_primary/`), vector stores (`vectorstore*/`), extraction output and `.env`. + +--- + +## Measure retrieval quality + +```bash +python eval_retrieval.py --goldset eval/goldset_kafka_human.csv --index ./vectorstore_pypdf_2000 --report eval/my_report.txt +``` + +- Options: `--mode dense|hybrid|union`, `--no-rerank`, `--judge MODEL`. +- `--judge` lets an OpenAI model check whether the answer is in the top 5. It needs the key; verdicts are cached in `eval/judge_cache.json`. +- Current numbers (whole corpus, hybrid, 25 candidates, fp16 reranker): + +| Goldset | R@5 | MRR | +|---|---|---| +| human (30 questions) | 0.73 | 0.68 | +| synthetic (43 questions) | 0.98 | 0.79 | + +--- + +## API + +| Endpoint | Purpose | +|---|---| +| `POST /api/chat` | Body: `message`, `history` (list of `{role, content}`), `persona` (`researcher`, `contemporary`, `kafka`), `llm_provider` (`ollama`, `openai`, `realtime`; API default `ollama`) | +| `GET /api/personas` | default persona and the public fields of all personas | +| `GET /api/providers` | which providers are available (`openai` and `realtime` need a key at startup) | +| `GET /api/health` | server status and whether all vector stores are loaded | + +**Response of `POST /api/chat`** + +| Field | Meaning | +|---|---| +| `answer` | the answer (empty on error) | +| `error` | error message or `null` | +| `retrieved` | chunks the LLM saw: `content`, `source`, `page`, `score` (reranker relevance, higher is better) | +| `searches` | the queries the LLM sent to the search tool (empty if it did not search) | + +--- + +## Debugging and timing + +The server terminal prints, per question: + +``` +[RAG] User question: '…' +[Zeit] LLM-Aufruf 1: 6.86s (Suche angefordert) +[RAG] Tool search '…': 5 chunk(s) +[Zeit] Suche 1.37s (davon Reranker 1.21s) +[Zeit] LLM-Aufruf 2: 15.31s (Antwort) +[Zeit] Gesamt 23.54s +``` + +After that it prints the retrieved chunks (source, page, score, preview) and the full answer, with provider, model and persona. + +In the browser console, each answer logs a collapsed group `[RAG] Retrieved N chunk(s)` with the full chunk texts. + +Where to look when an answer is wrong: +- **Wrong chunks:** tune retrieval, i.e. `CANDIDATES_K`, `RETRIEVAL_MODE` or the reranker, and measure with `eval_retrieval.py`. +- **Right chunks but a bad answer, or searching when it should not (or the reverse):** edit the persona's `system_prompt` or `tool_description`. + +To try retrieval without the server: ```python from local_rag import LocalRAG - -rag = LocalRAG(vectorstore_path="./vectorstore") # same path as server.py -rag.add_documents([ - "path/to/new1.pdf", - "path/to/notes.txt", -]) +rag = LocalRAG(vectorstore_path="./vectorstore_primary_pypdf_2000") +for doc, score in rag.retrieve("Wo versteckt sich Gregor?", k=5): + print(doc.metadata.get("page"), round(score, 3), doc.page_content[:80]) ``` -`add_documents` merges new chunks into the existing FAISS store and saves it again—the same behavior as [add_pdfs.py](add_pdfs.py) with `CLEAR_VECTORSTORE_FIRST = False`. - ---- - -## Swap or experiment with different vector stores - -The vector index is stored on disk under the folder given by **`VECTORSTORE_PATH`** (default `./vectorstore`). That folder contains files such as `index.faiss` and `index.pkl`. - -**To use a different index:** - -1. Set **`VECTORSTORE_PATH`** in both [server.py](server.py) and any script you use to build the index (e.g. [add_pdfs.py](add_pdfs.py)) to the **same** path, e.g. `./vectorstore_experiment`. -2. Rebuild the index (run `add_pdfs.py` or `add_documents`) so that folder is created. -3. **Restart** the web server so it loads the new path at startup. - -**Tips:** - -- Keep multiple copies of the folder (e.g. `vectorstore_backup`, `vectorstore_papers_only`) and swap `VECTORSTORE_PATH` to switch between them. -- If you change **chunk size**, **embedding model**, or **FAISS** usage in code, treat the old index as incompatible: use a new `VECTORSTORE_PATH` or delete the old folder and rebuild. - ---- - -## Run the chat web app - -With the Conda environment activated (`conda activate local_rag`) and (if needed) `OPENAI_API_KEY` set: - -```bash -uvicorn server:app --reload -``` - -Open [http://127.0.0.1:8000](http://127.0.0.1:8000) or [http://localhost:8000](http://localhost:8000). - -- Use the **LLM provider** dropdown: **Ollama** or **OpenAI** (OpenAI only works if the server was started with a valid `OPENAI_API_KEY`). -- You need a **non-empty vector store** (see above) for answers to work. - ---- - -## API (short reference) - -| Endpoint | Purpose | -|----------|---------| -| `POST /api/chat` | Body: `message`, optional `history`, optional `llm_provider` (`ollama` or `openai`) | -| `GET /api/providers` | Which providers are available (`openai` false if no API key at startup) | -| `GET /api/health` | Server and whether a vector store is loaded | - -**`POST /api/chat` response fields:** - -| Field | Type | Meaning | -|-------|------|---------| -| `answer` | string | The generated answer (empty string on error) | -| `error` | string or `null` | Error message, e.g. empty question or OpenAI not configured | -| `retrieved` | list or `null` | The chunks used as context for this answer | - -Each entry in `retrieved` has: - -| Field | Type | Meaning | -|-------|------|---------| -| `content` | string | The chunk text handed to the LLM | -| `source` | string | Path of the file the chunk came from | -| `page` | int or `null` | Page number (PDFs only) | -| `score` | float or `null` | FAISS **L2 distance** – **lower = more similar** (it is a distance, not a similarity percentage) | - ---- - -## Inspect what was retrieved (debugging) - -Every answer reports which chunks it was based on, in two places: - -- **Browser:** open the developer console (Chrome/Edge: `F12` or `Cmd`+`Option`+`I`, Firefox: `F12`) before sending a question. Each answer logs a collapsed group `[RAG] Retrieved N chunk(s)` with the full text, source file and page of every chunk. -- **Server terminal:** the terminal running `uvicorn` prints the search queries the LLM sent to the corpus search tool (or that it did not search), the scores, a 150-character preview of each chunk, and the raw LLM response. - -Use this when an answer looks wrong: if the right chunks were never retrieved, tune `RETRIEVAL_K`, `QUERY_ADDITIONAL_INSTRUCTIONS` or the chunk size; if the chunks were correct but the answer was not, or the LLM searched when it should not (or vice versa), tune `SYSTEM_PROMPT` (see [Configuration reference](#configuration-reference)). - ---- - -## Quick check from the command line - -[local_rag.py](local_rag.py) can be run directly for a smoke test without the web server: - -```bash -python local_rag.py -``` - -It loads the existing vector store, lists the documents in it, and runs **one hardcoded example question** through Ollama. This is only a demo: edit the `question` variable in `main()` to ask something else, and note that it always uses Ollama (the provider is not configurable from the command line). For normal use, run the web app instead. - ---- - -## How it works (high level) - -1. **Load documents** – PDFs via `PyPDFLoader`, text via `TextLoader`. -2. **Chunk** – `RecursiveCharacterTextSplitter` (defaults in [local_rag.py](local_rag.py)). -3. **Embed** – Hugging Face `sentence-transformers/all-MiniLM-L6-v2`. -4. **Store** – FAISS; retrieval uses `similarity_search_with_score`. -5. **Query** – The corpus search is a tool the LLM may call (`search_kafka_corpus`). It decides per message whether to search, writes the query itself from the chat history, and answers from the excerpts it found; greetings or thanks are answered without a search. - ---- - -## Configuration reference (what to edit) - -These are the main places to change behavior without restructuring the app. - -### [server.py](server.py) - -| What | Where | -|------|--------| -| Ollama model name | `OLLAMA_MODEL = "..."` | -| OpenAI model name | `OPENAI_MODEL = "..."` | -| Where the FAISS index is loaded from | `VECTORSTORE_PATH = "./vectorstore"` (must match your indexing script) | - -### [local_rag.py](local_rag.py) – `LocalRAG.__init__` - -| What | Where (approx.) | -|------|------------------| -| Default vector store folder | Parameter `vectorstore_path="./vectorstore"` | -| Embedding model | `HuggingFaceEmbeddings(model_name="sentence-transformers/...")` | -| Chunk size and overlap | Module-level `CHUNK_SIZE` and `CHUNK_OVERLAP` (used by `RecursiveCharacterTextSplitter` when adding documents) | -| Default Ollama / OpenAI model strings | Parameters `ollama_model`, `openai_model`, `ollama_base_url` | - -Changing the embedding model or chunk settings requires **rebuilding** the vector store (old index is not compatible). - -### [local_rag.py](local_rag.py) – `query_with_history` - -| What | Where | -|------|--------| -| Default number of chunks retrieved (`k`) | Module-level `RETRIEVAL_K` (overrides: pass `k=` to `query` / `query_with_history`) | -| Extra text appended only to the **FAISS query** (biases retrieval, not the final answer phrasing) | `QUERY_ADDITIONAL_INSTRUCTIONS` (concatenated to the search query before embedding) | -| **System** prompt – when to search, how to answer | Module-level `SYSTEM_PROMPT` | -| Search tool name, description, searches per message | `SEARCH_TOOL_NAME`, `SEARCH_TOOL_DESCRIPTION`, `MAX_TOOL_ROUNDS` | - -### [add_pdfs.py](add_pdfs.py) - -| What | Where | -|------|--------| -| Folder to scan for PDFs | `DATA_ROOT = Path("...")` | -| Output vector store folder | `VECTORSTORE_PATH = "./vectorstore"` (keep in sync with `server.py`) | -| Wipe index vs merge | `CLEAR_VECTORSTORE_FIRST = True` (delete and rebuild) or `False` (append to existing index) | - ---- - -## Dependencies (for developers) - -See [requirements.txt](requirements.txt) for the full list (LangChain, FAISS, sentence-transformers, FastAPI, uvicorn, etc.). +`python local_rag.py` still runs an old demo against `./vectorstore` and is not maintained. diff --git a/local_rag.py b/local_rag.py index 4f94184..921a2e8 100644 --- a/local_rag.py +++ b/local_rag.py @@ -28,6 +28,10 @@ CHUNK_OVERLAP = 400 # Multilingual embedding model with a long input window (8192 tokens), so chunks # are never truncated. Changing it requires rebuilding the index. EMBEDDING_MODEL = "BAAI/bge-m3" +# Same weights as safetensors (the official Hugging Face conversion, refs/pr/130). The main +# branch only has pytorch_model.bin, which transformers refuses to load with torch < 2.6; +# macOS x86 (conda env local_rag) is stuck at torch 2.2.2. Identical vectors, no index rebuild. +EMBEDDING_REVISION = "9a0624b896d81da7492a910ffa53731274b6cf3d" # How PDFs are read: "pypdf" (PyPDFLoader, one unit per page) or "docling" # (structured output from prepare_corpus.py, cleaned by docling_loader.py). @@ -105,6 +109,7 @@ def load_embeddings(model_name): # Normalized vectors: FAISS's L2 distance then ranks exactly like cosine similarity. _EMBEDDINGS[model_name] = HuggingFaceEmbeddings( model_name=model_name, + model_kwargs={"revision": EMBEDDING_REVISION} if model_name == EMBEDDING_MODEL else {}, encode_kwargs={"normalize_embeddings": True}, query_encode_kwargs={"normalize_embeddings": True}, )