Local RAG – Kafka Chat

Web chat with three conversation partners on Kafka's Die Verwandlung, built on LangChain, FAISS and either Ollama (local), OpenAI (Responses API) or the OpenAI Realtime API. The corpus search is a tool the LLM calls when it needs evidence.

A short German quick start is in EINSTIEG.md.

---

Environments

Conda env	Architecture	Use
docling	arm64 (Apple Silicon), Python 3.12, torch 2.14	Default. Also needed for Docling (prepare_corpus.py).
local_rag	x86_64 (Intel Mac or Rosetta), Python 3.10, torch 2.2.2	Alternative for x86. Chat, search and indexing tested; Docling does not run. Search is slower under Rosetta (about 2.3 s instead of about 1 s).

bge-m3 on x86

• The problem: macOS x86 is limited to torch 2.2.2. transformers refuses to load .bin weights below torch 2.6, and the main branch of bge-m3 only ships pytorch_model.bin.
• The fix: local_rag.py loads the official safetensors conversion (EMBEDDING_REVISION) in both environments. The weights are identical and so are the vectors, so the same indexes work everywhere.

New environment

envs/ holds one file per environment with exact versions, exported from the working environments. The header of each file has the exact commands.

CONDA_SUBDIR=osx-arm64 conda env create -f envs/environment-docling.yml

CONDA_SUBDIR=osx-64 conda env create -f envs/environment-local_rag.yml

CONDA_SUBDIR forces the architecture; without it, conda on Apple Silicon may pick the wrong one. requirements.txt lists only the direct dependencies, without versions.

Ollama

Ollama must be running and the model pulled once:

ollama pull gemma4:12b

OpenAI (ChatGPT and Realtime)

Put the key into a .env file in the project folder. It is git-ignored; protect it with chmod 600. An exported OPENAI_API_KEY also works and takes precedence.

OPENAI_API_KEY=sk-...

Without a key, both OpenAI entries in the model menu are greyed out.

---

Run the chat

cd /Users/Philipp/Desktop/workspace/diverses/local_rag && conda activate docling && uvicorn server:app --port 8010

Open http://localhost:8010.

• Startup takes about a minute. It loads the embedding model and the reranker and builds the BM25 indexes, so the first question is as fast as any other.
• --reload restarts the server on code changes. Without it, every change to a .py file needs a restart; for templates/chat.html, reloading the page is enough.

The header has two menus:

• Gegenüber: Forscherin / Zeitgenosse / Autor.
• Modell: Ollama / ChatGPT / Realtime Mini. ChatGPT is the default and falls back to Ollama without a key.

---

How it works

Browser (templates/chat.html)
   │  POST /api/chat {message, history, persona, llm_provider}
   ▼
server.py ── picks the persona (personas.py) and the RAG instance (provider × vector store)
   ▼
LocalRAG.query_with_history (local_rag.py)
   │  system prompt + history + question to the LLM, search tool offered
   │  ┌─ LLM calls search_kafka_corpus(query)  (at most MAX_TOOL_ROUNDS = 3 times)
   │  │    retrieve(): FAISS (bge-m3) + BM25 → Reciprocal Rank Fusion → 25 candidates
   │  │                → cross-encoder bge-reranker-v2-m3 → best 5 chunks
   │  └─ LLM answers from the excerpts, citing source and page
   ▼
answer + retrieved chunks + search queries → browser

• Personas: Each persona has its own system prompt, its own tool description and its own vector store.
  • The researcher searches the whole corpus: the primary text and the secondary literature.
  • The contemporary and Kafka himself search only Die Verwandlung.
• Shared resources: The embedding model, reranker and BM25 index exist once per process and are shared by all RAG instances, whatever the provider. They are loaded at startup with LocalRAG.preload().
• Realtime: realtime_chat.py wraps the OpenAI Realtime API (WebSocket, text only) as a LangChain chat model. The tool loop is therefore the same for all providers. It opens one session per LLM call.

Files

File	Content
server.py	FastAPI app, model names, RAG instances, startup preload, console trace
personas.py	the three personas: UI texts, images, system prompts, tool description, vector store
local_rag.py	model loading, retrieval (retrieve), tool loop (query_with_history), indexing (add_documents)
realtime_chat.py	ChatOpenAIRealtime, the Realtime API as a LangChain model
templates/chat.html	UI, persona and model menus
templates/visuals/	images (*.png originals, *.jpg web versions, *_avatar.jpg)
add_pdfs.py	build a vector store from a PDF folder
prepare_corpus.py, docling_loader.py, split_spreads.py	Docling pipeline: split double-page scans, OCR where needed, structured extraction
eval_retrieval.py	retrieval quality on a goldset (Recall@k, MRR, optional LLM judge)
grobid_qa.py	extraction quality checks (GROBID, Docling, pypdf)

---

Configuration reference

Personas – personas.py

Each persona is one entry in PERSONAS.

Field	Effect
system_prompt	role, tone, rules for searching and citing
tool_description	what the LLM is told about the search tool (default: SEARCH_TOOL_DESCRIPTION)
vectorstore	which index is searched (VECTORSTORE_PATH or PRIMARY_VECTORSTORE_PATH)
label, topic, name, role, place, status, greeting, suggestions	UI texts
image, image_position, image_alt, avatar	images

• The search rules shared by the contemporary and Kafka are in _TOOL_RULES.
• The researcher's prompt is SYSTEM_PROMPT in local_rag.py.
• PUBLIC_FIELDS lists what the UI receives; system prompts stay on the server.

Models – server.py and local_rag.py

What	Current	Where
Ollama model	gemma4:12b (thinks by default, reasoning not set)	OLLAMA_MODEL (server.py), ChatOllama(...) (local_rag.py)
OpenAI model	gpt-6.1-sol	OPENAI_MODEL (server.py)
OpenAI reasoning effort	low, via the Responses API	ChatOpenAI(..., reasoning_effort="low", use_responses_api=True) (local_rag.py)
Realtime model	gpt-realtime-2.1-mini	REALTIME_MODEL (server.py)
Default in the model menu	ChatGPT	selected in chat.html

GPT-6 models accept function tools on Chat Completions only with reasoning_effort="none", and gpt-6.1-sol does not support none at all. Tool calls therefore go through the Responses API.

Retrieval – module constants in local_rag.py

Constant	Current	Meaning
RETRIEVAL_K	5	chunks per search handed to the LLM
RETRIEVAL_MODE	hybrid	dense (vectors only), hybrid (vectors + BM25, RRF), union (BM25 adds candidates only)
CANDIDATES_K	25	candidates before reranking; 25 was faster than 50 and not worse on either goldset
RERANKER_MODEL	BAAI/bge-reranker-v2-m3	None disables reranking; runs in fp16 on the GPU (MPS/CUDA), fp32 on the CPU
MAX_TOOL_ROUNDS	3	searches per message before the LLM must answer
SEARCH_TOOL_NAME / SEARCH_TOOL_DESCRIPTION	search_kafka_corpus / text	tool name, default tool description
QUERY_ADDITIONAL_INSTRUCTIONS	empty	text appended to the search query; it lowered recall, so it is left empty

Index – rebuild required after changing these

Constant	Current
EMBEDDING_MODEL / EMBEDDING_REVISION	BAAI/bge-m3, safetensors revision (normalized vectors)
CHUNK_SIZE / CHUNK_OVERLAP	2000 / 400 characters
PDF_LOADER	pypdf; docling was not better in the goldset comparison
CHUNK_HEADERS	False

Each index stores its settings in index_info.json. LocalRAG refuses to load an index built with a different embedding model.

---

Build the vector stores

add_pdfs.py indexes every PDF under a folder, including subfolders. With CLEAR_VECTORSTORE_FIRST = True (the default in the script), the target folder is deleted and rebuilt; set it to False to merge into an existing index.

Whole corpus, used by the researcher:

python add_pdfs.py --data data --index ./vectorstore_pypdf_2000 --loader pypdf --no-headers

Primary text only, used by the contemporary and Kafka. data_primary/ contains just a link to data/Kafka_Die_Verwandlung_.pdf:

python add_pdfs.py --data data_primary --index ./vectorstore_primary_pypdf_2000 --loader pypdf --no-headers

Other options: --loader docling (needs prepare_corpus.py output, --processed), --chunk-size, --chunk-overlap.

• Docling pipeline (arm64 env only): python prepare_corpus.py [pdf_dir] writes to corpus_processed/data_split. Options: --only "name1;name2", --no-split, --layout, --no-headings.
• Not in git: the corpus PDFs (data/, data_primary/), vector stores (vectorstore*/), extraction output and .env.

---

Measure retrieval quality

python eval_retrieval.py --goldset eval/goldset_kafka_human.csv --index ./vectorstore_pypdf_2000 --report eval/my_report.txt

• Options: --mode dense|hybrid|union, --no-rerank, --judge MODEL.
• --judge lets an OpenAI model check whether the answer is in the top 5. It needs the key; verdicts are cached in eval/judge_cache.json.
• Current numbers (whole corpus, hybrid, 25 candidates, fp16 reranker):

Goldset	R@5	MRR
human (30 questions)	0.73	0.68
synthetic (43 questions)	0.98	0.79

---

API

Endpoint	Purpose
POST /api/chat	Body: message, history (list of {role, content}), persona (researcher, contemporary, kafka), llm_provider (ollama, openai, realtime; API default ollama)
GET /api/personas	default persona and the public fields of all personas
GET /api/providers	which providers are available (openai and realtime need a key at startup)
GET /api/health	server status and whether all vector stores are loaded

Response of POST /api/chat

Field	Meaning
answer	the answer (empty on error)
error	error message or null
retrieved	chunks the LLM saw: content, source, page, score (reranker relevance, higher is better)
searches	the queries the LLM sent to the search tool (empty if it did not search)

---

Debugging and timing

The server terminal prints, per question:

[RAG] User question: '…'
[Zeit] LLM-Aufruf 1: 6.86s (Suche angefordert)
[RAG] Tool search '…': 5 chunk(s)
[Zeit] Suche 1.37s (davon Reranker 1.21s)
[Zeit] LLM-Aufruf 2: 15.31s (Antwort)
[Zeit] Gesamt 23.54s

After that it prints the retrieved chunks (source, page, score, preview) and the full answer, with provider, model and persona.

In the browser console, each answer logs a collapsed group [RAG] Retrieved N chunk(s) with the full chunk texts.

Where to look when an answer is wrong:

• Wrong chunks: tune retrieval, i.e. CANDIDATES_K, RETRIEVAL_MODE or the reranker, and measure with eval_retrieval.py.
• Right chunks but a bad answer, or searching when it should not (or the reverse): edit the persona's system_prompt or tool_description.

To try retrieval without the server:

from local_rag import LocalRAG
rag = LocalRAG(vectorstore_path="./vectorstore_primary_pypdf_2000")
for doc, score in rag.retrieve("Wo versteckt sich Gregor?", k=5):
    print(doc.metadata.get("page"), round(score, 3), doc.page_content[:80])

python local_rag.py still runs an old demo against ./vectorstore and is not maintained.