Offline RAG on four budget PCs: FAISS, llama.cpp and Qwen2.5 on a LAN

Retrieval-augmented generation (RAG) is usually shown with a cloud LLM and a hosted vector database. I wanted the opposite for college use: a private document assistant that works with zero internet dependency after setup, on hardware that was already lying around.

The result is sret-rag. You upload PDF, DOCX or text files, ask questions, and get cited answers, all on the local network.

The system is spread across four plain Linux PCs (Intel i3, 8 GB RAM, 200 GB HDD), each with a single responsibility:

• PC1: FastAPI gateway and a ChatGPT-style web UI.
• PC2: a dedicated embedder running all-MiniLM-L6-v2 (384-dimensional vectors).
• PC3: the FAISS index (IVF50,PQ8) with SQLite for metadata.
• PC4: the LLM server, llama.cpp with Qwen2.5 7B quantised to Q4_K_M.

No GPU is required. llama.cpp uses the Intel Iris Xe integrated GPU through SYCL. Splitting the roles keeps each 8 GB machine from running out of memory, and a 3 GB lz4 zram swap on every node stretches the RAM further.

Documents are parsed with PyMuPDF, split into 512-token chunks at semantic boundaries, embedded with MiniLM and stored in FAISS. At query time the gateway embeds the question, retrieves the nearest chunks, builds a prompt and sends it to the llama.cpp node. The answer comes back with its sources.

Two small caches avoid repeated work: a SHA-256-keyed SQLite embedding cache means duplicate chunks are never re-embedded, and hashing each file skips documents that are already indexed.

Everything is async (FastAPI, aiosqlite, asyncio), with the C-extension calls pushed to a thread pool so the gateway stays responsive. The gateway exposes an OpenAI-compatible /v1/chat/completions endpoint with streaming, so existing clients can point at it. Auth is session-based with an admin panel for user management. Setup is done from offline wheel and package folders copied from a USB drive.

• all-MiniLM-L6-v2: small enough to run on a CPU, and its 384-dimensional vectors keep the index compact.
• FAISS IVF50,PQ8 with nprobe=5: the inverted file splits vectors into 50 clusters and searches only the 5 nearest, while product quantisation compresses each vector to 8 bytes, so the index fits easily in 8 GB of RAM.
• llama.cpp with a Q4_K_M quantised model: 4-bit weights let a 7B model run on an i3 with integrated graphics instead of a dedicated GPU.
• semantic-text-splitter at 512 tokens: chunks break at sentence and paragraph boundaries, so a retrieved chunk reads as a complete thought.
• SQLite through aiosqlite: metadata, users and the embedding cache need no separate database server.

The UI can also switch at runtime to smaller models (phi-2, Qwen2.5-1.5B, TinyLlama-1.1B) when a faster, lighter answer is enough.

• Four PCs with an Intel i3 (11th gen recommended), 8 GB RAM and a 200 GB HDD.
• Ubuntu or Linux Mint on each machine.
• A LAN with static IPs, here 192.168.1.10 to 192.168.1.13.
• No GPU and no internet connection after setup.

• Copy the project folder, offline Python wheels and Debian packages from a USB drive to each PC.
• Create a Python virtual environment and install with pip --no-index --find-links=./wheels, so nothing is downloaded.
• Run the zram setup script on every PC to add 3 GB of compressed swap.
• Start the LLM server on PC4 first and wait about 45 seconds for the model to load.
• Start the FAISS node (PC3), the embedder (PC2) and finally the gateway (PC1). A start_cluster.sh script does this over SSH.
• Put PDFs in the docs folder, run the processing script to chunk them, then build the FAISS index.
• Open the gateway in a browser on the LAN, log in, and change the default admin password in .env.

The gateway exposes a small REST API, so other tools can use the cluster without the web UI:

• POST /api/v1/ingest/file and /api/v1/ingest/url: add a document, returning its id and chunk count.
• POST /api/v1/query: ask a question, returning the answer, its sources and the latency in milliseconds.
• POST /v1/chat/completions: OpenAI-compatible chat with streaming, so existing OpenAI clients work by changing the base URL.
• GET /api/v1/health: index status and the loaded model.

Start the LLM node first and give it about 45 seconds to load the model; startup order matters. Change the default admin credentials before first use, and keep the system on the LAN unless you add your own protection.

Splitting roles across machines is what makes budget hardware work: no single 8 GB PC has to hold the embedder, the index and a 7B model at once.