Notes on Local LLM RAG in Practice

A full practical path from model selection to retrieval tuning.

Why run it locally

Many enterprise scenarios have hard privacy requirements — the knowledge base often contains internal material that must never leave the intranet. Deploying the model locally, so the data never leaves the network, is a non-negotiable premise for these cases.

The key stages

1. Model selection: balancing inference speed, VRAM footprint, and answer quality — I settled on a quantized mid-sized model.
2. Retrieval quality: RAG’s ceiling is largely set by retrieval. Adding a rerank layer after vector recall noticeably improves the ranking of relevant passages.
3. Context assembly: when stitching retrieved chunks into the prompt, order, deduplication, and length control all shape the final answer.

Pitfalls

The most overlooked part is the chunking strategy — how you split documents directly affects retrieval granularity. Split too fine and you lose context; too coarse and you introduce noise. There is no magic parameter; you tune it against your actual documents.