From Documents to Intelligent Answers: Building a RAG Agent from Scratch & Lessons Learned

What is RAG?

Retrieval-Augmented Generation combines two capabilities:

• Retrieval: The system searches a knowledge base and finds relevant information related to the user's question.
• Generation: The retrieved information is provided as context to an LLM, which generates a response based on that knowledge.

Instead of asking an LLM to remember everything, RAG allows the model to access external knowledge dynamically.

Technologies Used

Python, LangChain, Ollama (Local LLM), Embeddings, Vector Database, FastAPI.

What I Built

The core idea is to build a system that answers questions using only the content of a document, rather than whatever the underlying model already "knows."

Documents → Text Splitter → Embeddings → Vector Store → Retriever → LLM → Answer

1. Document loading - a company policy document, loaded with LangChain's TextLoader
2. Text Splitting - split into 200-character chunks with 50-character overlap, using RecursiveCharacterTextSplitter
3. Create Embeddings - generated with sentence-transformers/all-MiniLM-L6-v2 via HuggingFaceEmbeddings
4. Vector store - persisted in Chroma
5. Retrieval - top-2 most relevant chunks pulled per question
6. Generation - a strict prompt template that instructs the model to answer only from the retrieved context, run through Ollama's tinyllama

Lessons Learned

Building a RAG system taught me that successful AI applications are not only about selecting a powerful LLM. The quality of the final answer depends heavily on:

• Document quality
• Chunking strategy
• Retrieval accuracy
• Prompt design
• Evaluation methods

Conclusion

RAG provides a practical foundation for building enterprise AI assistants that can use private knowledge while reducing hallucination risks. This project became the foundation for my next experiments with multi-agent workflows, MCP servers, and autonomous AI systems.

Source Code: https://github.com/srirdeevi/agentic-ai-portfolio