RAG in Laravel: Embeddings and pgvector for a Knowledge-Base Bot

RAG in Laravel: Embeddings and pgvector for a Knowledge-Base Bot

In the last post we streamed AI responses over SSE. Now let's fix the problem every chatbot hits in week one: the model doesn't know your data. Ask it about your refund policy or your API docs and it either hallucinates or shrugs.

RAG - retrieval-augmented generation - fixes that. Instead of hoping the model knows your content, you store your documents as embeddings, retrieve the few chunks relevant to each question, and hand only those to the model. Here's the whole pipeline in Laravel 11 with PostgreSQL and pgvector - no vector-database SaaS required.

Do You Even Need RAG?

First, an honest gate. I recently shipped a chatbot for my own portfolio site. Total knowledge base: about 2,000 tokens of facts and FAQ. I didn't build RAG for it - I put the entire corpus in the system prompt and let prompt caching make it cheap. Simpler code, zero retrieval bugs, and the model sees everything on every question.

The rule of thumb I use with clients:

• Corpus under ~20k tokens → whole corpus in a cached system prompt. Done.
• Corpus larger than that, or updated constantly → RAG.

If you're in the first bucket, close this tab and go ship. Still here? Let's build.

The Stack

• Laravel 11, PostgreSQL 15+ with the pgvector extension
• openai-php/laravel for embeddings + chat
• pgvector/pgvector-php for the Eloquent cast

composer require openai-php/laravel pgvector/pgvector
php artisan vendor:publish --provider="OpenAI\Laravel\ServiceProvider"

On most managed Postgres (RDS, Supabase, Neon, Laravel Forge boxes) pgvector is one statement away:

CREATE EXTENSION IF NOT EXISTS vector;

The Schema

One table stores chunks of your documents plus their embedding vectors. text-embedding-3-small produces 1536 dimensions - good quality, cheap ($0.02 per million tokens).

use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Schema;

return new class extends Migration {
    public function up(): void {
        DB::statement('CREATE EXTENSION IF NOT EXISTS vector');
        Schema::create('document_chunks', function ($table) {
            $table->id();
            $table->string('source'); // e.g. "refund-policy.md"
            $table->unsignedInteger('position'); // chunk order within the source
            $table->text('content');
            $table->timestamps();
        });
        DB::statement('ALTER TABLE document_chunks ADD COLUMN embedding vector(1536)');
        // HNSW index makes similarity search fast past ~10k rows
        DB::statement('CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops)');
    }
};

The Model

namespace App\Models;

use Illuminate\Database\Eloquent\Model;
use Pgvector\Laravel\Vector;

class DocumentChunk extends Model {
    protected $fillable = ['source', 'position', 'content', 'embedding'];
    protected $casts = ['embedding' => Vector::class];
}

Ingestion: Chunk, Embed, Store

Chunking strategy matters more than people admit. My default: split on headings/paragraphs, target ~500 tokens per chunk, and never split mid-sentence. Fancy overlap windows can wait until you have evidence you need them.

namespace App\Console\Commands;

use App\Models\DocumentChunk;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\File;
use OpenAI\Laravel\Facades\OpenAI;

class IngestDocs extends Command {
    protected $signature = 'rag:ingest {path : Directory of markdown files}';

    public function handle(): int {
        foreach (File::files($this->argument('path')) as $file) {
            $chunks = $this->chunk($file->getContents());
            // One API call per file, not per chunk - batch input is supported
            $response = OpenAI::embeddings()->create([
                'model' => 'text-embedding-3-small',
                'input' => $chunks,
            ]);
            DocumentChunk::where('source', $file->getFilename())->delete();
            foreach ($response->embeddings as $i => $embedding) {
                DocumentChunk::create([
                    'source' => $file->getFilename(),
                    'position' => $i,
                    'content' => $chunks[$i],
                    'embedding' => $embedding->embedding,
                ]);
            }
            $this->info("{$file->getFilename()}: " . count($chunks) . ' chunks');
        }
        return self::SUCCESS;
    }

    /** @return string[] */
    private function chunk(string $text, int $targetChars = 2000): array {
        $paragraphs = preg_split('/\n{2,}/', $text);
        $chunks = [''];
        foreach ($paragraphs as $p) {
            $current = array_key_last($chunks);
            if (strlen($chunks[$current]) + strlen($p) > $targetChars && $chunks[$current] !== '') {
                $chunks[] = $p;
            } else {
                $chunks[$current] .= "\n\n" . $p;
            }
        }
        return array_map('trim', $chunks);
    }
}

Run it: php artisan rag:ingest storage/docs. Re-running replaces a file's chunks, so ingestion is idempotent - wire it to your deploy or a scheduled job and content stays fresh.

Retrieval: The Five-Line Heart of RAG

pgvector's <=> operator is cosine distance. Lower = more similar. Embed the user's question, sort by distance, take the top handful:

namespace App\Services;

use App\Models\DocumentChunk;
use OpenAI\Laravel\Facades\OpenAI;

class Retriever {
    /** @return array{content: string, source: string}[] */
    public function search(string $question, int $limit = 5): array {
        $embedding = OpenAI::embeddings()->create([
            'model' => 'text-embedding-3-small',
            'input' => $question,
        ])->embeddings[0]->embedding;

        return DocumentChunk::query()
            ->selectRaw('content, source, embedding <=> ? AS distance', [json_encode($embedding)])
            ->orderBy('distance')
            ->limit($limit)
            ->get()
            ->filter(fn ($c) => $c->distance < 0.55) // relevance floor - tune on real queries
            ->map(fn ($c) => ['content' => $c->content, 'source' => $c->source])
            ->all();
    }
}

That distance floor is your hallucination guard. If nothing scores under the threshold, the honest answer is "I don't know" - and you should say exactly that instead of feeding the model junk context.

Answering with Context

namespace App\Services;

use OpenAI\Laravel\Facades\OpenAI;

class KnowledgeBot {
    public function __construct(private Retriever $retriever) {}

    public function answer(string $question): string {
        $chunks = $this->retriever->search($question);

        if ($chunks === []) {
            return "I couldn't find that in our documentation - try rephrasing, or contact support.";
        }

        $context = collect($chunks)
            ->map(fn ($c) => "[{$c['source']}]\n{$c['content']}")
            ->implode("\n\n---\n\n");

        $response = OpenAI::chat()->create([
            'model' => 'gpt-4o-mini',
            'max_tokens' => 500,
            'messages' => [
                [
                    'role' => 'system',
                    'content' => <<<PROMPT
Answer using ONLY the context below. If the context doesn't contain the answer, say you don't know - never invent facts. Cite the source file name when you answer.

Context:
{$context}
PROMPT
                ],
                ['role' => 'user', 'content' => $question],
            ],
        ]);

        return $response->choices[0]->message->content;
    }
}

Wire KnowledgeBot::answer() into the streaming controller from part 2 of this series and you have a knowledge-base bot that streams grounded answers.

Production Checklist

• Relevance floor: filter retrieved chunks by distance threshold; return an honest "not found" below it. Tune the number against real user queries, not vibes.
• Idempotent ingestion: delete-and-replace per source file, run on deploy. Stale embeddings answer yesterday's questions.
• HNSW index: without it, every query is a full table scan. With it, sub-10ms at hundreds of thousands of rows.
• Batch embeddings: one API call per document, not per chunk - 10x fewer requests and the same price.
• Cost caps and rate limits: everything from part 1 still applies - each question costs one embedding call plus one chat call.
• Log question + retrieved sources (not full content): when an answer is wrong, you need to know whether retrieval or generation failed. That distinction decides your fix.
• Evaluate retrieval separately: keep 20 known question→source pairs and assert the right chunk lands in the top 5. This catches chunking regressions before your users do.

What's Next

Retrieval makes the model knowledgeable. The next step makes it capable: in part 4 we'll build AI agents in PHP - giving the model tools it can call (search orders, create tickets, query your database) and handling the tool-calling loop safely in Laravel. That's where chatbots turn into products.

I'm Aditya Kumar (adityakdevin) - Tech Lead & full-stack developer building AI-powered web products with Laravel, Vue, and LLM APIs. Find me at adityadev.in.