Direct Providers vs Portable Contracts - Ask-Your-Docs Semantic Search for SaaS RAG

Short answer: for a property-management SaaS that must turn support tickets into structured, cited answers, use a portable model contract for embeddings and chat completions, keep retrieval in the application, and add reranking only after an evaluation shows that first-pass semantic search is losing relevant passages. The least complex useful version is small: chunk approved support documents, generate embeddings for those chunks and the incoming question, retrieve the nearest matches, and ask a chat model to answer only from those passages. The application, not the model, owns the citation IDs and the final schema. That boundary matters more than an elaborate orchestration framework because a triage result can route a tenant's urgent maintenance report, expose account data to an agent, or become part of an audit trail. I would choose a stable gateway contract once a team expects to change the model or vendor behind any of those steps. Direct provider calls remain the better choice when one provider is an intentional compliance dependency and portability has no operational value. What should a SaaS semantic search RAG pipeline require from embeddings and chat completions? Start with the output contract. A property-support triage record needs, at minimum, a category, a confidence value, an escalation decision, and citations whose identifiers can be resolved back to immutable document versions. The model may propose those values, but the backend must validate the structure, reject unknown categories, and confirm that every cited identifier came from the retrieved set. A fluent answer with a fabricated citation is a failed transaction. Exactly once is an aspiration here, not a property supplied by an LLM call. Give each ticket revision a deterministic processing key, record the document corpus version, query hash, retrieved chunk IDs, model selection, validated result, and request ID, then make the database write idempotent. If a worker receives the same ticket revision twice, it should return the committed triage record rather than creating two agent tasks. This is the same discipline used around ledger postings: retries are expected; duplicate effects are not. The compliance boundary is equally concrete. Retrieved passages and prompts can contain tenant names, addresses, access instructions, and payment disputes. Confirm the chosen vendor's regional processing, retention, subprocessors, and contractual controls against the jurisdictions you serve; a technically correct JSON response does not establish GDPR, US state privacy, or sector-specific compliance. I'm not sure any generic vendor matrix can settle that question because the answer depends on the customer's contract and data map. Legal and security review must resolve it. Keep the schema narrow. For example, category can be an enum such as maintenance , billing , lease , or other ; needs_human is boolean; and citations is an array of chunk IDs. Don't ask the model to invent workflow actions. The application maps a validated category to an approved queue, while low-confidence or policy-sensitive cases go to a person. Consider the no-heat ticket used below. Revision 7 enters the queue with processing key ticket-1842-r7 ; retrieval is constrained to the building, tenant authorization, and current maintenance policy revision before similarity ranking begins. The first cited passage says that no heat is urgent and requires human dispatch, so the model can propose maintenance , needs_human: true , and that passage's ID. The backend still has several jobs: it verifies that the category belongs to the enum, checks that the confidence is in range, proves that the citation was in the retrieved set, and commits the proposal only if no record already exists for ticket-1842-r7 . If a worker retries after a 429, the inference may run again, but the queue assignment does not multiply. If the ticket changes to say the heat has returned, revision 8 receives a different key and a separately traceable result. This example is intentionally mundane. Auditability is the chain of ordinary identifiers that lets an operator reconstruct why one version of one ticket reached one queue; it isn't a promise that probabilistic inference became exactly once. Retrieval quality comes before model fluency Embeddings make document chunks and ticket questions comparable, but chunk identity and versioning make the result auditable. Store each vector beside a stable chunk ID, source document ID, revision, jurisdiction, access scope, and the exact text used to generate it. A property manager in Berlin must not retrieve a California lease rule merely because the wording is close. Filter by authorization and applicable corpus before ranking by similarity. Then test retrieval independently from answer generation. Build a modest evaluation set from representative, de-identified questions and label the passages that contain the answer. Measure whether those passages appear in the initial candidate set. If they do, but their order is poor, insert reranking after vector retrieval. If they never appear, reranking cannot rescue the pipeline; revisit chunking, metadata filters, or embedding choice. Stop there for now. Small and medium document sets are where an optional reranker is particularly easy to justify: retrieve a wider candidate set cheaply, rerank that bounded list, and pass only the strongest passages into chat completions. The catch is added latency and another model decision to log. A team should keep plain vector retrieval when its labeled questions already meet the target recall and the latency budget is tight. Token counting belongs before the answer request, not after a surprising invoice or a rejected context. Count during chunking and again while assembling the prompt; reserve room for the response, and remove the lowest-ranked passages deterministically when the prompt exceeds the selected model's limit. Cost estimates should be attached to the same audit record, although model catalogs and current rates must be treated as changing operational data rather than constants embedded in application code. A minimal, auditable Go path The following program indexes three example policy passages in memory, embeds an incoming ticket, retrieves two matches, and requests a grounded JSON answer. It uses two verified OpenAI-compatible routes, reads credentials and model IDs from environment variables, sets every HTTP method explicitly, and backs off on HTTP 429 while honoring Retry-After . A production vector store replaces the in-memory slice; the contract around chunk IDs stays the same. package main import ( "bytes" "context" "encoding/json" "fmt" "io" "math" "net/http" "os" "sort" "strconv" "strings" "time" ) type chunk struct { ID string Text string Vec []float64 } type embeddingResponse struct { Data []struct { Embedding []float64 json:"embedding" } json:"data" } type chatResponse struct { Choices []struct { Message struct { Content string json:"content" } json:"message" } json:"choices" } func post(ctx context.Context, client *http.Client, base, key, path string, body any, out any) error { payload, err := json.Marshal(body) if err != nil { return err } for attempt := 0; attempt 0 { delay = time.Duration(seconds) * time.Second } select { case = 300 { return fmt.Errorf("request failed with status %d: %s", resp.StatusCode, strings.TrimSpace(string(data))) } return json.Unmarshal(data, out) } return fmt.Errorf("rate limit retry budget exhausted") } func embed(ctx context.Context, client http.Client, base, key, model string, texts []string) ([][]float64, error) { var result embeddingResponse err := post(ctx, client, base, key, "/embeddings", map[string]any{ "model": model, "input": texts, }, &result) if err != nil { return nil, err } vectors := make([][]float64, len(result.Data)) for i, item := range result.Data { vectors[i] = item.Embedding } return vectors, nil } func cosine(a, b []float64) float64 { var dot, aa, bb float64 for i := range a { dot += a[i] * b[i] aa += a[i] * a[i] bb += b[i] * b[i] } if aa == 0 || bb == 0 { return 0 } return dot / (math.Sqrt(aa) * math.Sqrt(bb)) } func main() { key := os.Getenv("INFRAI_API_KEY") apiBase := strings.TrimRight(os.Getenv("AI_API_BASE"), "/") embeddingModel := os.Getenv("EMBEDDING_MODEL_ID") chatModel := os.Getenv("CHAT_MODEL_ID") if key == "" || apiBase == "" || embeddingModel == "" || chatModel == "" { panic("set AI_API_BASE, INFRAI_API_KEY, EMBEDDING_MODEL_ID, and CHAT_MODEL_ID") } docs := []chunk{ {ID: "policy-maint-07", Text: "No heat is an urgent maintenance category and requires human dispatch."}, {ID: "policy-billing-03", Text: "A duplicate rent charge is routed to billing review; do not promise a refund."}, {ID: "policy-access-04", Text: "Entry instructions may be shared only with the assigned maintenance team."}, } question := "My apartment has had no heat since last night. What happens next?" texts := make([]string, len(docs)) for i := range docs { texts[i] = docs[i].Text } ctx, cancel := context.WithTimeout(context.Background(), 45time.Second) defer cancel() client := &http.Client{Timeout: 30 * time.Second} docVectors, err := embed(ctx, client, apiBase, key, embeddingModel, texts) if err != nil { panic(err) } queryVectors, err := embed(ctx, client, apiBase, key, embeddingModel, []string{question}) if err != nil { panic(err) } for i := range docs { docs[i].Vec = docVectors[i] } sort.Slice(docs, func(i, j int) bool { return cosine(docs[i].Vec, queryVectors[0]) > cosine(docs[j].Vec, queryVectors[0]) }) contextText := fmt.Sprintf("[%s] %s\n[%s] %s", docs[0].ID, docs[0].Text, docs[1].ID, docs[1].Text) request := map[string]any{ "model": chatModel, "messages": []map[string]string{ {"role": "system", "content": "Answer only from the supplied passages. Return JSON with category, confidence, needs_human, and citations. Citations must use supplied bracketed IDs."}, {"role": "user", "content": "Passages:\n" + contextText + "\n\nTicket:\n"