From Lab to Production: Securing Local LLMs and AI Agents with Self-Hosted Infrastructure and GitOps Guardrails

Originally published on tamiz.pro. You've fine-tuned your model. It runs on your GPU server. Prompt injections are a lab curiosity. But the moment you expose it to real users - with tools, memory, and agent loops - the attack surface explodes. Local doesn't mean secure. In this deep dive, I'll walk you through the architecture, guardrails, and operational practices that turn a fragile lab prototype into a production-hardened self-hosted LLM system protected by GitOps. Why "Local" Doesn't Mean Secure Self-hosted LLMs offer privacy, cost control, and compliance advantages. But they also inherit every risk of any production system - plus new ones unique to generative AI. A local model serving over HTTP is just an API server, and API servers get probed, exploited, and exfiltrated. AI agents compound this with tool access, persistent memory, and autonomous action loops. The lab mindset treats these as features. Production treats them as threat vectors. Bridging that gap requires infrastructure that is auditable, version-controlled, and relentlessly automated - exactly what GitOps delivers. Architecture Overview A production self-hosted LLM stack has four logical layers, each with its own security responsibilities: ┌─────────────────────────────────────────────────┐ │ GitOps Control Plane │ │ (ArgoCD / Flux + OPA/Gatekeeper + SealedSeals) │ ├─────────────────────────────────────────────────┤ │ Model Serving Layer │ │ (vLLM / TGI / llama.cpp + GPU node pools) │ ├─────────────────────────────────────────────────┤ │ Agent & Tool Execution Layer │ │ (Sandboxed containers + policy enforcement) │ ├─────────────────────────────────────────────────┤ │ Data & Memory Layer │ │ (Encrypted stores + RAG pipeline + audit logs) │ └─────────────────────────────────────────────────┘ Every layer is defined as code. Every deployment flows through a pull-request gate. Every runtime decision is policy-enforced, not opinion-enforced. 1. The GitOps Control Plane GitOps is the backbone. Instead of imperatively applying Kubernetes manifests or Helm charts, you declare desired state in a Git repository and a controller (ArgoCD or Flux) reconciles the cluster to match. Core components: - ArgoCD for application lifecycle management, with application sets for multi-model or multi-environment deployments - OPA/Gatekeeper as the admission controller, enforcing policies like "no image from untrusted registries" or "GPU nodes must have nvidia.com/gpu requests validated" - SealedSecrets or External Secrets Operator for managing sensitive values (API keys, model weights encryption keys) without committing plaintext to Git - Kyverno as a complementary policy engine for more complex rule logic that OPA's Rego can't express comfortably Your Git repo structure should mirror your environment topology: infra/ clusters/ prod/ argocd/apps.yaml kyverno/policies.yaml sealed-secrets/ applications/ llm-serving/ values.yaml kustomization.yaml agent-runtime/ values.yaml kustomization.yaml rag-pipeline/ values.yaml kustomization.yaml policies/ network-policy.yaml pod-security.yaml image-policy.yaml This structure makes it trivial to audit: a single git log --oneline across the infra/ tree tells you exactly what changed, when, and by whom. 2. The Model Serving Layer Your model server is the most exposed component. It handles inbound traffic, loads weights into GPU memory, and streams outputs. Compromise here means full model theft, data poisoning, or worst-case - uncontrolled inference that drains your cluster. Hardening steps: Network isolation. Never expose the inference endpoint directly to the internet. Use a service mesh (Istio or Linkerd) with mutual TLS between all in-cluster components, and an ingress controller with WAF rules that strip or reject malformed prompt payloads. Resource limits as attack surface reduction. GPU nodes are expensive and targeted. Set strict limits and requests on every pod. Use Kubernetes LimitRanges and ResourceQuotas to prevent a single compromised pod from consuming the entire node pool. Image supply chain. Sign your serving images with Cosign and verify signatures at admission via Gatekeeper. Combine with in-toto attestation for build provenance. # Example: Gatekeeper constraint for image signature verification apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sRequiredAnnotations metadata: name: require-cosign-signature spec: match: kinds: - apiGroups: ["apps"] kinds: ["Deployment"] parameters: annotations: - "cosign.sigstore.dev/signed" Model weight protection. Weights are intellectual property. Encrypt them at rest using a KMS-integrated CSI driver like secrets-store-csi-driver with AWS KMS, GCP KMS, or HashiCorp Vault. Stream weights into the container at runtime only - never persist them on node local storage. 3. The Agent & Tool Execution Layer AI agents are the new attack surface frontier. Unlike static models, agents compose themselves at runtime - selecting tools, making API calls, writing files, executing commands. Each tool invocation is a potential privilege escalation path. Sandboxing is non-negotiable. Every agent execution should run in an isolated container or Pod with: - Read-only filesystems where possible - No privileged containers ( allowPrivilegeEscalation: false ) - Network policies restricting egress to only the tools' required endpoints - Dropped capabilities - remove all Linux capabilities except what's strictly necessary # Example: Strict PodSecurityProfile for agent containers apiVersion: v1 kind: Pod metadata: name: agent-execution spec: securityContext: runAsNonRoot: true runAsUser: 65534 fsGroup: 65534 containers: - name: agent image: registry.internal/agent-runtime:v2.3.1 securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] readOnlyRootFilesystem: true volumeMounts: - name: tmp mountPath: /tmp - name: agent-workdir mountPath: /home/agent/work readOnly: false volumes: - name: tmp emptyDir: sizeLimit: 100Mi - name: agent-workdir emptyDir: sizeLimit: 500Mi Tool policy enforcement. Define which tools each agent identity can access via a policy layer (not hardcoded). Use Open Policy Agent to evaluate tool requests before they reach the execution sandbox. # Example: OPA policy restricting agent tool access package agent.tool_policy denying_tool[msg] { input.agent_id == "read_write_agent" input.tool == "execute_command" msg := "read_write_agent cannot execute arbitrary commands" } denying_tool[msg] { input.tool == "write_file" not startswith(input.file_path, "/home/agent/work") msg := sprintf("write_file denied: path %s is outside allowed directory", [input.file_path]) } denying_tool[msg] { input.tool == "http_request" not startswith(input.url, "https://") msg := "http_request denied: only HTTPS URLs are permitted" } Guardrails for model output. Beyond tool policies, enforce guardrails on what the model itself produces. Use frameworks like NeMo Guardrails or Guardrails AI to detect and block: - Prompt injection attempts - PII leakage in outputs - Dangerous or disallowed content - Excessive tool use (budget enforcement) 4. The Data & Memory Layer RAG pipelines and agent memory (persistent conversations, tool outputs, external data lookups) are high-value targets. A compromised vector store or embedding model gives attackers indirect access to your proprietary data. Encryption everywhere. Encrypt data at rest (AES-256 via your cloud KMS or Vault) and in transit (TLS 1.3 minimum). Use field-level encryption for particularly sensitive documents before they enter your vector store. Access controls per document. Don't rely on a single vector store credential. Implement row-level or document-level access control by tagging embeddings with metadata that maps to RBAC policies, then filtering at query time. Audit logging. Every embedding operation, retrieval query, and memory read/write should be logged to an immutable audit trail. Ship these logs to a separate, access-restricted SIEM or logging backend - never store them in the same cluster. GitOps Workflows for LLM Systems The architecture above is only as strong as the workflow that maintains it. GitOps isn't just about declarative deployments - it's about creating a verifiable chain of custody for every component in your LLM stack. The Pull Request Gate Every change to your infrastructure or model artifacts goes through a PR. This isn't ceremony; it's your primary security control. PR checklist for LLM infrastructure changes: - Diff review. At least one other engineer must approve the change. For model-related changes, include someone who understands the threat model of that specific component. - Policy validation. Run kyverno test orconftest against your manifests before merging. Catch violations early. - Image verification. Ensure the referenced image tag exists in your signed registry and has a valid Cosign signature. - Secrets audit. Confirm no plaintext secrets appear in the diff. If new secrets are needed, they go through the SealedSecrets or External Secrets flow. - Rollback plan. Document the exact steps to revert if the change fails in production. Automated Reconciliation and Drift Detection ArgoCD or Flux continuously reconciles your cluster state against the Git repository. But reconciliation alone isn't enough - you need visibility into drift. Configure alerts for: - Out-of-sync applications - a pod spec was modified directly on the cluster without going through Git - Health degradation - readiness probes failing on the serving layer - Policy violations - Kyverno or Gatekeeper flags detected - Failed syncs - automated rollbacks triggered Set up a daily compliance report that summarizes the state of your cluster relative to Git: # ArgoCD Application for automated compliance scanning apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: compliance-scanner spec: source: repoURL: https://git.internal/security-policies.git targetRevision: main path: scans/daily destination: server: https://kubernetes.default.svc