Deploying Apache Pulsar as a Self-Hosted Google Pub/Sub Alternative

Apache Pulsar is a distributed messaging/streaming platform combining high-throughput delivery, durable storage, and built-in serverless compute (Pulsar Functions), a self-hosted alternative to Google Pub/Sub with multi-tenancy and tiered storage built in. This guide deploys Pulsar via Docker, sets up multi-tenancy, deploys a Pulsar Function, configures an IO connector, enables JWT auth, and covers migrating off Pub/Sub. Pulsar vs. Google Pub/Sub - Deployment: Pub/Sub is fully managed on GCP. Pulsar runs on any infra you control (bare metal, VMs, Kubernetes). - Multi-tenancy: Pub/Sub uses GCP projects for isolation. Pulsar has native tenants/namespaces/topic-level policies. - Storage: Pub/Sub abstracts storage entirely. Pulsar uses Apache BookKeeper with optional tiered storage to object storage. - Serverless: Pub/Sub integrates with Cloud Functions. Pulsar Functions run inline with the broker, no external system needed. - Pricing: Pub/Sub charges per volume/operation. Pulsar is free beyond infra cost. Concept Mapping | Google Pub/Sub | Apache Pulsar | Description | |---|---|---| | Topics | Topics | Named channels for publishing | | Subscriptions | Subscriptions | Named consumers reading from topics | | Push Subscriptions | Pulsar Functions | Use a Function to forward messages to an HTTP endpoint | | Ordering Keys | Key-Shared Subscriptions | Ordered delivery for same-key messages | | Dead-Letter Topics | Dead-Letter Topics | Stores unprocessable messages | | IAM Policies | Authorization Policies | RBAC for topics/namespaces | | Projects | Tenants/Namespaces | Tenants = project-level isolation; namespaces subdivide a tenant | Architecture: - Brokers - stateless, handle routing/connections, no direct storage - BookKeeper (Bookies) - durable, replicated storage layer - Metadata store - cluster coordination; standalone Docker deployments use RocksDB, not external ZooKeeper - Pulsar Manager - web UI for tenants/namespaces/topics/monitoring - Pulsar Functions - inline serverless stream processing Prerequisites: a Linux server, Docker + Docker Compose, a domain A record (e.g. pulsar.example.com ). Deploy with Docker Standalone mode runs broker, BookKeeper, and metadata store in one container. The broker advertises localhost , so pulsar-admin /pulsar-client commands run via docker exec rather than from outside. $ mkdir -p ~/pulsar/{data,conf,connectors,functions} $ cd ~/pulsar Environment file: $ nano .env PULSAR_DOMAIN=pulsar.example.com ACME_EMAIL=a****@example.com PM_USERNAME=admin PM_PASSWORD=PULSAR_MANAGER_PASSWORD PM_EMAIL=a****@example.com $ chmod 777 data Admin bootstrap script - creates the Pulsar Manager admin account on first run: $ nano init-admin.sh #!/bin/sh set -e CSRF=$(curl -s http://pulsar-manager:7750/pulsar-manager/csrf-token) curl -s \ -H "X-XSRF-TOKEN: $CSRF" \ -H "Cookie: XSRF-TOKEN=$CSRF;" \ -H "Content-Type: application/json" \ -X PUT http://pulsar-manager:7750/pulsar-manager/users/superuser \ -d "{"name":"$PM_USERNAME","password":"$PM_PASSWORD","description":"admin","email":"$PM_EMAIL"}" $ chmod +x init-admin.sh $ sudo usermod -aG docker $USER $ newgrp docker Docker Compose: $ nano docker-compose.yml services: traefik: image: traefik:v3.7.0 container_name: traefik command: - "--providers.docker=true" - "--providers.docker.exposedbydefault=false" - "--entrypoints.web.address=:80" - "--entrypoints.websecure.address=:443" - "--certificatesresolvers.le.acme.email=${ACME_EMAIL}" - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json" - "--certificatesresolvers.le.acme.httpchallenge=true" - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web" ports: - "80:80" - "443:443" volumes: - "/var/run/docker.sock:/var/run/docker.sock:ro" - "./letsencrypt:/letsencrypt" restart: unless-stopped pulsar: image: apachepulsar/pulsar:4.2.2 container_name: pulsar command: bin/pulsar standalone --advertised-address localhost ports: - "6650:6650" - "8080:8080" volumes: - ./data:/pulsar/data - ./connectors:/pulsar/connectors - ./functions:/pulsar/functions environment: - PULSAR_MEM=-Xms2g -Xmx2g -XX:MaxDirectMemorySize=1g restart: unless-stopped pulsar-manager: image: apachepulsar/pulsar-manager:v0.4.0 container_name: pulsar-manager environment: - SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties depends_on: - pulsar healthcheck: test: ["CMD-SHELL", "wget -qO- http://localhost:7750/pulsar-manager/csrf-token >/dev/null 2>&1"] interval: 10s timeout: 5s retries: 12 start_period: 45s labels: - "traefik.enable=true" - "traefik.http.services.pulsar-manager.loadbalancer.server.port=9527" - "traefik.http.routers.pulsar-manager.rule=Host(${PULSAR_DOMAIN})" - "traefik.http.routers.pulsar-manager.entrypoints=websecure" - "traefik.http.routers.pulsar-manager.tls=true" - "traefik.http.routers.pulsar-manager.tls.certresolver=le" restart: unless-stopped pulsar-manager-init: image: curlimages/curl:8.19.0 container_name: pulsar-manager-init depends_on: pulsar-manager: condition: service_healthy restart: "no" volumes: - ./init-admin.sh:/init-admin.sh:ro entrypoint: ["sh", "/init-admin.sh"] environment: - PM_USERNAME - PM_PASSWORD - PM_EMAIL $ docker compose up -d $ docker compose ps $ docker compose logs pulsar $ docker logs pulsar-manager-init pulsar-manager-init should print {"message":"Add super user success, please login"} and then exit - it's a one-shot job. Access Pulsar Manager Pulsar pre-creates two tenants: public (default namespace) and pulsar (internal system topics). - Visit https://pulsar.example.com , log in. - New Environment: name local , Service URLhttp://pulsar:8080 , Bookie URLhttp://pulsar:6650 , Confirm. - Click into the environment - sidebar shows Tenants, Namespaces, Topics, Tokens. Configure Multi-Tenancy $ docker exec -it pulsar bin/pulsar-admin tenants create my-tenant \ --admin-roles admin \ --allowed-clusters standalone $ docker exec -it pulsar bin/pulsar-admin namespaces create my-tenant/production $ docker exec -it pulsar bin/pulsar-admin namespaces set-retention my-tenant/production \ --size 10G \ --time 7d $ docker exec -it pulsar bin/pulsar-admin namespaces set-message-ttl my-tenant/production \ --messageTTL 3600 $ docker exec -it pulsar bin/pulsar-admin topics create persistent://my-tenant/production/orders $ docker exec -it pulsar bin/pulsar-admin topics list my-tenant/production Check the NAMESPACES tab under my-tenant in Pulsar Manager to confirm. Deploy a Pulsar Function Transform/filter/route messages inline, no external stream processor needed. $ nano ~/pulsar/functions/uppercase_function.py from pulsar import Function class UppercaseFunction(Function): def process(self, input, context): return input.upper() $ docker exec -it pulsar bin/pulsar-admin functions create \ --name uppercase \ --tenant public \ --namespace default \ --inputs persistent://public/default/input-topic \ --output persistent://public/default/output-topic \ --py /pulsar/functions/uppercase_function.py \ --classname uppercase_function.UppercaseFunction $ docker exec -it pulsar bin/pulsar-admin functions status \ --tenant public \ --namespace default \ --name uppercase Test it: $ docker exec -it pulsar bin/pulsar-client produce persistent://public/default/input-topic \ --messages "hello world" $ docker exec -it pulsar bin/pulsar-client consume persistent://public/default/output-topic \ --subscription-name test-sub \ --num-messages 1 \ --subscription-position Earliest Output should show content:HELLO WORLD . Set Up an IO Connector The standard image ships without connector NARs - download what you need. This demos the file source connector. $ curl -L -o ~/pulsar/connectors/pulsar-io-file-4.2.2.nar \ https://downloads.apache.org/pulsar/pulsar-4.2.2/connectors/pulsar-io-file-4.2.2.nar $ docker compose restart pulsar Wait for the broker (repeat until it returns ["standalone"] ): $ curl -s http://localhost:8080/admin/v2/clusters $ docker exec -it pulsar bin/pulsar-admin sources available-sources Should list file . Configure and deploy: $ mkdir -p ~/pulsar/data/input-files $ chmod 777 ~/pulsar/data/input-files $ nano file-source-config.yaml configs: inputDirectory: /pulsar/data/input-files recurse: false keepFile: true fileFilter: '[^.].*' minimumFileAge: 0 $ docker cp file-source-config.yaml pulsar:/pulsar/ $ docker exec -it pulsar bin/pulsar-admin sources create \ --name file-source \ --tenant public \ --namespace default \ --destination-topic-name persistent://public/default/file-data \ --source-type file \ --source-config-file /pulsar/file-source-config.yaml $ docker exec -it pulsar bin/pulsar-admin sources status \ --tenant public \ --namespace default \ --name file-source Test it: $ printf "log entry 1\nlog entry 2\n" > ~/pulsar/data/input-files/test.txt $ docker exec -it pulsar bin/pulsar-client consume persistent://public/default/file-data \ --subscription-name file-reader \ --num-messages 2 \ --subscription-position Earliest Each message includes file.name /file.path /file.modified.time metadata. Use a fresh --subscription-name if you repeat this - an existing subscription resumes where it left off rather than replaying. Configure JWT Authentication $ docker exec -it pulsar bin/pulsar tokens create-secret-key \ --output /pulsar/data/my-secret.key $ docker exec pulsar bin/pulsar tokens create \ --secret-key file:///pulsar/data/my-secret.key \ --subject admin > ~/pulsar/data/admin-token.txt $ docker exec pulsar bin/pulsar tokens create \ --secret-key file:///pulsar/data/my-secret.key \ --subject app-client > ~/pulsar/data/app-client-token.txt $ docker compose stop pulsar $ docker run --rm apachepulsar/pulsar:4.2.2 cat /pulsar/conf/standalone.conf > conf/standalone.conf $ nano conf/standalone.conf Append: authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken tokenSecretKey=file:///pulsar/data/my-secret.key authorizationEnabled=true authorizationProvider=org.apache.pulsar.broker.authorization.Puls