Manticore Search authentication rollout checklist for production

Introduction

In production, "turn it on and done" almost never works. On a standalone node, the technical sequence is short: enable auth, restart Manticore, create an admin user, and update clients. A topology with distributed tables or replication clusters needs extra preparation because nodes must authenticate to each other as well. Handle the rollout like a small release. Inventory clients and nodes, prepare the auth data, rehearse the procedure for your topology, and then cut over. A rehearsal exposes failures before the maintenance window.

This checklist is for users who plan to enable authentication and want to roll it out as safely as possible in an existing system. Remember that authentication is disabled until you configure auth; after cutover, clients that still omit credentials will fail.

Choose the procedure by topology

Phase 1: Inventory before changing anything

Start by listing every client that connects to Manticore Search, including each application and independently deployed application component. Do this before editing the config.

Common things to check:

• Application search frontends
• Ingest workers
• Cron jobs
• Dashboards and BI tools
• Support or admin tools
• Schema change/update scripts
• Backup and maintenance scripts
• Local scripts run manually

For each client, record something like this:

Then confirm the deployment basics:

• Is this a standalone node, a distributed-table topology, a replication cluster, or a combination of them?
• Are you running RT mode or plain mode?
• In plain mode, where should the auth file live?
• Is pid_file configured in the config? Bootstrap needs it.
• Are all participating nodes running a version that supports the same authentication protocol?
• Will SQL clients use SSL, and will HTTP clients use HTTPS when credentials cross a network?
• Where will credentials, including temporary Bearer tokens, be stored safely?
• Who is allowed to see the first admin password?
• If replication is present, also record every cluster name and its persisted user in each node's <data_dir>/manticore.json.

Back up the complete data directory, configuration, and any existing auth store before the rehearsal and again before production cutover. Do not skip the credential-handling questions. CREATE USER returns a raw Bearer token; TOKEN generates a new token for the specified user. SHOW TOKEN later shows the stored token hash, not the raw token. If the raw token is lost, rotate it with TOKEN and update the token in your application.

Phase 2: Set up least-privilege users

Create users based on specific workloads, rather than granting permissions based on what someone might need later. Use a small matrix like this:

Keep in mind that admin is a narrow permission. It only allows managing authentication and authorization state; it does not imply read, write, schema, or replication. That matters because a person who manages credentials does not automatically need to read business data. A service that searches products does not need to write documents. A migration job does not need to manage users.

Plan negative permission tests as well. For every user you create, choose at least one thing it should be able to do and one thing it should not be able to do. Examples:

• app_read can search products. app_read cannot insert into products.
• app_ingest can write to products. app_ingest cannot manage auth.
• schema_job can change schema for products. schema_job cannot read tables unless you grant read.

Phase 3: Test in staging

Use a staging environment to rehearse the sequence of steps you plan to follow in production.

RT mode

searchd {
    data_dir = /var/lib/manticore
    auth = 1
    auth_log_level = info
    ...
}

To turn auth off explicitly in RT mode:

searchd {
    data_dir = /var/lib/manticore
    auth = 0
    ...
}

Plain mode

searchd {
    auth = /var/lib/manticore/auth.json
    auth_log_level = info
    ...
}

Keep the auth file private. Before the first bootstrap, Manticore may create an empty auth file. After bootstrap, that file stores auth data and credential hashes.

This bootstrap sequence works on a standalone node or on an isolated temporary daemon that prepares auth data for several nodes. Never start an existing replication data directory with an empty auth store. Its persisted cluster user will be missing, and Manticore may skip the cluster descriptor.

Start searchd, then bootstrap the first administrator:

searchd --config /etc/manticoresearch/manticore.conf --auth

For scripted setup:

printf 'admin\nStrongPass#2026\nStrongPass#2026\n' | \
    searchd --config /etc/manticoresearch/manticore.conf --auth-non-interactive

⚠️ Warning: The command above includes the password in plain text. In production automation, feed the three input lines to standard input from your secret-management system.

Bootstrap creates the first administrator with all actions, including replication; no additional grant is needed. The command does not return a bearer token. If the administrator needs HTTP Bearer access, connect as that user and run TOKEN, or use the HTTP POST /token endpoint.

Multi-node deployment: create the auth store once

Use a temporary daemon with its own empty data directory, PID file, and listeners. Bootstrap the administrator and shared service users, then stop the daemon cleanly. Copy the resulting auth store to every participating node before enabling authenticated node-to-node traffic.

Do not recreate the same users independently: matching names and passwords can still produce different stored authentication material.

Create staging users

Based on the notes from the previous phases, for example:

CREATE USER 'app_read' IDENTIFIED BY 'ReadPass#2026';
GRANT read ON 'products' TO 'app_read';

CREATE USER 'app_ingest' IDENTIFIED BY 'IngestPass#2026';
GRANT write ON 'products' TO 'app_ingest';

CREATE USER 'schema_job' IDENTIFIED BY 'SchemaPass#2026';
GRANT schema ON 'products' TO 'schema_job';

CREATE USER 'security_admin' IDENTIFIED BY 'AdminPass#2026';
GRANT admin ON * TO 'security_admin';

Store the returned Bearer tokens in secure storage. Remember: raw tokens must not be left in logs, shell history, or unprotected files. If you need to rotate one, use:

TOKEN 'app_read';

SHOW TOKEN is not a way to recover the raw token:

SHOW TOKEN FOR 'app_read';

Review users and permissions:

SHOW USERS;
SHOW PERMISSIONS;
SHOW PERMISSIONS FOR 'app_read';

Run allow and deny tests

For the read-only user:

curl -H "Authorization: Bearer <app_read_token>" \
    http://127.0.0.1:9308/sql?mode=raw \
    -d "SELECT * FROM products LIMIT 10"

Then verify that an unauthorized operation is denied:

curl -H "Authorization: Bearer <app_read_token>" \
    http://127.0.0.1:9308/sql?mode=raw \
    -d "INSERT INTO products(id,title) VALUES(1,'test')"

For this HTTP request, expect a 403 Forbidden response. Over SQL/MySQL, a permission denial returns ERROR 1045 with a permission-denied message.

SQL clients should connect with a Manticore user name and password. The SQL/MySQL protocol in Manticore supports mysql_native_password:

MYSQL_PWD=ReadPass#2026 \
    mysql -h127.0.0.1 -P9306 -uapp_read \
    -e "SELECT * FROM products LIMIT 10"

HTTP clients can use Basic authentication or Bearer tokens:

curl -u app_read:ReadPass#2026 \
    http://127.0.0.1:9308/sql?mode=raw \
    -d "SELECT * FROM products LIMIT 10"

HTTP authentication schemes (Basic, Bearer) are case-insensitive; user names are case-sensitive.

If you edit the auth file outside the daemon during maintenance, reload it:

RELOAD AUTH;

Phase 4: Production rollout checklist

Use this checklist for every deployment, then follow the procedure for your topology.

• [ ] Confirm you have a current configuration and data-directory backup.
• [ ] Back up manticore.json and the existing auth store separately.
• [ ] Confirm existing network protections stay in place.
• [ ] Confirm pid_file is set in the config.
• [ ] Confirm where the auth file will be created or loaded from.
• [ ] Confirm password and token storage is ready.
• [ ] Confirm every participating node runs a compatible Manticore version.
• [ ] Rehearse the same topology and restart order in staging.
• [ ] Prepare one canonical auth store for users shared across nodes.
• [ ] Choose the standalone, distributed, or replication procedure below.
• [ ] Enable auth during a planned maintenance window.
• [ ] Expect unauthenticated clients to fail after auth is enabled.
• [ ] Store tokens in a protected secrets store immediately after they are issued. Do not store raw tokens in files, shell history, or logs.
• [ ] Update SQL connection code to send user names and passwords.
• [ ] Update HTTP connection code to use Basic authentication or Bearer tokens.
• [ ] Run the allow and deny tests from staging.
• [ ] Verify internal node-to-node operations when the deployment has remote agents or replication.
• [ ] Check the auth log.
• [ ] Run application stress tests that cover search, ingest, dashboards, and maintenance scripts.
• [ ] Rotate any temporary rollout credentials.
• [ ] Keep the first admin credential out of normal application use.

Standalone node

For a standalone node, the direct bootstrap sequence is sufficient:

1. Stop Manticore cleanly and take the final backup.
2. Configure auth and start searchd.
3. Bootstrap the first administrator with searchd --config <path> --auth.
4. Create the production users and permissions.
5. Update clients and run the planned allow and deny tests.
6. Confirm that existing tables and known rows are still available.

Distributed tables and remote agents

Distributed queries send remote-agent requests as the current session user. Each remote node must have the same stored authentication material for that user and grant the required permission on the remote table.

For a new rollout across distributed nodes:

1. Create the shared users once in the isolated bootstrap daemon from Phase 3.
2. Stop the affected agents and masters for the coordinated cutover.
3. Configure auth and place the same canonical auth store on every participating node. Preserve restrictive ownership and permissions, and compare checksums.
4. Start remote agents before the masters that query them.
5. Test a direct authenticated query on each agent, followed by the equivalent distributed query through the master.
6. Create node-local users only after shared traffic works, and keep shared user records synchronized whenever their passwords, tokens, or permissions change.

Create shared users only once. Independently created accounts can have different stored authentication material even when their names and passwords match.

Existing replication cluster

Moving an existing unauthenticated replication cluster to auth requires a coordinated restart. Do not enable auth and bootstrap the first user against the existing cluster data. The empty store does not contain the persisted cluster user, so Manticore may skip the cluster descriptor.

Use this sequence:

1. While the unauthenticated cluster is healthy, choose its future replication identity and persist it:

   ALTER CLUSTER products UPDATE user 'cluster_repl';
   

   UPDATE user writes the name to the cluster metadata. Authentication is still disabled, so Manticore neither creates nor checks the account at this point. Create it in step 3, before restarting any real cluster node with authentication enabled.

2. Verify that every node's <data_dir>/manticore.json now stores "user": "cluster_repl" for the cluster. If a node hosts several clusters, update each cluster to a user that will exist in the new auth store, or create and grant every persisted user before cutover.

3. Stop all cluster nodes cleanly and use the replication state to select the safe primary. After a clean shutdown, this is normally the node stopped last, with safe_to_bootstrap: 1 in its <data_dir>/grastate.dat. See Restarting a cluster.

4. Bootstrap cluster_repl as the first administrator in the isolated temporary daemon. The first administrator already has every action, including replication, so it needs no additional grant. If the cluster will use a separate least-privilege identity, create it once and grant replication before distributing the store.

5. Stop the temporary daemon.

6. Configure auth on every real cluster node and copy the exact generated auth store to each one. Keep the files private and byte-identical. Do not start a real cluster node before this store is in place.

7. Restart a two-node cluster in this order:

   • Start the non-primary peer normally and wait only for its daemon listener. Its cluster can report closed at this point; do not write to it.
   • Start the recorded safe primary with --new-cluster, or use the corresponding manticore_new_cluster service action.
   • Wait for the safe primary to report cluster_products_status=primary and cluster_products_node_state=synced.
   • Stop the first peer cleanly and start it normally again.
   • Wait for the same primary and synced values on that peer.
   • Never use --new-cluster on it.

The safe primary reads the persisted cluster user from a descriptor peer during startup, so the initial peer must already be listening.