Dynatrace Managed troubleshooting: common cluster incidents and how to fix them

It is 8:30 am and an application owner reports that their charts stop at 3 am. Or the CMC shows a node in red after a hypervisor restart. Or a new batch of servers installed the day before shows up nowhere.

Every Dynatrace Managed administrator ends up facing these situations, often at the worst possible moment. That is when methodical Dynatrace Managed troubleshooting makes the difference compared with a string of random restarts.

The particular difficulty with Managed comes from its many layers: the nodes’ operating system, the internal components (server, Cassandra, Elasticsearch, transaction storage), the network between nodes, the ActiveGates, the OneAgents, and sometimes the connection to Mission Control. The same symptom can have very different causes depending on which layer is at fault.

This article offers a systematic diagnostic method, then reviews the most frequent incidents with their checks and fixes. It does not replace Dynatrace support: it helps you reach support with a diagnosis that is already well advanced.

A method before the recipes

• Define the scope: one environment or all of them? One node or the whole cluster? One group of agents or all of them?
• Pin down when it started: correlate with recent changes (update, OS patch, network change, certificate renewal).
• Work down the layers: network, system, services, Dynatrace components, agents.
• Collect before acting: logs, command output, CMC screenshots. A restart often wipes out the clues.
• One change at a time, and check the effect before the next one.

Step 1 - Take stock of the cluster

Start with the CMC: status of each node, recent events, health alerts. Complete the picture with the Cluster API, which is useful if the UI is slow:

curl -s -w "\nHTTP %{http_code}\n" -H "Authorization: Api-Token {token}" \
  "https://{your-domain}/api/v1.0/onpremise/cluster"

Then, on each node:

df -h /opt/dynatrace-managed /var/opt/dynatrace-managed
free -h; uptime
systemctl --failed
journalctl -p err --since "6 hours ago" | tail -100

Step 2 - Incident: data disk full

Symptoms: interrupted charts, rejected data, components in error, storage alerts in the CMC.

Diagnosis:

df -h /var/opt/dynatrace-managed
du -xh --max-depth=2 /var/opt/dynatrace-managed 2>/dev/null | sort -h | tail -15
find /var/opt/dynatrace-managed -name "*.log" -size +500M 2>/dev/null

Fixes: never manually delete Cassandra or Elasticsearch data files, as you would corrupt the storage. Act on capacity instead (extend the volume, ideally online through LVM) or on data volume (reduce retention periods in the CMC, review environment quotas). Only files clearly outside storage - temporary archives, old installers - can be cleaned up. If the situation is critical, contact support before doing anything.

Step 3 - Incident: node unreachable or in error

Symptoms: node reported offline, slow UI, replication alerts.

Diagnosis: test connectivity between nodes on the cluster’s internal ports (notably 5701-5711 and 8018 for Hazelcast and Nodekeeper, 7000, 7001 and 9042 for Cassandra, 9200 and 9300 for Elasticsearch, 8443 for OneAgent data), then name resolution and time synchronisation, which is often forgotten:

ping -c 3 {other-node-ip}
nc -vz {other-node-ip} {internal-port}
timedatectl status
chronyc tracking 2>/dev/null || ntpq -p

Fixes: restore the network path or the clock, then let the node resynchronise. If a service stays stopped, read its logs in /var/opt/dynatrace-managed/log/ (for example cassandra/ or elasticsearch/) before restarting it with the scripts in /opt/dynatrace-managed/launcher/ (cassandra.sh start, elasticsearch.sh start …). Avoid restarting several nodes at once: the documentation recommends handling nodes one at a time, as the cluster (Cassandra in particular) needs a quorum to stay consistent.

Step 4 - Incident: OneAgents stop reporting

Symptoms: hosts missing or greyed out, new servers invisible.

Diagnosis on the host:

systemctl status oneagent
sudo /opt/dynatrace/oneagent/agent/tools/oneagentctl --get-server
sudo /opt/dynatrace/oneagent/agent/tools/oneagentctl --get-tenant
curl -sv -o /dev/null https://{activegate-address}:9999/ 2>&1 | grep -E "Connected|SSL|HTTP"

Common causes: a firewall between the host and the ActiveGate or the cluster, a renewed certificate that is not trusted, the wrong environment ID at installation, a network zone (--set-network-zone=) with no ActiveGate available, or the environment’s host unit quota reached. The last case can be checked directly in the CMC, in the environment settings.

Step 5 - Incident: ActiveGate disconnected

An ActiveGate that is down can cut off an entire network zone at once. Check the service and its logs:

systemctl status dynatracegateway
journalctl -u dynatracegateway --since "2 hours ago" | tail -50
ss -tlnp | grep 9999

On Linux, the service is called dynatracegateway (systemctl restart dynatracegateway to restart it). Check disk space, memory and the ActiveGate’s connectivity to the cluster. For critical zones, deploy at least two ActiveGates to avoid a single point of failure.

Step 6 - Incident: update stuck or failed

Prevention: before any update, check the status of all nodes, disk space (an update needs room to unpack its files) and that the last backup succeeded. In offline mode, make sure you use the right dynatrace-managed-{version}.sh package and the right version sequence.

If it fails: keep the installer logs, do not retry over and over, and open a support ticket with the source version, the target version and the logs.

Step 7 - Incident: slow UI

Often a symptom of overload: saturated memory, Cassandra compactions lagging behind, very broad dashboard queries. Check node load, identify dashboards covering very long time ranges across many entities, and consider adding a node if the saturation is structural.

Step 8 - Prepare the case for support

Gather: Managed version, number of nodes, timeline, actions already tried, output of the commands above and the diagnostic archive downloaded from the CMC (it bundles server logs, other component logs and NGINX errors, among others). If the cluster is connected to Mission Control, support may have additional information.

Best practices

• Never touch storage files (Cassandra, Elasticsearch) by hand.
• Collect before restarting: logs and state are your best clues.
• One node at a time for restarts and maintenance.
• Time synchronisation monitored on all nodes and ActiveGates.
• Runbooks written for every incident already encountered, enriched after each post-mortem.
• Prevention: good cluster health monitoring avoids most of these incidents.

Frequently asked questions

Can you delete files to free up the data disk?

Never Cassandra or Elasticsearch files: you would corrupt the storage. Act on capacity (extend the volume) or on data volume (retention periods, quotas), and only clean up files clearly outside storage, such as temporary archives or old installers.

Why not restart all nodes at the same time?

The cluster needs a quorum to stay consistent. Restart one node at a time, after collecting logs and state, and check that it has resynchronised before moving on to the next.

Hosts have disappeared: where do you start?

On the host, check the oneagent service, then the server and environment configured with oneagentctl, then connectivity to the ActiveGate. In the CMC, check that the environment’s host unit quota has not been reached.

Conclusion

Most incidents on a Managed cluster boil down to a few families: saturated storage, network or clock issues between nodes, agent and ActiveGate connectivity, and poorly prepared updates. With a rigorous method - define the scope, pin down the start, work down the layers, collect before acting - and standard Linux commands, you can solve a good share of them yourself, and turn the rest into fast, precise support tickets.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Always check values (ports, versions, sizes) against the official documentation for your Managed version.

Originally published on span.observer. French version: https://span.observer/fr/depannage-cluster-managed/