How Kubernetes Health Checks Brought Down a Payment Service

One afternoon while I was on-call, I received an alert that one of our team's Tier 2 payment services hadn't received any traffic for the past hour.

I logged into the Kubernetes dashboard to investigate and immediately noticed that every pod backing the service was stuck in a continuous restart cycle. At the same time, CPU and memory utilization had spiked across every container.

To mitigate the initial impact, I contacted the platform team and looking at the memory spike they decided to provision more memory for the containers. The containers restarted successfully, and the service was restored within 15 minutes.

The Investigation

Although it looked like the memory spike had caused the outage, something didn't add up. This service had never exhibited memory pressure before, and its traffic that day was consistent with its normal daily pattern.

I reviewed the Datadog dashboards, looking for anything that could explain the sudden increase in memory usage - like an unexpected traffic surge, a recent deployment, or a JVM leak - but found nothing unusual.

While increasing the memory allocation had restored the service, it felt more like a mitigation than a root-cause fix. I couldn't justify submitting a code change to permanently increase the service's memory limits in its Helm chart simply to match the temporary configuration the platform team had applied through GitOps during the incident. Before making that change, I wanted to understand why the service suddenly needed more memory in the first place.

Looking at logs to narrow down the cause

Datadog metrics didn't help much to understand the root cause, so I started reading the service logs from an hour before the outage. I started noticing repeated downstream service health check failure logs starting roughly 30 minutes before the payment service outage.

However, the downstream failures did not appear to be causing the payment application to return errors, and the payment service itself looked healthy from an application perspective.

While investigating further, I noticed repeated Kubernetes probe failures in the logs. At the time, I did not have a deep understanding of how Kubernetes probes influenced application lifecycle behavior, so I decided to dig deeper into the different types of Kubernetes probes and how they were configured for the payment service.

Deep Dive into Kubernetes Probes

According to the Kubernetes documentation, Kubernetes uses probes to continuously monitor the health of containers in a pod. The kubelet, an agent that runs on each node in a cluster, periodically executes a probe - such as an HTTP request, gRPC request, a TCP check, or a command inside the container - to determine its state.

Based on the probe results, Kubernetes can restart unhealthy containers or stop sending traffic to containers until they are ready to serve requests again.

The kubelet can execute three types of probes, each serving a different purpose:

• Startup probes: Startup probes verify whether the application within a container has successfully started. This probe is only executed at startup, unlike liveness and readiness probes, which the kubelet runs periodically. If the startup probe fails, the kubelet kills the container and tries to restart the container.

• Liveness probes: Liveness probes determine when to restart a container. For example, liveness probes could catch a failure where an application is running but unable to make progress. The kubelet tries to restart containers in such a state to help make the application more available despite the bugs.

• Readiness probes: Readiness probes determine when a container is ready to accept traffic. These probes are also useful later in the container's lifecycle, for example, when recovering from temporary failures. They run on the container during its whole lifecycle.

Read the complete article here: Production Engineering Diaries - GitHub