CI/CD Pipeline Passed, but the Deployment Is Broken: How to Troubleshoot It

A green CI/CD pipeline does not always mean a successful release. Your tests passed. The build completed. The deployment job returned success. Then users report that the application is returning HTTP 500 errors, failing to connect to a database, or showing unexpected behavior.

What went wrong? The problem is that deployment success and application health are two different things. A pipeline can successfully deliver an application that cannot serve real requests. In this article, we'll walk through a practical troubleshooting process for a Kubernetes application that becomes unhealthy after a deployment.

The scenario

Imagine a new application version has just been deployed. The pipeline reports success, but shortly afterward:

• Users receive HTTP 500 responses.
• Some application instances restart repeatedly.
• API response times increase.
• Monitoring shows a sudden increase in errors.

The first thing to do is establish the impact. Is the entire service down, or are only some requests failing? That distinction helps determine whether to mitigate immediately or continue investigating the failure.

Step 1: Confirm what actually changed

Start by identifying the release that introduced the problem. Check:

• The deployed image tag or digest.
• The commit associated with the release.
• Configuration changes.
• Secret or credential changes.
• Database migrations.
• Infrastructure changes made alongside the application.

If you use Kubernetes, inspect the current Deployment:

kubectl get deployment api -n production

Check the rollout history:

kubectl rollout history deployment/api -n production

Inspect the deployed image:

kubectl get deployment api -n production \
  -o jsonpath='{.spec.template.spec.containers[*].image}'

Compare the current release with the last known healthy version. A useful question is: What changed between the last successful release and the first observed failure?

The answer may not be the application code. A configuration change, expired credential, incompatible database migration, or changed dependency can also break a release.

Step 2: Check the rollout status

A deployment command returning success is not sufficient evidence that every application instance is ready. Run:

kubectl rollout status deployment/api \
  -n production \
  --timeout=120s

Then inspect the Deployment:

kubectl describe deployment api -n production

And list its pods:

kubectl get pods -n production -o wide

Look for:

• Pods that are not ready.
• Containers repeatedly restarting.
• Image pull failures.
• Failed readiness or liveness probes.
• Replica counts that do not match expectations.
• Scheduling or resource problems.

If the rollout is stalled, inspect the events and affected pods before changing anything. A completed rollout is useful evidence, but it does not prove that the application is functioning correctly for users.

Step 3: Read the application logs

Next, investigate what the application is reporting. For a Deployment, you can inspect its current logs with:

kubectl logs deployment/api \
  -n production \
  --all-pods=true \
  --tail=100

If your Kubernetes version or logging setup does not support the options shown, list the pods and inspect each relevant container individually.

For a restarted container, retrieve the previous instance's logs:

kubectl logs <pod-name> \
  -n production \
  --previous

Search for evidence such as:

• Database authentication failures.
• Missing environment variables.
• Failed database migrations.
• Connection timeouts.
• Unhandled exceptions.
• Permission errors.
• Dependency services that are unavailable.

Avoid focusing only on the first error you see. Correlate timestamps with the deployment time and the beginning of the incident.

Step 4: Check configuration and dependencies

An application can build successfully while failing at runtime because its production configuration differs from the test environment. Check whether the new release changed:

• Environment variables.
• ConfigMap references.
• Secret references.
• Database connection settings.
• External API endpoints.
• Service account permissions.
• Network policies.

For example, a newly introduced environment variable might be required by the application but absent from the production configuration. You can inspect the Deployment's environment variable references:

kubectl describe deployment api -n production

Be careful when investigating secrets. Do not print secret values into terminal logs, CI/CD output, screenshots, or public articles.

If the application depends on a database or external service, test connectivity using approved diagnostic methods and the correct identity. Also consider compatibility. A new application version may expect a database schema or API contract that has not been updated yet.

Step 5: Determine whether the problem is application health or infrastructure

Now separate application-level failures from infrastructure-level symptoms.

• HTTP 500 errors: Application exceptions, dependency failures, and configuration
• HTTP 502 or 503 responses: Upstream health, routing, readiness, and gateway configuration
• High response latency: CPU, memory, database queries, connection pools, and downstream latency
• Pods restarting: Previous container logs, termination reasons, probes, and memory limits
• Database connection failures: Credentials, network access, connection limits, and database health
• Only the new version fails: Release changes, configuration differences, and dependency compatibility

These are starting points, not definitive diagnoses. For example, a 503 response can originate from different components depending on the architecture. Use metrics, logs, traces, and request outcomes to confirm the actual failure.

Step 6: Decide whether to roll back

If the release is causing significant customer impact and a known-good version is available, rollback may be the safest immediate mitigation. For a Kubernetes Deployment:

kubectl rollout undo deployment/api \
  -n production

Then monitor the rollback:

kubectl rollout status deployment/api \
  -n production \
  --timeout=120s

Verify the current replicas:

kubectl get deployment api -n production
kubectl get pods -n production

Finally, check application health and the relevant customer-facing metrics.

Important: A Kubernetes Deployment rollback reverts the Deployment's pod template to an earlier revision. It does not automatically reverse database migrations, restore deleted data, or undo every external change made by a release. If the release includes database changes, verify backward compatibility and the rollback procedure before acting.

Follow your organization's incident and change-management process, especially in production. If the deployment is managed by Helm, Argo CD, or another GitOps system, use the approved rollback mechanism so that the declared desired state and the live environment do not remain inconsistent.

Step 7: Add a post-deployment health gate

The better long-term solution is to prevent a release from being considered successful merely because the deployment command completed. A robust deployment workflow should verify:

• The expected version has been deployed.
• The required replicas become ready.
• The application health endpoint responds.
• A smoke test can complete a representative request.
• Error rates and latency remain within acceptable limits.

For example, after deploying a Kubernetes application, a pipeline can run:

kubectl rollout status deployment/api \
  -n production \
  --timeout=120s

That command checks rollout progress, but it is not a substitute for an application-level smoke test. A separate test might call an approved health endpoint:

curl --fail --silent --show-error \
  --max-time 10 \
  https://api.example.com/health

Replace the example URL with your actual endpoint. A successful health response is useful, but a deeper smoke test may be needed to verify database access and critical application behavior.

For important services, also consider automated rollback criteria based on customer-facing error rates, latency, and availability. Do not make rollback decisions based on a single noisy metric without accounting for the service's architecture and operational requirements.

Step 8: Turn the incident into an engineering improvement

Once the service has recovered, document:

• The customer impact and incident timeline.
• The change that introduced the failure.
• How the issue was detected.
• The root cause and contributing factors.
• The mitigation and recovery steps.
• The preventive actions assigned to the team.

Avoid stopping at statements such as "the deployment failed" or "the pod restarted." Those describe symptoms. A useful root-cause analysis explains why the failure occurred and what will reduce the likelihood or impact of recurrence.

Possible improvements include stronger automated tests, configuration validation, progressive delivery, safer database migrations, better alerts, and tested rollback procedures.

My troubleshooting checklist

When a CI/CD pipeline passes but the deployed application is broken, I would follow this sequence:

1. Assess customer impact and mitigate if necessary.
2. Identify the release and changes associated with the incident.
3. Check deployment and pod health.
4. Correlate application logs with the incident timeline.
5. Verify configuration and dependency connectivity.
6. Roll back when justified and safe.
7. Validate recovery with customer-facing checks.
8. Add preventive actions to the engineering backlog.

The objective is not to restart everything until the error disappears. It is to find the cause, restore service safely, and make the next release more reliable.

Final thoughts

A successful pipeline tells us that certain automated steps completed. It does not guarantee that the application will behave correctly in production. Reliable delivery requires both deployment automation and post-deployment verification. A green pipeline should be the beginning of release verification, not the end of it.

Official references: Kubernetes: Deployments and rollout status, Kubernetes: Update a Deployment and roll back, GitHub Actions: Deployments and environments