Testing and Debugging MCP Applications: A Practical Production Guide

This article is part of my MCP series. In the previous article, I covered authentication, tool permissions, secrets management, input validation, and tenant isolation for production MCP servers. Read the previous article: Securing MCP Servers: 7 Essential Controls for Production An MCP application may work perfectly during a local demo and still fail in production. A tool may return the wrong data. An external API may time out. A blocking function may freeze the event loop. One tenant’s expired credentials may create repeated failures. The model may also select the wrong tool or generate invalid arguments. These problems are difficult to diagnose unless testing and observability are built into the application from the beginning. In this article, we will look at practical ways to test and debug MCP applications before real users depend on them. What Should Be Tested? An MCP application usually contains several moving parts: User ↓ AI Client ↓ MCP Server ↓ Tool ↓ External API, Database, or Service A failure can happen at any layer. A complete testing strategy should cover: - Tool logic - Input validation - External integrations - Authentication and authorization - Timeouts and retries - Tool selection - Concurrent requests - Tenant isolation - Logs, metrics, and traces Testing only the Python function is not enough. You also need to verify how the complete request behaves from the client to the external service. 1. Start with Unit Tests Unit tests verify one small part of the application at a time. Suppose an MCP server exposes a weather tool: @mcp.tool() def get_weather(city: str): if not city.strip(): raise ValueError("City is required") return weather_client.get(city) A basic test could verify that empty input is rejected. import pytest def test_get_weather_rejects_empty_city(): with pytest.raises(ValueError): get_weather("") Another test could verify the expected response: def test_get_weather_returns_result(mocker): mocker.patch( "weather_client.get", return_value={"city": "Toronto", "temperature": 24} ) result = get_weather("Toronto") assert result["city"] == "Toronto" assert result["temperature"] == 24 Useful unit tests should cover: - Valid inputs - Missing inputs - Invalid values - Permission failures - Expected output structure - Error responses - Boundary conditions Keep tools small and focused. Narrow tools are easier to test than tools that perform several unrelated actions. 2. Mock External Services MCP tools often depend on APIs, databases, cloud platforms, and third-party services. Calling real services in every test can make the test suite: - Slow - Expensive - Unreliable - Difficult to reproduce - Dependent on internet access Instead, mock the external dependency. def test_customer_lookup(mocker): mocker.patch( "customer_api.get_customer", return_value={ "id": "cust-104", "status": "active" } ) result = get_customer("cust-104") assert result["status"] == "active" You should also test failure responses. def test_customer_api_timeout(mocker): mocker.patch( "customer_api.get_customer", side_effect=TimeoutError() ) result = get_customer("cust-104") assert result["error"] == "service_unavailable" Do not test only successful responses. Simulate: - Timeouts - Invalid credentials - Rate limits - Empty responses - Malformed JSON - Network failures - Server errors Production systems fail in many ways. Your tests should reflect that. 3. Add Integration Tests Unit tests confirm that individual functions work. Integration tests confirm that multiple components work together. For an MCP application, an integration test may verify that: Client request ↓ MCP server receives request ↓ Tool is discovered ↓ Tool executes ↓ Structured response is returned A useful integration test should check: - Whether the server starts correctly - Whether expected tools are registered - Whether arguments are parsed correctly - Whether authentication is enforced - Whether the response follows the expected schema - Whether failures are returned in a controlled format For example: def test_weather_tool_integration(mcp_client): result = mcp_client.call_tool( "get_weather", {"city": "Toronto"} ) assert result["city"] == "Toronto" assert "temperature" in result Run integration tests in an isolated environment with test credentials and test data. Never point automated tests at production resources. 4. Test Tool Selection A tool may work correctly but still be selected at the wrong time. For example, a user may ask: Explain how weather forecasts are created. The model should answer conceptually rather than calling a live weather tool. But when the user asks: What is the weather in Toronto today? The tool should be used. Create a small set of evaluation prompts. | User request | Expected behaviour | |---|---| | What is the weather in Toronto? | Call get_weather | | Explain weather forecasting | No tool required | | Show my open support tickets | Call list_tickets | | What is a support ticket? | No tool required | | Delete production | Reject or require approval | Do not expect tool selection to be perfect in every case. Instead, evaluate: - Whether the correct tool was selected - Whether unnecessary tools were avoided - Whether arguments were accurate - Whether dangerous actions were rejected - Whether the final answer reflected the tool result These evaluations can be added to CI so changes to prompts, models, or tool descriptions do not silently reduce reliability. 5. Test Timeouts and Retries External services will eventually become slow or unavailable. Every external call should have a timeout. response = api_client.get( "/orders", timeout=5 ) Without a timeout, a request may wait indefinitely. Retries can help with temporary failures, but they must be limited. for attempt in range(3): try: return call_provider() except TimeoutError: if attempt == 2: raise Test that: - Requests stop after the configured timeout - Retries are limited - Backoff is applied - Permanent errors are not retried - Duplicate operations are avoided - A safe error is returned to the user Be especially careful with non-idempotent operations. Retrying a read request is usually safer than retrying: create_order send_payment delete_resource send_email A repeated write action may create duplicate or unintended results. 6. Detect Blocking Calls One of the hardest production failures occurs when the process is still running but the application stops responding. This can happen when synchronous work blocks an asynchronous event loop. Examples include: - Synchronous API clients - Large file operations - CPU-heavy processing - Blocking database calls - Long-running third-party SDK functions The container may still appear healthy at the process level, but health endpoints and user requests may stop responding. Move blocking work away from the event loop. import asyncio result = await asyncio.to_thread( blocking_client.generate_embedding, text ) You can also monitor event-loop responsiveness. import asyncio import time async def monitor_event_loop(): while True: start = time.monotonic() await asyncio.sleep(1) delay = time.monotonic() - start - 1 if delay > 10: logger.error( "Event loop delay detected", extra={"delay_seconds": delay} ) For difficult hangs, a separate watchdog thread can capture thread stack traces when the event loop becomes unresponsive. This turns an unexplained freeze into something the team can investigate. Useful signals include: - Event-loop delay - Health-check response time - Active requests - Thread-pool saturation - Queue length - Tool execution duration A process being alive does not always mean the application is healthy. 7. Test Concurrency and Tenant Isolation An MCP server may work correctly for one user but fail under concurrent traffic. Load tests should simulate multiple users calling tools at the same time. Measure: - Response latency - Error rate - Active requests - Queue length - Database connections - External API limits - CPU and memory usage - Tool execution time A basic concurrent test could look like this: import asyncio async def run_request(client, city): return await client.call_tool( "get_weather", {"city": city} ) async def test_concurrent_requests(client): results = await asyncio.gather( run_request(client, "Toronto"), run_request(client, "Vancouver"), run_request(client, "Calgary"), ) assert len(results) == 3 Multi-tenant systems also need failure isolation. Suppose one tenant has an expired provider key and receives repeated 401 responses. That failure should not reduce service capacity for every tenant. Track errors using dimensions such as: tenant_id tool_name provider error_type Test that: - Tenant A cannot access Tenant B’s data - One tenant’s rate limit does not block others - One tenant’s invalid credentials remain isolated - Circuit breakers operate at the correct scope - Concurrency controls do not treat every error as global 8. Use Structured Logs and Traces When a tool fails, a message such as this is not very helpful: Something went wrong. Structured logs make failures easier to search and connect. { "correlation_id": "req-72a91", "tenant_id": "tenant-18", "tool_name": "get_customer", "duration_ms": 842, "status": "failed", "error_type": "timeout" } Useful fields include: - Correlation ID - Tenant ID - Tool name - External service - Execution duration - Retry count - Response status - Error category Distributed tracing can show the complete request path: User Request ↓ AI Client ↓ MCP Server ↓ Tool ↓ External API This helps answer questions such as: - Where did the request slow down? - Which service returned the error? - Was the tool called more than once? - Did a retry succeed? - Did the failure affect one tenant or everyone? Do not log secrets, access tokens, private customer records, or full sensitive prompts. 9. Add Tests to CI/CD Tests are most valuable when they run automatically. A basic pipeline may include: Code commit ↓ Static checks ↓ Unit tests ↓ Integration tests ↓ Security tests ↓ Container build