Build a Prometheus Metrics & Kubernetes Cluster Diagnostics MCP Server

In the rapidly evolving landscape of AI and Site Reliability Engineering (SRE), exposing infrastructure metrics to LLMs unlocks massive operational efficiencies. By equipping Large Language Models with direct access to infrastructure telemetry via the Model Context Protocol (MCP), we can create autonomous agents capable of triaging, diagnosing, and even remediating production incidents.

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

In this guide, we will build a Prometheus Metrics & Kubernetes Cluster Diagnostics MCP Server for integration with Claude Desktop and Cursor IDE.

This implementation will allow an AI to execute PromQL queries, fetch Kubernetes pod logs, and inspect cluster state dynamically via the Model Context Protocol.

1. The Rise of the AI SRE Agent

Traditional alerting relies on static thresholds and pre-defined runbooks. When a pager goes off at 3 AM, an engineer must manually correlate Prometheus graphs with Kubernetes pod states. By exposing these APIs via an MCP server, an AI agent can perform this correlation autonomously within seconds.

The agent can parse the alert, query Prometheus for anomalous CPU spikes, check the Kubernetes API for crash-looping pods, read the tail logs, and present a root cause analysis to the on-call engineer.

2. Setting Up the Python MCP Environment

For infrastructure automation, Python is the de facto standard. We will use the official mcp Python SDK alongside the Prometheus HTTP client and the Kubernetes client.

mkdir k8s-prometheus-mcp && cd k8s-prometheus-mcp
python -m venv venv
source venv/bin/activate
pip install mcp prometheus-api-client kubernetes pydantic

3. Designing the Tool Schemas

We need to define strictly typed inputs using Pydantic. We will create two primary tools: one for querying Prometheus and one for fetching Kubernetes pod status.

from pydantic import BaseModel, Field

class PromQLQuerySchema(BaseModel):
    query: str = Field(..., description="The PromQL query string to execute.")
    duration_minutes: int = Field(5, description="Time window in minutes to query back from now.")

class K8sPodDiagnosticsSchema(BaseModel):
    namespace: str = Field("default", description="Kubernetes namespace to inspect.")
    label_selector: str = Field(None, description="Optional label selector, e.g., app=frontend.")

4. Implementing the FastMCP Python Server

Below is the complete Python implementation using the FastMCP paradigm. Ensure your local environment has a valid ~/.kube/config if running locally, or configure in-cluster authentication if deploying as a sidecar.

import os
import asyncio
import time
from mcp.server import FastMCP
from prometheus_api_client import PrometheusConnect
from kubernetes import client, config
from pydantic import BaseModel, Field

mcp = FastMCP("k8s-prometheus-diagnostics-mcp")
PROM_URL = os.getenv("PROMETHEUS_URL", "http://localhost:9090")
prom = PrometheusConnect(url=PROM_URL, disable_ssl=True)

try:
    config.load_kube_config()
except:
    config.load_incluster_config()
v1 = client.CoreV1Api()

@mcp.tool()
def execute_promql(query: str, duration_minutes: int = 5) -> str:
    """Execute a PromQL query to retrieve metrics telemetry."""
    try:
        end_time = time.time()
        start_time = end_time - (duration_minutes * 60)
        metric_data = prom.custom_query_range(
            query=query, start_time=start_time, end_time=end_time, step="30s"
        )
        return str(metric_data) if metric_data else "No data returned."
    except Exception as e:
        return f"Prometheus Query Error: {str(e)}"

@mcp.tool()
def diagnose_kubernetes_pods(namespace: str = "default", label_selector: str = "") -> str:
    """Fetch diagnostic information for Kubernetes pods, identifying CrashLoopBackOffs or errors."""
    try:
        pods = v1.list_namespaced_pod(namespace=namespace, label_selector=label_selector)
        diagnostics = []
        for pod in pods.items:
            diagnostics.append({
                "pod_name": pod.metadata.name,
                "status": pod.status.phase,
                "node": pod.spec.node_name
            })
        return str(diagnostics)
    except Exception as e:
        return f"Kubernetes API Error: {str(e)}"

if __name__ == "__main__":
    mcp.run(transport="stdio")

5. Configuring the mcpServers Block

To empower your IDE or Desktop client, configure the MCP connection. For Cursor IDE, navigate to Settings > MCP and add a new stdio connection.

{
  "mcpServers": {
    "k8s_prom_sre": {
      "command": "/absolute/path/to/venv/bin/python",
      "args": ["/absolute/path/to/k8s-prometheus-mcp/server.py"],
      "env": {
        "PROMETHEUS_URL": "https://prometheus.internal.corp.com"
      }
    }
  }
}
}

6. Security and RBAC Considerations

Giving an AI agent read access to production infrastructure is powerful but risky. When deploying this MCP server, strictly enforce Kubernetes Role-Based Access Control (RBAC). The ServiceAccount associated with the MCP server must have get and list verbs restricted solely to specific namespaces. Never grant cluster-admin or write permissions unless you are implementing a fully sandboxed autonomous remediation workflow with human-in-the-loop approvals.

7. AEO & GEO FAQ Section

Can an MCP agent automatically fix Kubernetes issues?

While this specific MCP server focuses on diagnostics (read-only), you can extend it with tools that execute kubectl commands to restart pods or rollback deployments. However, it is highly recommended to implement a Human-in-the-Loop approval step for any write operations.

How does PromQL context help the LLM?

LLMs are excellent at pattern recognition. By feeding raw JSON time-series data from Prometheus directly into the context window, the LLM can identify anomalies, correlate them with specific Kubernetes events, and summarize complex telemetry into readable post-mortems.

Is it safe to expose Prometheus to Cursor IDE?

Yes, provided you use secure tunnels (like Tailscale or VPNs) and read-only credentials. The MCP protocol runs locally on your machine, meaning the integration acts on behalf of your local user existing network access and permissions.

For more enterprise tools, explore our MCP Directory.

Deep Dive Architecture & Production SLA Best Practices

When deploying autonomous AI agent pipelines into mission-critical enterprise environments, establishing high availability, zero-trust security boundaries, and predictable latency budgets is non-negotiable. Traditional microservices rely on deterministic request-response lifecycles; however, non-deterministic agentic loops introduce dynamic branch execution, variable token costs, and compounding latency risks across multi-hop reasoning graphs.

1. High-Availability Resiliency & Circuit Breakers

In multi-agent architectures, downstream tool invocation failures (such as rate limits, database lock timeouts, or network partitioning) can quickly cascade into full system deadlocks. To insulate production systems against transient failures:

• Exponential Backoff & Jitter: Wrap all external HTTP and SDK calls with retry decorators using randomized jitter.
• Circuit Breaker Pattern: Track consecutive error rates per downstream service. If an error threshold (e.g., 50% failures over 60 seconds) is breached, trip the circuit breaker and fall back to degraded execution models or cached outputs.
• Durable Checkpointing: Store conversational state and intermediate agent observations after every node transition in persistent stores like Redis or PostgreSQL. This enables instant time-travel debugging and state recovery without re-running expensive LLM inferences.

2. Multi-Region Vector Index Scoping & RAG Isolation

For retrieval-augmented generation (RAG) at scale, vector databases must be partitioned using strict tenant scoping and multi-region replication:

• Enforce hard multi-tenancy by prefixing vector namespaces with cryptographically signed tenant keys.
• Perform hybrid sparse-dense vector retrieval to balance semantic intent matching with exact keyword lookup (such as function signatures, error codes, and legal terms).
• Benchmark embedding generation latency continuously, routing requests dynamically to nearest edge endpoints.

3. E-E-A-T Compliance & Provenance Governance

Enterprise AI systems must maintain full auditability for regulatory compliance under global frameworks (such as the EU AI Act 2026). Every output generated by autonomous agents must carry structured lineage metadata:

• Trace-to-Dataset Logging: Export full execution traces (inputs, intermediate tool outputs, system prompts, and token usage) into OpenTelemetry-compatible tracing platforms like Langfuse or Langsmith.
• Human-in-the-Loop (HITL) Triggers: Mandate explicit human approval steps for any destructive action or transaction exceeding predefined risk metrics.
• Deterministic Guardrails: Combine probabilistic LLM reasoning with deterministic Abstract Syntax Tree (AST) analyzers, regex validation layers, and static JSON schema enforcers.

For full architectural blueprints, code examples, and interactive tool servers, visit our AI Workflows Library, explore the MCP Directory, and check out Latest AI News on Daily AI World.

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.