Production Debugging Skills

Production Debugging Skills

Master production debugging with free flashcards and hands-on techniques that transform mysterious failures into resolved incidents. This lesson covers systematic debugging approaches, observability-driven investigation, and real-world troubleshooting workflows—essential skills for maintaining reliable production systems.

Welcome to Production Debugging 🔍

Production debugging is fundamentally different from local development debugging. You can't just attach a debugger, restart the service freely, or add println statements everywhere. Production environments demand non-invasive investigation techniques that maintain system stability while you diagnose issues affecting real users.

The modern production debugging workflow relies on three pillars of observability: logs, metrics, and traces. These signals provide windows into system behavior without disrupting operations. Mastering how to navigate these signals efficiently separates effective production engineers from those who struggle during incidents.

💡 Did you know? Google's Site Reliability Engineering team estimates that time to detection (TTD) and time to resolution (TTR) are the two most critical metrics for service reliability. Effective debugging skills directly reduce TTR, often by 10x or more.

Core Concepts: The Debugging Mindset 🧠

1. Signal-Driven Investigation

Production debugging starts with observability signals, not assumptions. The scientific method applies:

The Investigation Loop:

┌─────────────────────────────────────┐
│   🚨 Alert/Report of Issue         │
└──────────────┬──────────────────────┘
               ↓
┌──────────────────────────────────────┐
│  📊 Gather Observable Signals        │
│  • Metrics (what changed?)           │
│  • Logs (what errors occurred?)      │
│  • Traces (where did it slow down?)  │
└──────────────┬───────────────────────┘
               ↓
┌──────────────────────────────────────┐
│  🧪 Form Hypothesis                  │
│  "If X is the problem, I should      │
│   see Y in the signals"              │
└──────────────┬───────────────────────┘
               ↓
┌──────────────────────────────────────┐
│  🔍 Test Hypothesis                  │
│  Query specific signals to validate  │
└──────────────┬───────────────────────┘
               ↓
         ┌─────┴─────┐
         ↓           ↓
    ✅ Confirmed  ❌ Rejected
         │           │
         ↓           └──→ (Loop back)
┌──────────────────┐
│  🎯 Root Cause   │
│  Identified      │
└──────────────────┘

Key principle: Never debug by randomly changing things. Each investigation step should either confirm or refute a specific hypothesis based on observable data.

2. The Time Dimension ⏰

Production issues exist in time series. Understanding temporal patterns is crucial:

Time Pattern	Likely Root Cause	Investigation Strategy
Sudden spike	Deployment, traffic surge, dependency failure	Compare before/after timestamps, check deploy logs
Gradual degradation	Memory leak, resource exhaustion, data growth	Plot resource metrics over hours/days
Periodic pattern	Cron job, batch process, scheduled task	Correlate timing with known scheduled operations
Random/intermittent	Race condition, network flap, external API timeout	Look for statistical outliers, percentile metrics

💡 Pro tip: Always ask "When did this start?" before "What is broken?" The timeline often points directly to the root cause.

3. The Signal Hierarchy 📊

Not all observability signals are equally useful for debugging. Here's the hierarchy:

Metrics → Traces → Logs

Signal Type	Best For	Limitations
Metrics (counters, gauges, histograms)	• Detecting WHAT is wrong • Showing scope/magnitude • Trending over time	Low cardinality, no individual request details
Traces (distributed spans)	• Finding WHERE slowness occurs • Understanding request flow • Timing breakdowns	Sampling may miss rare issues, overhead considerations
Logs (structured events)	• Understanding WHY it failed • Exception details • Business logic context	Volume challenges, expensive to query, may miss context

Debugging workflow:

1. Start with metrics to understand scope and timing
2. Use traces to narrow down which service/function is problematic
3. Dive into logs for specific error messages and stack traces

4. The Correlation Technique 🔗

Production debugging often involves finding correlations between signals:

## Example: Correlating error rate with deployment
## Pseudo-code for investigation query

metrics.query(
    metric="error_rate",
    time_range="last_2_hours",
    group_by="service_version"
)
## Result: version v2.3.1 has 15% errors, v2.3.0 had 0.1%
## Hypothesis: Regression introduced in v2.3.1

traces.query(
    service="payment-api",
    version="v2.3.1",
    status="error",
    limit=10
)
## Result: All failures occur in new checkout_validation function

logs.query(
    service="payment-api",
    version="v2.3.1",
    level="ERROR",
    function="checkout_validation"
)
## Result: "NullPointerException: billingAddress.zipCode"
## Root cause identified: Missing null check in new validation logic

Correlation patterns to look for:

• Error rate ↑ + CPU usage normal = Logic error (not load issue)
• Latency ↑ + Database query time ↑ = Database bottleneck
• Memory usage ↑ over time + GC time ↑ = Memory leak
• Request rate ↑ + Error rate flat = Scaling working correctly

5. The Scope-Down Strategy 🎯

Production systems are complex. Effective debugging requires systematically narrowing the problem space:

🌍 ENTIRE SYSTEM
   |
   ├─ ❓ Is it affecting all users or specific segments?
   |     └─→ Specific segment = narrows to feature/region
   |
   ├─ ❓ Is it one service or multiple?
   |     └─→ One service = problem is localized
   |
   ├─ ❓ Is it all endpoints or specific operations?
   |     └─→ Specific operation = focus on that code path
   |
   ├─ ❓ Is it all requests or specific patterns?
   |     └─→ Pattern exists = find common attributes
   |
   └─→ 🎯 SPECIFIC CODE PATH + CONDITIONS

Example scoping questions:

## Question: "Users can't log in"
## Vague - need to scope it down

## Query 1: What percentage of login attempts fail?
login_success_rate = metrics.query(
    "login_attempts{status='success'}"
) / metrics.query("login_attempts{status='*'}")
## Result: 85% success (so 15% failing, not total outage)

## Query 2: Is it specific to a user segment?
metrics.query(
    "login_attempts{status='failure'}",
    group_by="user_region"
)
## Result: 90% of failures from 'us-west-2'
## Scope: Regional issue, not global

## Query 3: Which auth method is failing?
metrics.query(
    "login_attempts{status='failure',region='us-west-2'}",
    group_by="auth_method"
)
## Result: 100% OAuth failures, password auth working
## Scope: OAuth integration issue in us-west-2

## Query 4: When did it start?
timestamp_of_first_failure = logs.query(
    "auth_method='oauth' AND status='failure' AND region='us-west-2'",
    order="timestamp ASC",
    limit=1
)
## Result: Started at 2026-03-15 14:32 UTC
## Hypothesis: Check deploys/config changes at that time

Example 1: Debugging High API Latency 🐌

Scenario: Your API's p99 latency jumped from 200ms to 3000ms. Users are complaining about slow page loads.

Investigation Process:

Step 1: Confirm the symptom with metrics

## Query your APM/metrics system
query = """
  histogram_quantile(0.99,
    rate(http_request_duration_seconds_bucket[
      service="api",
      status="success"
    ][5m])
  )
"""
## Result confirms: p99 jumped at 10:15 AM

Step 2: Check what changed around that time

## Check deployment history
kubectl rollout history deployment/api-service
## Output: v1.8.3 deployed at 10:12 AM
## Suspicious timing - 3 minutes before latency spike

Step 3: Use distributed tracing to find WHERE the slowness occurs

## Query traces for slow requests
traces = query_traces(
    service="api",
    min_duration="2s",
    start_time="10:15 AM",
    limit=20
)

## Analyze span durations
for trace in traces:
    print_span_breakdown(trace)

## Output shows:
## ├─ api.handle_request: 2850ms
##    ├─ validate_input: 5ms
##    ├─ database.query_user: 2800ms ⚠️
##    └─ format_response: 45ms

Finding: Database query time increased from ~50ms to ~2800ms.

Step 4: Investigate the database query

## Get SQL from slow trace
slow_trace_sql = get_trace_attribute(trace_id, "db.statement")
print(slow_trace_sql)

## Output:
## SELECT * FROM users 
## WHERE email = ? 
## AND account_status IN ('active', 'trial', 'suspended')
## ORDER BY last_login DESC

Step 5: Check database query performance

-- Explain the query
EXPLAIN ANALYZE 
SELECT * FROM users 
WHERE email = 'test@example.com' 
AND account_status IN ('active', 'trial', 'suspended')
ORDER BY last_login DESC;

-- Output:
-- Seq Scan on users (cost=0.00..45000.00)
-- Planning time: 0.5ms
-- Execution time: 2843ms

Root Cause: The query is doing a sequential scan. Checking the deploy diff:

## v1.8.3 changes
- WHERE email = ? AND account_status = 'active'
+ WHERE email = ? AND account_status IN ('active', 'trial', 'suspended')

The existing index was on (email, account_status) with equality, but the new IN clause changed the query plan.

Resolution:

-- Add composite index supporting the new query pattern
CREATE INDEX idx_users_email_status_login 
ON users(email, account_status, last_login);

Latency returned to normal within 2 minutes after index creation.

💡 Key lesson: When latency increases after a deployment, distributed tracing can pinpoint the exact span causing slowdown. Always check query plans when database queries are involved.

Example 2: Debugging Intermittent 5xx Errors 💥

Scenario: Your checkout service returns 500 errors for about 2% of requests, but you can't reproduce it locally.

Investigation Process:

Step 1: Understand the error pattern

## Check error rate distribution
metrics.query("""
  sum(rate(http_requests_total{status="500"}[5m])) 
  / sum(rate(http_requests_total[5m]))
""")
## Result: Steady 2% error rate, not correlated with traffic

## Check if errors are concentrated
metrics.query("""
  http_requests_total{status="500"}
""", group_by="pod_name")
## Result: Errors spread across all pods evenly
## Conclusion: Not a single bad instance

Step 2: Sample failed requests

## Get trace IDs for failed requests
failed_traces = query_traces(
    service="checkout",
    status_code=500,
    limit=50,
    sample_method="random"
)

## Look for common attributes
analysis = analyze_trace_attributes(failed_traces)
print(analysis)

## Output:
## Common patterns:
## - 100% have user_type="premium"
## - 98% have cart_size > 10 items
## - 87% have discount_code present

Pattern found: Failures correlate with premium users, large carts, and discount codes.

Step 3: Examine logs for actual error messages

## Query logs matching the pattern
logs.query("""
  service="checkout" AND
  level="ERROR" AND
  user_type="premium" AND
  discount_code EXISTS
""", limit=20)

## Output (example log):
## {
##   "timestamp": "2026-03-15T14:23:11Z",
##   "level": "ERROR",
##   "message": "Failed to calculate discount",
##   "exception": "ArithmeticException: Division by zero",
##   "stack_trace": "at DiscountCalculator.applyBulkDiscount(line 143)...",
##   "context": {
##     "discount_code": "BULK20",
##     "cart_items": 15,
##     "user_tier": "premium"
##   }
## }

Root Cause Found: Division by zero in bulk discount calculation.

Step 4: Locate the bug in code

## File: discount_calculator.py, line 143
def apply_bulk_discount(cart, discount_code):
    if discount_code.startswith("BULK"):
        # Extract percentage from code like "BULK20"
        percentage = int(discount_code[4:])
        
        # Bug: When cart has mixed eligibility, this can be zero
        eligible_items = [item for item in cart if item.bulk_eligible]
        
        # Division by zero when no eligible items!
        avg_price = sum(item.price for item in eligible_items) / len(eligible_items)
        
        discount_per_item = avg_price * (percentage / 100)
        return discount_per_item * len(cart)  # Wrong calculation too!

Why it was intermittent: Only triggered when premium users with bulk discount codes had carts with NO bulk-eligible items (rare but possible).

Fix:

def apply_bulk_discount(cart, discount_code):
    if discount_code.startswith("BULK"):
        percentage = int(discount_code[4:])
        eligible_items = [item for item in cart if item.bulk_eligible]
        
        # Handle edge case
        if not eligible_items:
            logger.warning(f"No bulk-eligible items for code {discount_code}")
            return 0
        
        avg_price = sum(item.price for item in eligible_items) / len(eligible_items)
        discount_per_item = avg_price * (percentage / 100)
        
        # Apply only to eligible items
        return discount_per_item * len(eligible_items)

💡 Key lesson: Intermittent errors often reveal edge cases. Use trace sampling to find patterns in failed requests, then correlate with logs for exception details.

Example 3: Debugging Memory Leak 🧠

Scenario: Your service's memory usage grows steadily over 48 hours until it hits the container limit and gets OOMKilled (Out Of Memory).

Investigation Process:

Step 1: Confirm the memory growth pattern

## Query container memory metrics
metrics.query("""
  container_memory_usage_bytes{pod=~"api-.*"}
""", time_range="7d")

## Visualize the pattern:

Memory Usage Over Time
  2GB ┤                                    ╱
      │                              ╱╱╱╱╱╱
  1.5G┤                        ╱╱╱╱╱╱
      │                  ╱╱╱╱╱╱
  1GB ┤            ╱╱╱╱╱╱
      │      ╱╱╱╱╱╱
  500M┤╱╱╱╱╱╱
      │
    0 ┼────┴────┴────┴────┴────┴────┴────
      0    8   16   24   32   40   48hrs
        Classic sawtooth pattern:
        growth → restart → growth → restart

Pattern identified: Linear memory growth suggesting leak.

Step 2: Correlate with application metrics

## Check if memory correlates with request count
metrics.query("""
  rate(http_requests_total[1h])
""", overlay_with="container_memory_usage_bytes")

## Result: Memory grows regardless of traffic
## Conclusion: Not simply caching legitimate data

Step 3: Enable memory profiling (if available)

## For Python services, enable tracemalloc snapshot
import tracemalloc
import logging

tracemalloc.start()

## After service runs for several hours
def dump_memory_snapshot():
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    
    logging.info("Top 10 memory allocations:")
    for stat in top_stats[:10]:
        logging.info(f"{stat}")

## Triggered via admin endpoint or signal

Output:

Top 10 memory allocations:
/app/metrics_collector.py:67: 1250 MB
/app/request_handler.py:142: 89 MB
/app/database.py:203: 45 MB
...

Step 4: Examine the problematic code

## File: metrics_collector.py, line 67
class MetricsCollector:
    def __init__(self):
        self.request_history = []  # ⚠️ Unbounded list!
    
    def record_request(self, request_data):
        # Bug: Never removes old entries
        self.request_history.append({
            'timestamp': time.time(),
            'endpoint': request_data.path,
            'duration': request_data.duration,
            'user_id': request_data.user_id,
            'response_size': request_data.response_size
        })
        
        # Original intent: Keep last hour for debugging
        # Reality: Grows indefinitely!

Root Cause: Unbounded list accumulating every request's metadata.

At 1000 requests/second × 48 hours = 172 million entries ≈ 1.2GB of memory.

Fix:

from collections import deque
import time

class MetricsCollector:
    def __init__(self, max_age_seconds=3600):
        # Use deque with max length
        self.request_history = deque(maxlen=3600000)  # ~1 hour at 1000 rps
        self.max_age = max_age_seconds
    
    def record_request(self, request_data):
        now = time.time()
        self.request_history.append({
            'timestamp': now,
            'endpoint': request_data.path,
            'duration': request_data.duration,
            'user_id': request_data.user_id,
            'response_size': request_data.response_size
        })
        
        # Periodically clean old entries
        self._cleanup_old_entries(now)
    
    def _cleanup_old_entries(self, current_time):
        # Remove entries older than max_age
        while (self.request_history and 
               current_time - self.request_history[0]['timestamp'] > self.max_age):
            self.request_history.popleft()

💡 Key lesson: Memory leaks often involve unbounded data structures. Memory growth patterns (linear vs. stepped vs. sawtooth) provide clues about the cause. Always use bounded collections or implement cleanup logic.

Example 4: Debugging Cascading Failures 🔥

Scenario: A single service degradation causes your entire system to fail. Multiple services are timing out and error rates are spiking across the board.

Investigation Process:

Step 1: Identify the blast radius

## Check error rates across all services
services_status = metrics.query("""
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  / sum(rate(http_requests_total[5m])) by (service)
""")

## Output:
## payment-service: 67% errors
## order-service: 45% errors  
## notification-service: 89% errors
## user-service: 12% errors
## recommendation-service: 3% errors (baseline)

Step 2: Map the dependency graph

Service Dependency Graph:

┌──────────────┐
│   Frontend   │
└───────┬──────┘
        │
    ┌───┴────┬──────────────┐
    ↓        ↓              ↓
┌───────┐ ┌────────┐ ┌──────────────┐
│ User  │ │ Order  │ │Recommendation│
│Service│ │Service │ │   Service    │
└───────┘ └───┬────┘ └──────────────┘
              │
         ┌────┴─────┐
         ↓          ↓
    ┌─────────┐ ┌──────────────┐
    │ Payment │ │Notification  │
    │ Service │ │  Service     │
    └─────────┘ └──────────────┘
         ⚠️           ⚠️
     High errors  High errors

Pattern: Services downstream from order-service are failing.

Step 3: Check resource saturation

## Check thread pool utilization
metrics.query("""
  thread_pool_active_threads{service="order-service"}
  / thread_pool_max_threads{service="order-service"}
""")
## Result: 100% - all threads blocked!

## Check what threads are doing
metrics.query("""
  thread_pool_active_threads{service="order-service"}
""", group_by="thread_state")
## Output:
## WAITING: 45 threads
## TIMED_WAITING: 5 threads
## Conclusion: Threads waiting on I/O

Step 4: Identify what threads are waiting for

## Sample traces showing long-running requests
traces = query_traces(
    service="order-service",
    min_duration="10s",
    limit=30
)

## Analyze common pattern
for trace in traces:
    print_waiting_spans(trace)

## Output: All traces show long wait on payment-service calls:
## ├─ order.create_order: 25000ms
##    ├─ validate_cart: 50ms
##    ├─ payment.authorize: 24500ms ⚠️
##    └─ notification.send: (not reached)

Step 5: Check payment service health

## Check payment service metrics
metrics.query("""
  rate(http_requests_total{service="payment-service"}[5m])
""")
## Result: Request rate is 10x normal

## Check where requests are coming from
metrics.query("""
  rate(http_requests_total{service="payment-service"}[5m])
""", group_by="caller")
## Output:
## order-service: 950 rps (normal: 100 rps)
## Conclusion: order-service is retrying aggressively

Step 6: Find the retry logic

## File: order_service/payment_client.py
class PaymentClient:
    def authorize_payment(self, amount, payment_method):
        max_retries = 10  # ⚠️ Too aggressive!
        retry_delay = 0.1  # 100ms
        
        for attempt in range(max_retries):
            try:
                response = self.http_client.post(
                    url="/authorize",
                    json={"amount": amount, "method": payment_method},
                    timeout=30  # ⚠️ Long timeout
                )
                return response
            except Timeout:
                if attempt < max_retries - 1:
                    time.sleep(retry_delay)
                    # Bug: No exponential backoff!
                    # Bug: No circuit breaker!
                    continue
                raise

Root Cause Chain:

1. Payment service experienced temporary latency spike (maybe dependency issue)
2. Order service requests started timing out (30s each)
3. Order service retried immediately, 10 times per request
4. This amplified load on payment service by 10x
5. Payment service became overwhelmed, latency increased further
6. Order service exhausted its thread pool waiting for payment responses
7. Order service stopped accepting new requests
8. Notification service (called by order service) also backed up
9. Cascading failure across multiple services

Fix: Implement circuit breaker and better retry logic

from circuitbreaker import CircuitBreaker, CircuitBreakerError
import random

class PaymentClient:
    def __init__(self):
        self.circuit_breaker = CircuitBreaker(
            failure_threshold=5,
            recovery_timeout=60,
            expected_exception=Timeout
        )
    
    @CircuitBreaker.decorate
    def authorize_payment(self, amount, payment_method):
        max_retries = 3  # Reduced
        base_delay = 0.5
        
        for attempt in range(max_retries):
            try:
                response = self.http_client.post(
                    url="/authorize",
                    json={"amount": amount, "method": payment_method},
                    timeout=5  # Reduced timeout
                )
                return response
            except Timeout as e:
                if attempt < max_retries - 1:
                    # Exponential backoff with jitter
                    delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
                    time.sleep(delay)
                    continue
                raise

💡 Key lesson: Cascading failures often result from retry storms and lack of backpressure mechanisms. Circuit breakers, timeouts, and exponential backoff are essential resilience patterns. Always trace the dependency chain during widespread outages.

Common Mistakes in Production Debugging ⚠️

1. Debugging by Guessing

❌ Wrong approach:

## "Maybe it's a caching issue, let me restart Redis"
## "Perhaps the database is slow, let me add an index"
## "Could be network, let me restart the load balancer"
## Random changes without data

✅ Right approach:

## Form hypothesis, test with data
## "If Redis is the issue, cache hit rate should be low"
cache_hit_rate = metrics.query("redis_cache_hits / redis_cache_requests")
print(f"Cache hit rate: {cache_hit_rate}")  # 94% - Redis is fine

## "If database is slow, query time should be high"
db_query_time_p99 = metrics.query("db_query_duration_p99")
print(f"DB p99 latency: {db_query_time_p99}ms")  # 45ms - DB is fine

## Systematically eliminate hypotheses with data

2. Ignoring the Timeline

❌ Wrong: Looking at current state only without considering when things changed

✅ Right: Always establish "When did this start?" and compare before/after:

## Compare metrics before and after the issue started
metrics.query(
    metric="error_rate",
    time_range="6h",  # Include time before issue
    annotate_deploys=True
)
## Often reveals: Deploy, config change, or external event correlation

3. Sampling Bias in Traces

❌ Wrong: Only looking at successful requests when debugging errors

## This query won't help debug failures!
traces = query_traces(
    service="api",
    status="success",
    limit=100
)

✅ Right: Sample specifically from failed requests:

traces = query_traces(
    service="api",
    status="error",  # Focus on failures
    sample_rate=1.0,  # Don't sample errors
    limit=100
)

4. Not Considering Cardinality

❌ Wrong: Querying high-cardinality logs without filters

## This query will timeout or be too expensive
logs.query(
    "level='ERROR'",  # Millions of results
    time_range="7d"
)

✅ Right: Add specific filters to reduce cardinality:

logs.query(
    "level='ERROR' AND service='checkout' AND error_type='ValidationError'",
    time_range="1h",  # Narrow time window first
    limit=100
)

5. Over-relying on Logs

❌ Wrong: Diving straight into logs without understanding the big picture

✅ Right: Follow the hierarchy: Metrics → Traces → Logs

DEBUGGING FUNNEL

📊 METRICS (start here)
   "What services are affected?"
   "How bad is it?"
   "When did it start?"
   ↓
🔍 TRACES (narrow down)
   "Where in the request flow?"
   "Which function is slow?"
   ↓
📝 LOGS (root cause)
   "What's the exact error?"
   "What were the conditions?"

6. Ignoring Percentiles

❌ Wrong: Only looking at average latency

avg_latency = metrics.query("avg(http_request_duration)")
## Average might be 200ms, hiding the fact that
## 5% of requests take 10+ seconds

✅ Right: Always check p95, p99, and max:

latency_percentiles = {
    'p50': metrics.query('http_request_duration_p50'),
    'p95': metrics.query('http_request_duration_p95'),
    'p99': metrics.query('http_request_duration_p99'),
    'max': metrics.query('http_request_duration_max')
}
## Reveals: p99 is 8000ms while p50 is 180ms
## Conclusion: Intermittent issue affecting small percentage

7. Not Testing Hypotheses

❌ Wrong: "I think I found the issue" → Immediately deploy fix

✅ Right: Validate your hypothesis first:

## Before:
## "I think the issue is in the user authentication cache"

## Validate:
if issue_is_cache_related():
    # Temporarily increase cache TTL in one pod
    # Monitor if that specific pod has fewer errors
    # If validated, roll out fix to all pods
    pass
else:
    # Hypothesis rejected, investigate further
    pass

Key Takeaways 🎯

The Debugging Checklist ✅

When responding to a production incident:

1. ⏰ Establish timeline: When did symptoms start?
2. 📊 Check metrics: What changed? (error rate, latency, throughput)
3. 🎯 Scope the blast radius: All users or specific segments?
4. 🔍 Correlate with events: Recent deploys? Config changes? Traffic spikes?
5. 🔬 Sample traces: Where is time spent in failed requests?
6. 📝 Examine logs: What are the actual error messages?
7. 🧪 Form hypothesis: "If X is the cause, I should see Y"
8. ✅ Test hypothesis: Query specific signals to validate
9. 🎯 Identify root cause: Confirmed through multiple signal types
10. 🔧 Implement fix: With validation before full rollout

Essential Debugging Queries 🔧

Keep these patterns handy:

## 1. Error rate by service
error_rate_by_service = """
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/ sum(rate(http_requests_total[5m])) by (service)
"""

## 2. Latency percentiles
latency_percentiles = """
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket[5m])
)
"""

## 3. Recent errors with context
recent_errors = """
level='ERROR' AND
timestamp > now() - 15m
ORDER BY timestamp DESC
LIMIT 50
"""

## 4. Slow traces
slow_traces = """
service='api' AND
duration > 2s AND
timestamp > now() - 1h
SAMPLE 100
"""

## 5. Resource saturation
resource_check = """
(
  cpu_usage_percent,
  memory_usage_percent,
  thread_pool_utilization,
  connection_pool_utilization
)
WHERE timestamp > now() - 30m
"""

📚 Further Study

• Google SRE Book - Monitoring Distributed Systems - Comprehensive guide to observability and debugging at scale
• The USE Method: Linux Performance Checklist - Systematic methodology for resource debugging (Utilization, Saturation, Errors)
• Distributed Tracing in Practice - Deep dive into using traces for debugging microservices

Mastering production debugging transforms you from reactive firefighter to proactive investigator. The key is systematic, hypothesis-driven investigation using the right signals at the right time. Practice these skills on smaller issues to build the muscle memory needed during critical incidents. 🚀