• Author: Amin Boulouma, Software Engineer
• Github source code: https://github.com/aminblm/ai_systems_design_from_scratch
• Engineering Blog: https://aminblm.github.io/ai_systems_design_from_scratch/blog/

The Resilience Gap: Why Your Retries Are Likely Killing Your Production

We’ve all been there: the 2:00 AM pager alert signaling a “midnight deployment spike.” Your dashboard is a sea of red, latency is trending toward infinity, and your database CPU is pegged at 100%. The instinctual reaction? Scale up and add more retries to the failing service.

This is the single most dangerous mistake in distributed systems engineering.

In this post, we’ll dismantle the naive retry pattern and implement a strategy that protects your infrastructure from the dreaded “thundering herd.”

The Problem: The Retry Feedback Loop

When a service experiences a transient failure, adding simple, infinite retries creates a positive feedback loop. The failing service consumes resources trying to recover, and your upstream services, by retrying aggressively, increase the load on an already buckling system.

Why we choose Exponential Backoff with Jitter over naive Retries: A naive retry sends requests at fixed, predictable intervals. If 1,000 clients fail at once, they will all retry simultaneously, creating a synchronized “thundering herd” that ensures the downstream service remains offline.

Implementation

Simple Example: The Naive Approach (Avoid this)

import time

def call_service():
    # Simulate flaky dependency
    raise ConnectionError("Service Down")

def risky_retry():
    for i in range(3):
        try:
            return call_service()
        except ConnectionError:
            time.sleep(1) # Predictable, synchronized interval

Complex Example: Production-Grade Resiliency

A robust implementation requires Exponential Backoff (increasing wait time) and Jitter (randomizing timing to desynchronize clients).

import time
import random

class ResilientClient:
    def __init__(self, max_retries=5, base_delay=0.1, max_delay=10):
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.max_delay = max_delay

    def execute(self, func):
        for attempt in range(self.max_retries):
            try:
                return func()
            except Exception as e:
                if attempt == self.max_retries - 1:
                    raise e
                
                # Exponential Backoff: base * 2^attempt
                # Full Jitter: random between 0 and delay
                delay = min(self.max_delay, self.base_delay * (2 ** attempt))
                sleep_time = random.uniform(0, delay)
                
                print(f"Attempt {attempt} failed, retrying in {sleep_time:.2f}s")
                time.sleep(sleep_time)

Quick Reference: Resilience Strategies

Strategy	When to use	Why?
Naive Retry	Never	Causes synchronized thundering herds.
Exponential Backoff	Mild latency spikes	Reduces pressure gradually.
Backoff + Jitter	High-concurrency systems	Breaks synchronization between clients.
Circuit Breaker	Persistent outages	Stops traffic to failing services to allow recovery.

Developer Checklist

• Are retries bounded? (Never retry indefinitely)
• Is there Jitter? (Are your clients synchronized?)
• Is there a Circuit Breaker to stop the bleeding if retries fail?
• Is observability in place? (Can you visualize the retry count vs. total requests?)

Final Takeaways

1. Never trust your retry logic. If it doesn’t include jitter, it is a liability.
2. Fail Fast. Sometimes, returning an error to the user is better than keeping a broken connection open.
3. Decouple the failure. Use queues or circuit breakers to ensure that a localized fault doesn’t become a system-wide outage.