Performance

Use Throttling and Rate Limiting to Prevent Overload and Improve Stability

Manuel A Rodriguez

— 11 min read

Use Throttling and Rate Limiting to Prevent Overload and Improve Stability

Throttling and rate limiting control the rate of operations (requests, calls, messages), preventing overload and improving system stability. By limiting how many operations are allowed per time window, you protect backends, databases, and external APIs from being overwhelmed. The trade-off: rate limiting can cap maximum throughput and may reject valid requests when limits are exceeded. Use throttling for public APIs, resource-limited systems, backend protection, and applications consuming external APIs. Avoid aggressive rate limiting when maximum throughput is required and resources can handle the load.

Executive Summary (TL;DR)

Throttling and rate limiting control the rate of operations (requests, calls, messages), preventing overload and improving system stability. By limiting how many operations are allowed per time window, you protect backends, databases, and external APIs from being overwhelmed. The trade-off: rate limiting can cap maximum throughput and may reject valid requests when limits are exceeded. Use throttling for public APIs, resource-limited systems, backend protection, and applications consuming external APIs. Typical benefits: prevents overload, better stability, protects resources, better user experience under load. Common mistakes: no rate limiting on public APIs, limits too strict or too loose, not returning proper HTTP status (429), not distinguishing between throttling and rate limiting.

Problem Context

Understanding the Basic Problem

What happens when you don't use rate limiting?

Imagine a public API that accepts unlimited requests per second:

// ❌ Bad: No rate limiting - unlimited requests
[ApiController]
public class BadUnlimitedApiController : ControllerBase
{
    [HttpGet("data")]
    public async Task<IActionResult> GetData()
    {
        // Every request hits the database - no limit
        var data = await _db.QueryAsync("SELECT * FROM LargeTable");
        return Ok(data);
        // What happens: Traffic spike = 10,000 req/sec = database overload = system crash
    }
}

What happens:

  • No limit: Every request is accepted and processed
  • Traffic spike: Sudden increase in traffic (e.g., viral post, bot attack) sends many requests
  • Resource exhaustion: Database, CPU, or memory is overwhelmed
  • Cascading failure: System becomes slow or crashes, affecting all users
  • Performance impact: No protection; stability degrades under load

Why this is bad:

  • Overload: Backend cannot handle unlimited concurrent requests
  • No fairness: A few clients can consume all resources
  • No stability: System has no guardrail against traffic spikes
  • Poor UX: When overloaded, everyone gets slow or failed responses

With rate limiting:

// ✅ Good: Rate limiting - control request rate
[ApiController]
public class GoodRateLimitedApiController : ControllerBase
{
    private readonly RateLimiter _rateLimiter;

    public GoodRateLimitedApiController()
    {
        _rateLimiter = new SlidingWindowRateLimiter(new SlidingWindowRateLimiterOptions
        {
            PermitLimit = 100,
            Window = TimeSpan.FromSeconds(1),
            SegmentsPerWindow = 2
        });
    }

    [HttpGet("data")]
    public async Task<IActionResult> GetData()
    {
        using var lease = await _rateLimiter.AcquireAsync();
        if (!lease.IsAcquired)
            return StatusCode(429, "Too Many Requests"); // Reject when limit exceeded

        var data = await _db.QueryAsync("SELECT * FROM LargeTable");
        return Ok(data);
        // What happens: Max 100 req/sec = database protected = stable system
    }
}

What happens:

  • Limit enforced: Only a fixed number of requests per time window are allowed
  • Excess rejected: Requests beyond the limit get 429 Too Many Requests (or wait, depending on policy)
  • Backend protected: Database and CPU are not overwhelmed
  • Stability: System remains stable under traffic spikes
  • Benefit: Prevents overload; improves stability and predictability

Improvement: Prevents overload and degradation; improves stability and user experience under load.

Key Terms Explained (Start Here!)

What is throttling? Limiting the rate at which operations (requests, calls) are processed. Throttling can mean slowing down requests (e.g., queueing) or rejecting excess requests. Example: Allow at most 100 requests per second; reject or delay the rest.

What is rate limiting? A form of throttling that enforces a maximum number of operations per time window. When the limit is reached, additional operations are rejected (or delayed). Example: 100 requests per minute per user.

What is a rate limiter? A component that tracks how many operations have occurred in a time window and decides whether to allow or reject new operations. Example: .NET RateLimiter (e.g., SlidingWindowRateLimiter, TokenBucketRateLimiter).

What is a token bucket? A rate-limiting algorithm that refills "tokens" at a fixed rate. Each operation consumes one token; if no tokens are available, the operation is rejected or delayed. Example: Bucket with 100 tokens, refill 10 per second → average 10 ops/sec, burst up to 100.

What is a sliding window? A rate-limiting algorithm that counts operations in a rolling time window (e.g., last 1 second). Limits are enforced over that window. Example: Max 100 requests in any 1-second window.

What is a fixed window? A rate-limiting algorithm that counts operations in fixed, non-overlapping time windows (e.g., 0–60 sec, 60–120 sec). Simpler than sliding window but can allow bursts at window boundaries. Example: Max 100 requests per minute, window resets at minute boundary.

What is overload? When a system receives more work than it can handle, leading to high latency, errors, or crash. Example: Database receives 10,000 queries/sec but can only handle 1,000 → overload.

What is 429 Too Many Requests? HTTP status code indicating the client has sent too many requests in a given time (rate limit exceeded). Clients should retry after a delay. Example: API returns 429 when user exceeds 100 requests/minute.

What is throughput? The number of operations completed per unit of time. Rate limiting caps maximum throughput to protect the system. Example: 1000 requests per second = throughput.

What is backpressure? A mechanism where a system signals to producers to slow down when it cannot keep up. Rate limiting is one way to apply backpressure (reject or delay requests). Example: API returns 429 → clients slow down or retry later.

Common Misconceptions

"Rate limiting only hurts performance"

  • The truth: Rate limiting trades maximum throughput for stability. Without it, overload can cause everyone to get slow or failed responses. With it, some requests are rejected but the system stays stable.

"Rate limiting is only for public APIs"

  • The truth: Rate limiting is useful anywhere you need to protect a limited resource: internal services, database callers, consumers of external APIs, etc.

"Throttling and rate limiting are the same"

  • The truth: Rate limiting is a type of throttling. Throttling can also mean slowing down (e.g., queueing) rather than hard limiting. Often used interchangeably in practice.

"Strict limits are always better"

  • The truth: Limits that are too strict reject valid traffic and hurt UX. Limits should be tuned to resource capacity and acceptable load.

"Rate limiting is only for abuse prevention"

  • The truth: Rate limiting also prevents accidental overload (e.g., buggy client, traffic spike) and ensures fair sharing of resources.

How It Works

Understanding How Rate Limiting Works

Token bucket (conceptual):

Tokens: 100 (max)
Refill: 10 tokens per second

Request arrives → Consume 1 token
  - If tokens > 0: Allow request, tokens--
  - If tokens = 0: Reject request (or wait for refill)
Every second: tokens = min(100, tokens + 10)

What happens:

  • Burst: Up to 100 requests can be served immediately (bucket full)
  • Sustained: After burst, only 10 requests per second (refill rate)
  • Protection: Prevents sustained overload; allows short bursts

Sliding window (conceptual):

Window: last 1 second
Limit: 100 requests per window

Request arrives → Count requests in last 1 second
  - If count < 100: Allow request, increment count
  - If count >= 100: Reject request
Window slides: only requests in [now-1s, now] are counted

What happens:

  • Smooth limit: No sudden reset at a fixed boundary (unlike fixed window)
  • Fair: Limits are enforced over a rolling period
  • Protection: Caps rate over any 1-second interval

Key insight: Rate limiters track usage over time and allow or reject each operation based on the chosen algorithm (token bucket, sliding window, fixed window, etc.).

Technical Details: .NET Rate Limiting (ASP.NET Core 7+)

Sliding window rate limiter:

using System.Threading.RateLimiting;

var options = new SlidingWindowRateLimiterOptions
{
    PermitLimit = 100,           // Max 100 operations
    Window = TimeSpan.FromSeconds(1),  // Per 1-second window
    SegmentsPerWindow = 2        // Internal segments for sliding behavior
};
var rateLimiter = new SlidingWindowRateLimiter(options);

Using the limiter:

using var lease = await rateLimiter.AcquireAsync();
if (lease.IsAcquired)
{
    // Process request
}
else
{
    // Reject (e.g., return 429) or wait
}

Token bucket (conceptual in .NET):

var options = new TokenBucketRateLimiterOptions
{
    TokenLimit = 100,                    // Max tokens (burst)
    ReplenishmentPeriod = TimeSpan.FromSeconds(1),
    TokensPerPeriod = 10                 // Refill rate
};
var rateLimiter = new TokenBucketRateLimiter(options);

Key insight: .NET 7+ provides built-in rate limiters; use them in middleware or in controllers to enforce limits per endpoint, per user, or globally.

Why This Becomes a Bottleneck

Not using rate limiting becomes a problem because:

Overload: Unlimited requests can overwhelm the backend. Example: 10,000 requests/sec to a database that handles 1,000/sec → database overload → timeouts and errors.

Cascading failure: One overloaded component can bring down the whole system. Example: Database slow → threads block → thread pool exhausted → all requests fail.

No fairness: A few clients (or one buggy client) can consume all resources. Example: One client sending 50,000 req/sec starves others.

Unpredictable performance: Under load, latency and error rate spike with no bound. Example: Normal 50 ms, under spike 30 s or timeout.

Resource exhaustion: CPU, memory, connections, or file descriptors can be exhausted. Example: Too many concurrent connections → "too many open files" or OOM.

Poor UX: When the system is overloaded, all users get bad experience. Example: Rate limiting rejects some requests but keeps the system responsive for accepted ones.

Advantages

Prevents overload: Limits the rate of work so backends are not overwhelmed. Example: Cap at 100 req/sec → database stays within capacity.

Better stability: System behavior is predictable under traffic spikes. Example: Spike to 10,000 req/sec → only 100/sec accepted → system stays stable.

Protects resources: Databases, external APIs, and CPU are protected from excessive load. Example: Rate limit calls to external API to avoid hitting provider limits and bans.

Better UX under load: Accepted requests get consistent latency; rejected ones get clear signal (e.g., 429) to retry later. Example: Better than everyone getting timeouts.

Fairness: Pre-user or per-client limits ensure no single consumer takes all capacity. Example: 100 req/min per user → fair sharing.

Abuse and accident protection: Mitigates abuse (e.g., scrapers) and accidental overload (e.g., buggy client loop). Example: Runaway script capped at N req/sec.

Disadvantages and Trade-offs

Can limit throughput: Hard cap on operations per second. Example: Need 2000 req/sec but limit is 1000 → maximum throughput is 1000.

Can reject valid requests: When limit is exceeded, valid clients get 429 or similar. Example: Legitimate burst of traffic gets rejected.

Requires configuration: Must choose limits, windows, and policies. Example: Too low = reject too much; too high = insufficient protection.

Adds latency or complexity: Checking limits and possibly waiting or rejecting. Example: Acquire lease, check IsAcquired, return 429.

Per-user vs global: Per-user limits are fair but more complex (need identity); global limits are simple but one user can fill the bucket. Example: 100/min per user vs 10,000/min total.

When to Use This Approach

Use throttling and rate limiting when:

  • Public APIs (APIs exposed to many or unknown clients). Example: REST API for mobile app. Limit per user or per IP to prevent abuse and overload.

  • Resource-limited systems (database, external API, or CPU with known capacity). Example: Database handles 1000 queries/sec → limit incoming requests to ~1000/sec.

  • Backend protection (preventing a service from being overwhelmed). Example: Rate limit ingress to a payment service so it never exceeds capacity.

  • Consuming external APIs (third-party APIs with rate limits or cost). Example: External API allows 100 calls/min → limit your outgoing calls to stay under that.

  • Fairness (ensuring one client does not starve others). Example: Per-user or per-tenant limits in a multi-tenant API.

Recommended approach:

  • Public APIs: Always use rate limiting (per user, per IP, or both).
  • Internal services: Use when downstream has limited capacity.
  • Outgoing calls: Use when calling rate-limited or costly external APIs.
  • Tune limits: Set limits based on capacity and load tests; expose metrics (e.g., reject count) and adjust.

When Not to Use It

Don't rely on rate limiting (or use very high limits) when:

  • Maximum throughput is required and resources can handle the load. Example: Internal batch job that must process as fast as possible on a dedicated cluster.

  • Traffic is fully trusted and controlled (e.g., internal only, single known client). Example: Internal service-to-service call with no risk of spike.

  • System is over-provisioned and overload is not a concern. Example: Small internal tool with few users.

Note: Even in these cases, a very high limit can still protect against bugs (e.g., infinite loop). The question is how strict the limit should be.

Performance Impact

Typical impact of rate limiting:

  • Stability: Prevents overload and cascading failure; keeps latency and error rate bounded under traffic spikes. Impact is on stability and predictability, not on peak throughput of the limited path.

  • Throughput: Caps maximum throughput at the limit. Example: Limit 1000 req/sec → throughput cannot exceed 1000/sec for that path.

  • Latency for accepted requests: Usually minimal (one check per request). For "wait" policies, latency increases when the limit is hit (request waits for permit).

  • Rejected requests: Get 429 (or similar) and no backend work; they save server resources but require client retry logic.

Important: The main benefit is avoiding degradation and outage under load, not making the happy path faster. Tune limits so that under normal load almost no requests are rejected, and under spike the system stays stable.

Common Mistakes

No rate limiting on public APIs: Exposing an API without any limit. Example: Public endpoint that hits the database with no cap → one spike can take down the system.

Limits too strict or too loose: Too strict → reject valid traffic; too loose → insufficient protection. Example: 10 req/min for a normal app is too strict; 1M req/min may be too loose. Tune with metrics and load tests.

Not returning 429 (or equivalent): Rejecting requests but not clearly signaling "rate limited". Example: Returning 503 or 500 for rate limit → clients don't know to retry after delay. Use 429 and Retry-After when appropriate.

Confusing throttling with rate limiting: Using terms interchangeably is fine, but implement clearly: hard limit (reject) vs. slow down (queue/delay). Example: Document whether excess requests are rejected or queued.

One global limit only: Single global limit can let one user consume everything. Example: 1000 req/min total → one user can use all 1000. Consider per-user or per-key limits for fairness.

Ignoring limits when calling external APIs: Not throttling outgoing calls to a rate-limited API. Example: Sending 1000 req/sec to an API that allows 100/min → get blocked or banned.

How to Measure and Validate

Track request rate, reject rate, latency, and backend utilization:

  • Request rate: Incoming requests per second (total and per user if applicable).
  • Reject rate: Requests rejected by rate limiter (429 or similar) per second.
  • Latency: p50, p95, p99 for accepted requests; should remain stable under load.
  • Backend utilization: CPU, DB connections, external API usage; should stay within capacity when rate limiting is applied.

Practical validation:

  1. Define limits: Choose limits based on backend capacity and target load.
  2. Add rate limiting: Apply limiter at API edge or before heavy work.
  3. Load test: Send traffic above the limit; verify reject rate and that backend does not overload.
  4. Tune: Adjust limits and windows so that under normal load reject rate is low and under spike the system stays stable.

Tools: Application metrics (request count, 429 count), load testing (e.g., k6, JMeter), APM to observe latency and errors under load.

Example Scenarios

Scenario 1: Public API with per-user rate limit

Problem: Public API must stay stable under traffic spikes and prevent any single user from consuming all capacity.

Bad approach (no rate limiting):

// ❌ Bad: No rate limiting
[HttpGet("items")]
public async Task<IActionResult> GetItems()
{
    var items = await _db.GetItemsAsync();
    return Ok(items);
}

Good approach (sliding window per user):

// ✅ Good: Rate limit per user (e.g., via middleware or endpoint filter)
[HttpGet("items")]
[EnableRateLimiting("PerUser")]
public async Task<IActionResult> GetItems()
{
    var items = await _db.GetItemsAsync();
    return Ok(items);
}
// Configure "PerUser" policy: e.g., 100 requests per minute per user

Results:

  • Bad: Traffic spike or abusive user can overload the database.
  • Good: Each user capped (e.g., 100/min); excess get 429; system stays stable.

Scenario 2: Protecting a backend with global limit

Problem: Backend service can handle 500 req/sec; front-end must not send more.

Bad approach (no throttling):

// ❌ Bad: Send all requests to backend
public async Task<Response> CallBackendAsync(Request req)
{
    return await _httpClient.PostAsync(_backendUrl, req); // No limit
}

Good approach (global rate limiter):

// ✅ Good: Throttle outgoing requests
private readonly RateLimiter _limiter = new SlidingWindowRateLimiter(
    new SlidingWindowRateLimiterOptions
    {
        PermitLimit = 500,
        Window = TimeSpan.FromSeconds(1)
    });

public async Task<Response> CallBackendAsync(Request req)
{
    using var lease = await _limiter.AcquireAsync();
    if (!lease.IsAcquired)
        throw new RateLimitExceededException("Backend limit reached");
    return await _httpClient.PostAsync(_backendUrl, req);
}

Results:

  • Bad: Burst of 2000 req/sec can overwhelm backend.
  • Good: At most 500 req/sec sent; backend stays within capacity.

Scenario 3: Consuming an external API with rate limit

Problem: External API allows 100 calls per minute; your app must not exceed that.

Approach (outgoing rate limit):

// ✅ Good: Limit outgoing calls to external API
private readonly RateLimiter _limiter = new FixedWindowRateLimiter(
    new FixedWindowRateLimiterOptions
    {
        PermitLimit = 100,
        Window = TimeSpan.FromMinutes(1)
    });

public async Task<ExternalData> FetchFromExternalApiAsync(string id)
{
    using var lease = await _limiter.AcquireAsync();
    if (!lease.IsAcquired)
        throw new RateLimitExceededException("External API limit reached; retry later.");
    return await _externalHttpClient.GetAsync($"/data/{id}");
}

Results: Outgoing calls stay under 100/minute; you avoid being throttled or banned by the provider.

Summary and Key Takeaways

Throttling and rate limiting control the rate of operations to prevent overload and improve stability. Use them for public APIs, resource-limited systems, backend protection, and when consuming external APIs. The trade-off: maximum throughput is capped and some valid requests may be rejected when limits are exceeded. Typical benefits: prevents overload, better stability, protects resources, better UX under load. Common mistakes: no rate limiting on public APIs, limits too strict or too loose, not returning 429, one global limit only, ignoring limits when calling external APIs. Always tune limits based on capacity and load; use metrics (request rate, reject rate, latency) to validate. Rate limiting is about stability and fairness, not about making the happy path faster.

Read more