Rate Limiting and Caching at the API Gateway Layer: Protecting Your Backend From Its Own Traffic

Rate Limiting and Caching at the API Gateway Layer: Protecting Your Backend From Its Own Traffic

Most outages aren't caused by attackers. They're caused by your own traffic, a mobile app retrying too aggressively on a bad connection, a partner integration polling every second instead of every minute, or a single enthusiastic user's script hammering an endpoint that was never designed to be called that often. None of that is malicious, and all of it can take a backend down just as effectively as an actual attack.

Rate limiting and caching, enforced at the API gateway, the single layer every request passes through before it reaches your application, are how you stop that from happening without writing defensive code into every endpoint by hand. This is a concept-first walkthrough of how the two work together, using plain examples instead of implementation code, because the ideas matter more than any specific library.

Why the gateway, and not the application code

Every request to your API has to pass through one door before it reaches anything else , the gateway. That makes it the cheapest place to stop a problem, because rejecting a request there costs almost nothing: no database connection opened, no business logic run, no code path touched beyond "check a counter and decide." Handling the same problem inside application code means every endpoint needs its own defensive logic, which is inconsistent to maintain and, worse, only kicks in after the request has already consumed a database connection and some CPU time getting there.

Think of the gateway as a building's front desk rather than trusting every office inside the building to independently decide who's allowed up the elevator. One consistent checkpoint, one set of rules, enforced before anyone reaches the parts of the building that are expensive to disturb.

What rate limiting actually controls

Rate limiting answers one question: how many requests is this client allowed to make in a given window of time? When a client goes over that limit, the gateway rejects the extra requests with an HTTP 429 status , "Too Many Requests" , instead of forwarding them to your application, and it typically includes a header telling the client exactly when it's allowed to try again. That header matters more than it looks: it turns "you got blocked" into "you got blocked, and here's exactly when to try again," which is the difference between a client that backs off gracefully and one that immediately retries and makes the problem worse.

Not every limit needs to be the same shape. A few common patterns, and what they're each good for:

• A steady cap per time window , good for general endpoints where you mainly want to prevent one client from monopolizing capacity, not micromanage burst behavior.
• A smoother, rolling limit , good when you want to avoid the awkward edge case of a client sending nothing for 59 seconds, then a full window's worth of requests in one burst right as the window resets.
• A limit that allows short bursts but refills gradually , good for real-world usage patterns like a user scrolling quickly through a feed, then pausing, where you want to tolerate the burst without permanently raising the baseline limit.

The right choice depends less on theory and more on how your actual clients behave , a chat app's message-sending pattern looks nothing like a bulk data-import tool's, and a single limit type rarely fits both well.

What actually happens with a real rate limit in practice

Picture a search endpoint on a SaaS product. Under normal use, a logged-in user searches a handful of times a minute , well within a reasonable limit. Then one day a browser extension a user installed starts auto-searching on every keystroke instead of waiting for them to finish typing, sending forty requests in ten seconds. Without a rate limit, that single misbehaving client competes for the exact same database connections and CPU time as every other user on the platform, and everyone's search feels slightly slower during that window , for a problem caused by one client, invisible to everyone else.

With a rate limit at the gateway, that client hits its cap, starts receiving 429 responses with a clear retry time, and every other user's traffic is completely unaffected. Common starting points for production REST APIs are 60 to 300 requests per minute for general endpoints, a much lower 10 to 30 requests per minute for expensive write or search operations, and 1,000 to 5,000 requests per minute for lightweight, cacheable read endpoints. The specific numbers matter less than the principle: expensive operations get tighter limits than cheap ones, because the cost of one bad client is proportional to how much work each of their requests actually triggers.

Where caching does the other half of the job

Rate limiting controls how often a client can ask for something. Caching controls how expensive it is when they do. The two solve related but different problems, and using only one leaves an obvious gap.

Consider a product listing endpoint that a thousand different users all request within the same minute, all asking for essentially the same data , the current catalog. Without caching, that's a thousand separate trips to the database doing the same work a thousand times over. With a cache sitting at the gateway, the first request does the real work and every subsequent request for the same data, for as long as the cache entry stays valid, gets served instantly without ever reaching the database , the same result, delivered nearly a thousand times cheaper.

This is where the two techniques reinforce each other directly: a well-cached endpoint can safely support a much higher rate limit, because most of the requests within that limit never generate real backend work at all. An endpoint with no caching needs a much stricter limit, because every single request costs the same regardless of how many times the same answer was already given a second ago.

Setting limits without guessing

The most common mistake isn't picking the wrong algorithm , it's picking limit numbers with no relationship to what the backend can actually absorb. A limit set too loose doesn't protect anything; a limit set too tight blocks legitimate users and generates support tickets. The fix is treating it as a measurement problem, not a guess:

1. Look at how your real traffic actually behaves before setting any number , typical request volume per user, per integration, and during peak hours, so the limit reflects reality instead of a round number that felt reasonable.
2. Set tighter limits on expensive operations and looser limits on cheap, cacheable ones, since the whole point is matching the cost of an operation to how freely it can be requested.
3. Give different tiers of clients different limits, so a paying enterprise integration isn't throttled at the same threshold as an anonymous free-tier user hitting the same endpoint.
4. Publish the limits and expose usage in response headers, so well-behaved clients can see how close they are to a limit and back off on their own instead of finding out only after getting blocked.
5. Watch what actually happens after you deploy a limit , a limit that's constantly triggering false positives against real users needs adjusting just as much as one that never triggers at all.

A simple decision model

Add caching first when: the same data is being requested repeatedly by many different clients, and that data doesn't change on every single request , product catalogs, configuration, anything read far more often than it's written.

Add rate limiting first when: individual clients can trigger genuinely unique, non-cacheable work on every request , searches with different terms, write operations, anything where caching can't help because there's nothing repeated to serve from a cache.

Use both, at the gateway, when your API serves a mix of clients you don't fully control , public APIs, mobile apps on unpredictable networks, third-party integrations , which describes almost every production API serving more than a handful of trusted internal clients.

Wrapping up

Rate limiting and caching are often talked about as separate concerns, but at the gateway they're really two halves of the same job: making sure your backend's real capacity gets spent on the traffic that actually needs it, instead of being consumed unevenly by whichever client happens to be loudest, buggiest, or least well-behaved that day. Neither one requires exotic infrastructure to start , what matters is getting both enforced consistently at the one layer every request has to pass through, before any of that traffic reaches the parts of your system that are expensive to fix after the fact.

This sits on the same principle covered in handling users smoothly from the Flutter client to the backend , per-user rate limiting was exactly the tool for containing a misbehaving minority of clients at scale , and the underlying infrastructure choices are the same ones covered in building my own Firebase alternative.

Related Posts

• From 0 to 100K Users: Handling Users Smoothly from the Flutter Client to the Backend
• The Backup Strategy That Actually Protects Your VPS Data (And Why Automation Isn't Optional)
• Building My Own Firebase Alternative (flexdocs) , Self-Hosting a Realtime BaaS with Docker, Nginx & MongoDB