AI Gateway Architecture: LLM Rate Limiting, Caching, and Routing

If your application calls more than one LLM provider, you already have a gateway problem. You just do not have a gateway solution. Every direct integration between your application code and an LLM API is an unmanaged connection with no rate limiting, no caching, no fallback logic, and no centralized observability. Multiply that by five services calling OpenAI, three calling Claude, and two calling Gemini, and you have a sprawl of uncontrolled LLM traffic that is impossible to govern.

An AI gateway sits between your application layer and every LLM provider. All traffic flows through a single control plane where you enforce rate limits, cache responses, route requests intelligently, handle failovers, and collect telemetry. Think of it as what Kong or NGINX does for REST APIs, but purpose-built for the unique challenges of LLM inference: variable latency, token-based pricing, semantic similarity in requests, and provider-specific quirks.

The economics are compelling. Organizations running 1M+ LLM calls per month typically see 30 to 60% cost reductions after implementing a proper AI gateway, primarily through semantic caching and intelligent routing. Beyond cost, you get reliability improvements (automatic failovers when OpenAI has an outage), compliance controls (PII filtering before data leaves your network), and operational visibility (which team, which feature, which user is consuming your LLM budget).

The alternative is what most companies do today: scattered try/catch blocks around API calls, hardcoded model selections, zero caching, and a monthly AWS bill that nobody can explain. That approach works for a prototype. It collapses at production scale. If you are managing LLM API costs across multiple teams and applications, a gateway is not optional. It is infrastructure.

A production AI gateway handles five distinct concerns that would otherwise be scattered across your codebase. Understanding each capability helps you evaluate whether to build or buy, and which solution fits your architecture.

Rate Limiting

LLM providers enforce their own rate limits (OpenAI: 10,000 RPM on Tier 5, Anthropic: 4,000 RPM on standard plans). Your gateway enforces your limits on top of theirs. Per-user limits prevent a single customer from exhausting your quota. Per-feature limits ensure that your chatbot cannot starve your document processing pipeline. Burst handling absorbs traffic spikes without dropping requests.

Response Caching

Unlike traditional API caching (exact URL match), LLM caching needs semantic understanding. "What is the capital of France?" and "Tell me France's capital city" should return the same cached response. A proper AI caching layer uses embedding similarity to identify equivalent requests and serves cached responses in under 50ms instead of waiting 2 to 8 seconds for a fresh LLM call.

Intelligent Routing

Not every request needs GPT-4 Turbo at $10 per million input tokens. Simple classification tasks can use Claude Haiku at $0.25 per million tokens. Your gateway evaluates incoming requests and routes them to the optimal model based on complexity, cost constraints, latency requirements, or custom business rules. This single capability often delivers the largest cost savings.

Fallback Chains

When your primary model provider goes down (and they all do, regularly), your gateway automatically routes to a backup. No code changes, no deployments, no 3am pages. The fallback chain fires in milliseconds, your users never notice, and you get an alert to investigate later.

Observability

Every request through the gateway generates telemetry: tokens consumed, latency, model used, cache hit/miss, cost, and custom metadata (user ID, feature name, team). This data feeds dashboards that answer questions like "Which feature consumed 40% of our LLM budget last week?" and "Why did latency spike on Tuesday?"

Traditional caching matches exact strings. If the request body is byte-for-byte identical, serve the cached response. This works for REST APIs but is nearly useless for LLM traffic. Users ask the same question in hundreds of different phrasings, and each phrasing triggers a fresh (expensive) LLM call.

Semantic caching solves this by computing an embedding vector for each incoming prompt, then comparing it against cached prompt embeddings using cosine similarity. If the similarity score exceeds your threshold (typically 0.92 to 0.97), the gateway returns the cached response without calling the LLM. The result: response times drop from 2 to 8 seconds down to 30 to 80 milliseconds, and you pay zero tokens for the cached response.

Implementation Architecture

The caching pipeline has four stages. First, the incoming prompt is embedded using a fast, cheap embedding model (OpenAI text-embedding-3-small at $0.02 per million tokens, or a self-hosted model for zero marginal cost). Second, the embedding is compared against your vector store (Redis with vector search, Qdrant, Pinecone, or pgvector). Third, if a match above threshold is found, the cached response is returned immediately. Fourth, if no match is found, the request flows to the LLM, and both the prompt embedding and the response are stored in the cache.

The similarity threshold is your precision/recall tradeoff. At 0.97, you only cache nearly identical questions (high precision, low hit rate). At 0.92, you cache more aggressively but risk serving responses that do not exactly match the user's intent. Start at 0.95 and tune based on your use case. Customer support FAQ responses tolerate lower thresholds. Code generation requires higher thresholds because small prompt differences produce very different outputs.

Cache Invalidation and TTL

LLM responses are not static. Pricing information changes, product features ship, policies update. Your semantic cache needs TTL (time to live) policies that vary by content type. Factual knowledge that rarely changes (geography, math, definitions) can cache for 30+ days. Dynamic business information (pricing, availability, current events) should cache for hours or less. Some implementations tag cached entries with metadata and invalidate selectively when underlying data changes.

Real-world cache hit rates vary dramatically by use case. Customer support chatbots see 40 to 70% hit rates because users ask the same questions repeatedly. Creative writing assistants see 5 to 15% hit rates because each prompt is genuinely unique. Internal knowledge-base tools land around 25 to 45%. Even at the low end, a 15% cache hit rate on 1M monthly calls saves you 150,000 LLM invocations per month.

The single biggest waste in LLM spending is using expensive models for simple tasks. When every request goes to GPT-4 Turbo or Claude Opus regardless of complexity, you are paying $10 to $15 per million input tokens for work that a $0.25 model handles perfectly. Smart routing fixes this by classifying requests and sending them to the cheapest model that can handle them well.

Routing Strategies

Complexity-based routing uses a lightweight classifier (often a small model like Haiku or a fine-tuned BERT) to evaluate the incoming request and assign a complexity score. Simple requests (classification, extraction, formatting) route to cheap, fast models. Medium requests (summarization, Q&A with context) route to mid-tier models. Complex requests (multi-step reasoning, creative generation, code architecture) route to flagship models. This approach typically reduces costs by 40 to 55% with less than 3% quality degradation on aggregate.

Latency-based routing sends time-sensitive requests (real-time chat, autocomplete) to the fastest available model, even if it costs slightly more. Background tasks (batch processing, nightly reports) route to the cheapest option regardless of latency. This is particularly useful for applications that mix real-time and batch workloads.

Cost-budget routing assigns each request or user session a cost ceiling. The gateway tracks cumulative spending and progressively routes to cheaper models as the budget depletes. A customer support session might start with Sonnet for the initial complex question, then drop to Haiku for follow-up clarifications. For a deeper dive into routing economics, see our AI model routing guide.

Quality Verification

Routing to cheaper models only works if you verify quality. Implement a sampling-based quality check: route 5% of cheap-model responses through the flagship model in parallel, compare outputs, and track quality drift. If the cheap model's quality drops below your threshold on a specific category of request, update your routing rules to send that category to a more capable model. This feedback loop ensures routing stays calibrated as models update and traffic patterns shift.

Rate limiting LLM traffic is fundamentally different from limiting REST API calls. A single LLM request can cost 100x more than another depending on token count. A request that generates 4,000 output tokens is not equivalent to one that generates 50. Your rate limiting strategy needs to account for this asymmetry.

Per-User Token Budgets

Rather than limiting requests per minute (which treats a 50-token query the same as a 10,000-token generation), limit by tokens consumed. Assign each user a token budget: 100,000 input tokens and 50,000 output tokens per hour, for example. The gateway tracks token consumption in real-time and rejects requests when the budget is exhausted. This prevents a single user running a batch job from consuming your entire monthly LLM budget in an afternoon.

Tiered Rate Limits

Match rate limits to your pricing tiers. Free tier users get 10,000 tokens per day. Pro users get 500,000. Enterprise gets custom limits with burst allowances. The gateway enforces these limits transparently, returning 429 responses with clear information about when the limit resets and how to upgrade. This rate limiting doubles as a monetization lever.

Burst Handling and Queue Management

Traffic spikes are inevitable. A product launch, a viral moment, or simply Monday morning across time zones can spike LLM traffic 10x above baseline. Rather than hard-rejecting requests during bursts, implement a request queue with priority levels. High-priority requests (paid users, real-time features) process immediately. Lower-priority requests queue and process as capacity becomes available. Set queue depth limits and TTLs so that stale requests do not accumulate indefinitely.

Provider-Aware Limiting

Your gateway must track your usage against each provider's limits independently. If you are at 90% of your OpenAI rate limit, new requests should route to Claude or Gemini rather than queueing. This provider-aware approach maximizes throughput across all available capacity. The gateway maintains real-time counters for each provider and factors remaining capacity into routing decisions. When comparing gateway solutions for this capability, our API gateway comparison covers how traditional gateways handle similar multi-backend routing.

Every LLM provider experiences outages. OpenAI had 12 significant incidents in 2025. Anthropic had 8. Google Cloud had 6. If your application depends on a single provider, you inherit their availability as your SLA ceiling. With a fallback chain, you compose multiple providers into an aggregate availability that exceeds any individual provider.

Configuring Fallback Chains

A typical fallback chain for a production application: primary model is Claude Sonnet (best quality/cost ratio for the use case), first fallback is GPT-4 Turbo (comparable quality, different provider), second fallback is Gemini 1.5 Pro (different infrastructure entirely), emergency fallback is a self-hosted Llama model (guaranteed availability, lower quality). The gateway attempts each in sequence, with configurable timeout thresholds at each level. If Claude does not respond within 10 seconds, try GPT-4 Turbo. If that fails within 8 seconds, try Gemini. If all cloud providers are down, fall back to your self-hosted model.

Health Checks and Proactive Routing

Do not wait for a request to fail before activating fallbacks. Implement continuous health checks that probe each provider every 30 seconds with a lightweight test request. Track response times, error rates, and quality scores over rolling windows. When a provider's health score degrades (increasing latency, elevated error rate), the gateway proactively shifts traffic before users experience failures. This "warm failover" approach means your users never see the outage.

Response Normalization

Different providers return responses in different formats. OpenAI uses one schema, Anthropic another, Google yet another. Your gateway should normalize all responses into a consistent format that your application consumes. This decouples your application code from any specific provider's API contract. When you switch providers (whether due to a fallback or a strategic decision), your application code does not change. The gateway handles the translation layer.

Composite availability math: if Provider A has 99.5% uptime and Provider B has 99.5% uptime, a properly configured fallback chain (where B handles A's downtime) gives you 99.9975% theoretical availability. In practice, you will not hit that theoretical number due to correlated failures and switchover latency, but 99.95% is realistic and significantly better than any single provider.

The AI gateway market has matured rapidly. You have serious options ranging from fully managed services to open-source self-hosted solutions. Here is how they compare for production workloads.

Portkey

The most feature-complete managed AI gateway as of early 2030. Portkey handles routing, caching, fallbacks, rate limiting, and observability in a single platform. Supports 200+ LLM providers. Pricing is usage-based (per gateway request), which can get expensive at high volumes but eliminates operational overhead. Best for teams that want turnkey functionality without managing infrastructure. Their semantic caching implementation is particularly strong, with configurable similarity thresholds and automatic TTL management.

LiteLLM

Open-source Python library and proxy server that provides a unified interface to 100+ LLM providers. LiteLLM excels at response normalization and basic routing/fallbacks. You self-host it, so infrastructure is your responsibility, but you get full control and zero per-request costs beyond your own compute. Great for teams with DevOps capacity who want to avoid vendor lock-in. Lacks built-in semantic caching (you add your own) but handles rate limiting and fallback chains well.

Helicone

Focused primarily on observability and cost tracking, with growing gateway capabilities. Helicone's strength is its analytics: detailed cost breakdowns by feature, user, and model, plus quality scoring and prompt versioning. It integrates as a proxy (one-line code change) and adds caching and rate limiting on top. Best for teams whose primary pain point is "we cannot see where our LLM budget goes." The caching is exact-match only (no semantic), which limits its cost-saving potential compared to Portkey.

Cloudflare AI Gateway

Part of Cloudflare's Workers AI platform. Leverages their global edge network for extremely low-latency caching and rate limiting. Basic routing and fallback capabilities. The advantage is latency: cached responses serve from edge locations worldwide in under 20ms. The disadvantage is limited routing intelligence compared to specialized AI gateways. Best for teams already on Cloudflare's platform who want gateway basics without adding another vendor.

OpenRouter

Operates as both a model marketplace and a routing layer. OpenRouter provides access to 100+ models through a single API with automatic pricing optimization. It handles model selection but gives you less control over caching, rate limiting, and observability compared to dedicated gateway solutions. Useful as a component within a larger gateway architecture rather than a complete solution.

Custom Build

For organizations with strict compliance requirements, unique routing logic, or massive scale (10M+ monthly requests), building a custom gateway on top of Kong, Envoy, or AWS API Gateway may be justified. You get total control but inherit significant engineering and maintenance costs. Budget 3 to 6 months of engineering time for a production-grade implementation with semantic caching, smart routing, and comprehensive observability. Most teams should start with a managed solution and migrate to custom only when they hit concrete limitations.

The numbers on AI gateway ROI are consistent across our client engagements. Organizations spending $10,000+ per month on LLM APIs see 30 to 60% cost reductions within the first month of gateway deployment. The savings come from three sources: semantic caching eliminates 15 to 40% of redundant calls, smart routing reduces per-request costs by 30 to 50% on routed traffic, and rate limiting prevents budget overruns from runaway processes or abusive users.

A concrete example: one of our clients processed 2.4 million LLM calls per month, spending $47,000 on a mix of OpenAI and Anthropic APIs. After implementing a gateway with semantic caching (0.94 similarity threshold) and complexity-based routing, their monthly spend dropped to $19,500. Semantic caching handled 34% of requests from cache. Routing shifted 52% of remaining requests from flagship models to mid-tier models with no measurable quality impact. The gateway infrastructure itself cost $800 per month (self-hosted LiteLLM on existing Kubernetes cluster plus a managed vector database for cache embeddings).

Implementation Roadmap

Week 1: Deploy a basic proxy gateway (LiteLLM or Portkey) with logging only. No caching, no routing, just observability. Understand your traffic patterns, peak loads, and cost distribution across models and features.

Week 2 to 3: Implement rate limiting and basic fallback chains. Set per-user and per-feature token budgets based on observed usage. Configure primary/fallback model pairs for each use case.

Week 4 to 6: Add semantic caching. Start with a conservative similarity threshold (0.96) and monitor cache hit rates and user satisfaction. Gradually lower the threshold as you build confidence. Implement TTL policies by content category.

Week 7 to 8: Deploy smart routing. Build or configure a complexity classifier. Start routing simple requests to cheaper models. Monitor quality with A/B sampling. Tune routing rules based on quality and cost data.

By week 8, you should see 25 to 40% cost savings with improved reliability and full observability. Continued optimization over the following months pushes savings toward 50 to 60% as you tune thresholds, expand caching coverage, and refine routing rules.

If you are spending more than $5,000 per month on LLM APIs and do not have a gateway in place, you are leaving money on the table and accepting unnecessary reliability risk. The implementation is straightforward, the tooling is mature, and the ROI is immediate. Book a free strategy call and we will audit your LLM traffic patterns and recommend the gateway architecture that fits your stack, budget, and compliance requirements.