Feature Flags for AI: Safely Rolling Out LLM Features in Production

Traditional feature flags are binary. You flip a toggle, and a button appears, a new page loads, or a workflow changes. The blast radius of a bad deployment is usually limited to a broken UI or a failed API call. Your monitoring catches it, you roll back, and life goes on.

AI features are fundamentally different. A bad rollout does not just break a page. It can generate offensive content, leak private data through prompt injection, hallucinate medical or legal advice, or run up a five-figure cloud bill in a single afternoon. The failure modes are unpredictable, the consequences are more severe, and traditional rollback strategies are not sufficient because the damage is often done before your error-rate alerts fire.

We have seen this firsthand across dozens of AI product launches. One client pushed a new system prompt to their customer support chatbot on a Friday afternoon. The prompt was tested internally and looked fine. In production, it started apologizing for things the company never did, offering unauthorized refunds, and contradicting the product documentation. By Monday morning, 3,200 support tickets had received inaccurate responses. A percentage-based rollout with quality gates would have caught the issue after the first 50 conversations.

The core problem is that AI outputs are non-deterministic. The same input can produce different outputs across calls, across model versions, and across temperature settings. You cannot fully test an AI feature in staging because staging traffic does not represent the diversity of real user inputs. The long tail of edge cases only appears at production scale. This is why you need a rollout strategy that treats every AI configuration change as a potentially breaking change, with graduated exposure, automated quality checks, and instant kill switches.

The most important pattern for AI rollouts is percentage-based routing. Instead of switching all traffic from Model A to Model B in a single deploy, you route a small percentage of requests to the new model and compare results before increasing exposure. This sounds obvious, but most teams we work with are not doing it. They swap model IDs in an environment variable, deploy, and hope for the best.

How to Structure Model Routing Flags

Your feature flag for a model change should not be a simple boolean. It should be a structured configuration object that your flag system evaluates per-request. At minimum, you need the model identifier, the percentage of traffic allocated to it, a list of user segments included or excluded, and a kill switch that routes 100% of traffic back to the stable model instantly.

In practice, this looks like a flag that says: route 5% of requests to Claude Opus 4, keep 95% on Claude Sonnet 4, exclude enterprise customers from the experiment, and auto-kill if error rate exceeds 2% or p95 latency exceeds 3 seconds. LaunchDarkly, Statsig, and Unleash all support this kind of multi-variate flag with targeting rules. If you are building a custom solution, store these configs in a low-latency key-value store like Redis or DynamoDB so flag evaluation does not add meaningful overhead to your inference pipeline.

Graduated Rollout Schedules

We recommend a standard rollout schedule for model upgrades: 1% for 24 hours, 5% for 48 hours, 25% for 72 hours, 50% for 48 hours, then 100%. At each stage, you check three things: output quality scores from your LLM evaluation pipeline, cost per request compared to the baseline model, and user-facing metrics like satisfaction scores or task completion rates.

This schedule feels slow, and that is the point. A two-week rollout for a model upgrade is dramatically faster than a two-week incident response when a bad model change degrades your product for every user simultaneously. The math is simple: controlled slowness beats uncontrolled damage every time.

Sticky Sessions and Consistency

One detail that trips teams up is session consistency. If a user starts a conversation on Claude Sonnet and their next message gets routed to Claude Opus because of a percentage-based split, the conversation quality and tone will shift mid-stream. Your routing logic needs to be sticky per user session, not per request. Hash the user ID or session ID to determine the routing bucket, and keep that user in the same bucket for the duration of their session. Most feature flag SDKs handle this automatically through consistent hashing, but verify this behavior before relying on it.

Prompt engineering is the most underrated source of production risk in AI systems. A single word change in a system prompt can shift model behavior in ways that are invisible during manual testing and catastrophic at scale. Feature flags let you treat prompt changes with the same rigor you apply to code changes: test in production with controlled exposure, measure impact, and promote or roll back based on data.

What to Test and What to Ship Directly

Not every prompt change needs an A/B test. Minor formatting adjustments, typo fixes, and clarifying rewrites that do not change the intent of the prompt can ship directly. But any change that modifies the model's behavior, tone, output format, or decision-making logic should be tested. This includes adding or removing instructions, changing few-shot examples, adjusting output constraints, modifying guardrails, and updating factual content like product descriptions or policy language.

The distinction matters because prompt A/B testing adds operational overhead. If you test everything, your team will burn out on flag management. If you test nothing, you will eventually push a prompt that breaks your product. Draw the line at behavioral changes and enforce it through code review.

Measuring Prompt Quality in Production

The hard part of prompt A/B testing is defining your success metric. Unlike traditional A/B tests where you measure click-through rates or conversion, AI output quality is multi-dimensional. A prompt variant might produce more accurate responses but take a less friendly tone. It might follow instructions more precisely but generate shorter, less helpful answers.

We use a composite quality score that combines three signals: automated LLM-as-a-judge evaluations (where a separate model scores each response on predefined criteria), user feedback signals (thumbs up/down, regeneration rate, conversation abandonment), and task completion rate (did the user accomplish what they came to do). Weight these signals based on your product priorities. For a customer support bot, task completion matters most. For a content generation tool, the LLM-as-a-judge score on creativity and accuracy matters most.

Statsig is particularly strong here because its experimentation platform natively supports custom metrics and statistical significance calculations. You define your composite score as a metric, assign users to prompt variants through their feature gate system, and Statsig tells you when you have enough data to make a statistically significant decision. This removes the guesswork from "is Variant B actually better, or did we just get lucky with the sample?" For a deeper comparison of these tools, see our feature flag platform comparison.

Standard boolean and percentage flags are just the starting point. AI systems have configuration dimensions that traditional web features do not, and your flag system needs to account for them. Here are the patterns we implement for every client shipping LLM features to production.

Model Version Flags

Every major provider releases model updates that change behavior. OpenAI's model snapshots, Anthropic's version bumps, and Google's Gemini updates can all shift output quality, latency, and cost. Your flag system should treat model version as a first-class configuration parameter, not a hardcoded string in your codebase. When Anthropic releases Claude Opus 4.1, you should be able to route 5% of traffic to the new version without touching your code, evaluate performance through your observability pipeline, and promote to 100% when you are confident. Hardcoding model versions in environment variables means every version change requires a deploy, which means every version change is an all-or-nothing bet.

Temperature and Parameter Flags

Temperature, top-p, max tokens, and other inference parameters significantly affect output quality and cost. A temperature change from 0.7 to 0.3 can turn a creative writing assistant into a rigid template engine. These parameters should be flagged independently so you can tune them in production without redeploying. We have seen teams discover that dropping temperature from 1.0 to 0.4 on their content generation feature reduced user complaints by 40% because the model stopped producing wildly inconsistent outputs. They found this through a flag-controlled parameter sweep, not through guesswork.

System Prompt Variant Flags

Your system prompt is the most impactful configuration in your AI stack, and it should be managed as a flag, not as a code artifact. Store system prompt variants in your flag system with versioning, and route traffic to specific variants based on user segment, feature, or experiment. This gives you instant rollback capability (revert to the previous prompt version without deploying), audit trails (who changed the prompt, when, and what was the impact), and A/B testing capability without engineering involvement. Product managers and prompt engineers should be able to create and test prompt variants through a flag management UI, not through pull requests.

Fallback Chain Flags

Every AI feature needs a fallback strategy, and your flag system should control the fallback chain. Define a primary model, a secondary model, and a non-AI fallback for each feature. When the primary model times out, errors, or returns low-confidence results, the flag configuration determines what happens next. Does the request retry on a smaller, faster model? Does it fall back to a cached response? Does it show a non-AI default experience? The answer depends on the feature, and flags let you tune this per-feature without code changes. We typically configure: primary (Claude Opus, 3-second timeout), secondary (Claude Haiku, 1-second timeout), fallback (cached response or static content).

The single most important feature flag in your AI stack is the kill switch. When your LLM provider goes down, when your model starts hallucinating at elevated rates, or when you discover a prompt injection vulnerability, you need to turn off AI features in under 30 seconds. Not minutes. Seconds.

Building Effective Kill Switches

A kill switch is more than a boolean flag that disables a feature. It is a pre-planned degradation path that maintains user experience when AI is unavailable. For each AI feature, define what the user sees when AI is off. A chatbot might show a "talk to a human" button. A content generator might display pre-written templates. A recommendation engine might fall back to popularity-based rankings. Design these fallback experiences before you need them, not during an incident at 2 AM.

Your kill switch should be evaluable locally without network calls. If your flag system's SDK caches flag values and evaluates them in-process, a kill switch activation propagates within seconds as the SDK polls for updates. If your flag evaluation requires a network call to an external service, your kill switch depends on that service being available, which defeats the purpose. LaunchDarkly's SDK architecture is strong here because it streams flag updates via SSE and evaluates locally. Unleash works similarly with its client-side SDK. Custom solutions should use a local cache with a background polling interval of 10 seconds or less.

Automated Kill Switch Triggers

Manual kill switches are necessary but not sufficient. You also need automated triggers that activate the kill switch based on real-time metrics. Set thresholds on error rate (if more than 5% of LLM calls fail, kill), latency (if p95 exceeds 5 seconds, fall back to the faster model), cost (if hourly spend exceeds 2x the baseline, throttle), and quality (if the automated quality score drops below a threshold, pause and investigate).

Wire these triggers into your AI observability stack. Your monitoring system should be able to flip a feature flag programmatically through the flag platform's API. LaunchDarkly, Statsig, and Unleash all expose APIs for updating flag values, so your alerting system can call the API to activate a kill switch when thresholds are breached. This closes the loop between detection and mitigation without requiring a human to be awake and available.

Cost Circuit Breakers

LLM costs can spike unexpectedly. A prompt injection that causes the model to generate maximum-length responses, a traffic surge from a viral feature, or a bug that creates an infinite retry loop can all run up thousands of dollars in minutes. Your flag system should include cost circuit breakers that cap spending per feature, per user, and per time window. When a user exceeds 100 requests per hour, throttle them to the cheapest model tier. When a feature exceeds its hourly budget, degrade to cached responses. When total LLM spend exceeds the daily cap, activate the global kill switch and page the on-call engineer. These are not hypothetical scenarios. We have seen each of them happen in production.

The most sophisticated AI teams do not just use flags for manual rollout control. They connect their feature flags directly to their evaluation pipeline, creating automated promotion gates that advance a rollout only when quality metrics pass. This is the AI equivalent of continuous deployment with test gates, and it is the single biggest operational maturity leap you can make.

How Evaluation-Driven Rollouts Work

The flow is straightforward. You create a flag with a new model version or prompt variant at 1% traffic. Your evaluation pipeline continuously scores the outputs from that variant against a set of quality criteria. When the variant accumulates enough data points (typically 200 to 500 interactions) and the quality scores meet your thresholds, an automation script calls the flag platform's API to promote the rollout to the next stage (5%, then 25%, then 50%, then 100%). If quality scores drop below thresholds at any stage, the automation rolls back to the previous stage and alerts the team.

This removes humans from the rollout loop for routine changes while keeping them in the loop for anomalies. Your prompt engineers update a prompt variant, the system tests it at 1%, promotes it through stages over a few days, and it reaches 100% without anyone manually clicking "increase percentage." But if quality degrades at any stage, the system stops and escalates. It is the best of both worlds: speed for normal cases, caution for edge cases.

Defining Quality Gates

Your quality gates should cover four dimensions. Functional correctness measures whether the model produces accurate, factually correct outputs. Behavioral alignment measures whether the model follows its instructions, stays in character, and respects guardrails. Safety checks measure whether the model produces harmful, offensive, or policy-violating content. Performance metrics measure latency, cost per request, and throughput. Each gate should have a hard threshold (auto-rollback if breached) and a soft threshold (pause and alert if breached). For example, a safety gate might hard-rollback if any response triggers a content policy violation, while a latency gate might soft-pause if p95 increases by more than 20% over baseline.

Tools for Evaluation-Driven Rollouts

Building this from scratch requires gluing together your flag platform's API, your evaluation framework (Braintrust, Patronus, or a custom pipeline), and an orchestration layer (a simple cron job or a more sophisticated workflow engine like Temporal). Statsig has the most native support for this pattern because its experimentation engine already tracks custom metrics and can trigger actions based on statistical significance. LaunchDarkly requires more custom integration but its robust API makes it feasible. If you are running a lean team, start with a simple script that polls your evaluation metrics every hour and calls the flag API to adjust rollout percentages. You can add sophistication later.

Let us talk about the practical realities of implementing feature flags for AI features: what it costs, which tools to use, and what mistakes to avoid.

Cost of Flag Infrastructure vs Cost of Incidents

LaunchDarkly starts at roughly $10 per seat per month for their Pro plan, which includes the multi-variate flags and targeting rules you need for AI rollouts. Statsig offers a generous free tier for up to 1 million events and charges based on event volume beyond that. Unleash is open-source and free to self-host, with a managed cloud option starting around $80 per month. A custom solution using Redis and a simple API can be built in a week and costs whatever your Redis instance costs (typically $50 to $200 per month on AWS).

Compare this to the cost of a single bad AI rollout. A prompt that generates inaccurate customer support responses for 48 hours costs you in customer trust, support escalation time, potential legal exposure, and brand reputation. We worked with a fintech client whose AI feature generated incorrect tax estimates for 12 hours before it was caught. The cost of correcting those estimates, notifying affected users, and rebuilding trust dwarfed their entire annual infrastructure budget. A $100/month flag system with automated quality gates would have caught the issue within the first 30 minutes.

Which Tool to Choose

For most teams shipping AI features, our recommendation is Statsig if you value native experimentation and metric tracking, LaunchDarkly if you value enterprise reliability and a mature SDK ecosystem, or Unleash if you want open-source flexibility and are comfortable self-hosting. If you are already using one of these platforms for traditional feature flags, extend it to your AI features rather than adding a separate tool. The overhead of managing multiple flag systems is not worth it.

Safety Considerations Specific to AI Flags

AI feature flags carry unique safety responsibilities. First, never store sensitive user data in flag targeting rules. Targeting by user segment is fine. Targeting by individual user attributes that include PII is a compliance risk. Second, log every flag evaluation for AI features with the corresponding model output. When something goes wrong, you need to correlate the flag state with the outputs it produced. Third, implement flag change approvals for production AI flags. A prompt engineer should not be able to push a new system prompt to 100% of traffic without review. LaunchDarkly's approval workflows and Statsig's change management features support this natively.

Getting Started: Your First AI Feature Flag

If you are not using feature flags for your AI features today, start with one flag for your highest-risk AI feature. Define a kill switch, set up a percentage-based rollout for your next model or prompt change, and wire your monitoring to trigger the kill switch on anomalies. That single flag will save you from your next AI incident, and once your team sees the value, extending the pattern to every AI feature becomes an easy sell.

We help teams design and implement AI rollout infrastructure every week. If you want to move faster without the production risk, book a free strategy call and we will walk through your specific architecture and rollout needs.