A lot of AI systems still treat runtime behavior as deployment behavior. The model route lives in an environment variable. The prompt version is buried in application code. Tool access changes require a redeploy. Retrieval index selection is decided by whatever the service booted with. When something goes wrong, the team has to ship code just to make the AI path stop doing the wrong thing.
That is a weak control surface for production AI. Model routes, prompts, retrieval corpora, reasoning modes, and tool authority are not ordinary constants. They change answer quality, safety, cost, compliance posture, and blast radius. They should be observable runtime decisions, not hidden release assumptions.
In this issue, we build a local Python runtime-control layer that uses feature flags before an AI workflow executes. The companion repo evaluates flags for model route, prompt version, tool mode, retrieval corpus, reasoning mode, and the global runtime kill switch. Deterministic policy code then decides whether the request is allowed to reach the generative path.
The Boundary I Care About Here
The important boundary is not whether a flag can return a value. That part is easy. The boundary is whether a runtime flag is allowed to change AI behavior without making the system unreviewable.
OpenFeature's evaluation context gives us a useful vocabulary: a flag decision can depend on a targeting key and custom fields such as tenant, environment, region, or risk tier. OpenTelemetry's feature-flag semantic conventions give us a telemetry shape for recording flag evaluations. flagd shows how portable flag definitions and targeting rules can be treated as configuration rather than ad hoc code.
This issue does not build a full OpenFeature provider or require a running flag service. It builds the smaller thing first: an inspectable local control layer that makes each AI runtime decision explicit before a model call happens.
What We Are Actually Building
The companion repository contains a Python project named ai-runtime-flags. It has no model dependency and no cloud dependency. That is deliberate. The point is to expose the runtime control contract without hiding it behind a successful answer from an LLM.
The app does this:
- loads a JSON feature flag document
- loads a runtime policy document
- loads sample request contexts
- evaluates required flags for each request
- selects model route, prompt version, tool mode, retrieval corpus, and reasoning mode
- blocks requests when the runtime kill switch is off or policy is violated
- writes OpenTelemetry-shaped feature flag events
- writes a JSON report explaining each allow or block decision
The result is not a feature flag product. It is the narrow control model I want in front of one.
The Shape Of The Runtime
The flow is intentionally small. Request context enters first. Flags are evaluated against that context. Policy code checks whether the selected values are allowed. Only then would the AI workflow be allowed to call a model, retrieve context, or expose tools.
The model is not in charge of the route. The prompt is not in charge of the kill switch. The tool call is not in charge of its own permission level. Runtime configuration proposes a value, and deterministic policy decides whether the system can proceed.
Flags Are Runtime Contracts
The flag document controls six things. That is enough to show the pattern without building a whole product platform.
{
"flags": {
"ai.runtime.enabled": {
"state": "ENABLED",
"valueType": "boolean",
"owner": "platform-ai",
"description": "Global kill switch for the generative runtime path.",
"defaultVariant": "on",
"variants": {
"on": true,
"off": false
}
},
"ai.model.route": {
"state": "ENABLED",
"valueType": "object",
"owner": "platform-ai",
"description": "Selects the model route and generation limits."
},
"ai.prompt.version": {
"state": "ENABLED",
"valueType": "string",
"owner": "support-ai"
},
"ai.tools.mode": {
"state": "ENABLED",
"valueType": "string",
"owner": "platform-ai"
},
"ai.retrieval.corpus": {
"state": "ENABLED",
"valueType": "string",
"owner": "knowledge-platform"
},
"ai.reasoning.mode": {
"state": "ENABLED",
"valueType": "string",
"owner": "support-ai"
}
}
}This is already better than scattering these values across prompts, environment variables, and service startup code. Each flag has a key, type, owner, default, variants, and rules. That gives reviewers something concrete to challenge.
In a larger system, these flags could live behind OpenFeature, flagd, LaunchDarkly, Azure App Configuration, or another provider. The local JSON file is not the product. The contract is the product.
Evaluation Context Decides The Variant
Feature flags become useful when the same deployed service can behave differently for different contexts. A staging internal user can receive a candidate model. A high-risk healthcare request can receive a stricter model route and safer prompt. A European request can use a regional corpus. Incident mode can shut the generative path down.
{
"request_id": "REQ-002",
"targeting_key": "tenant:northwind-health:user:452",
"tenant": "northwind-health",
"tenant_tier": "enterprise",
"environment": "prod",
"region": "eu",
"workflow": "claims_summary",
"risk_tier": "high",
"incident_mode": false
}The evaluator uses that request context to select variants. The implementation is intentionally plain:
def evaluate(self, flag_key: str, context: EvaluationContext) -> EvaluationDetails:
flag = self._flags.get(flag_key)
if flag is None:
return EvaluationDetails(
flag_key=flag_key,
value=None,
variant="missing",
reason="ERROR",
error="flag_not_found",
metadata={"provider": self.provider_name},
)
default_variant = str(flag["defaultVariant"])
chosen_variant = default_variant
reason = "DEFAULT"
rule_name: str | None = None
if flag.get("state") == "DISABLED":
reason = "DISABLED"
else:
for rule in flag.get("rules", []):
if self._matches(rule.get("when", {}), context):
chosen_variant = str(rule["variant"])
reason = "TARGETING_MATCH"
rule_name = str(rule.get("name", "unnamed-rule"))
break
value = flag["variants"][chosen_variant]
return EvaluationDetails(
flag_key=flag_key,
value=value,
variant=chosen_variant,
reason=reason,
rule_name=rule_name,
metadata={
"provider": self.provider_name,
"owner": flag.get("owner", "unknown"),
"description": flag.get("description", ""),
},
)There is no model judgment in that code. The evaluator does one job: turn request context into typed runtime configuration.
The Kill Switch Is Not An Afterthought
The global runtime switch is the simplest flag and the most important one. It decides whether the generative path is allowed to run at all.
"ai.runtime.enabled": {
"state": "ENABLED",
"valueType": "boolean",
"owner": "platform-ai",
"description": "Global kill switch for the generative runtime path.",
"defaultVariant": "on",
"variants": {
"on": true,
"off": false
},
"rules": [
{
"name": "disable-during-incident-mode",
"when": {
"incident_mode": true
},
"variant": "off"
},
{
"name": "disable-for-blocked-tenant",
"when": {
"tenant": "blocked-bank"
},
"variant": "off"
}
]
}A kill switch should not be a paragraph in the system prompt. It should not rely on the model understanding that today is an incident day. It should be evaluated before the prompt is assembled and before tools are exposed.
The companion repo includes docs/KILL_SWITCH_MATRIX.md because the switch is operational. It maps failure modes to controls: disable the generative path, disable tools, freeze retrieval, roll back a prompt package, or return traffic to the baseline model route.
Model And Prompt Selection Are Data
A model change should not be smuggled into production as a code diff that nobody can correlate to requests. A prompt change should not be an inline string edit with no runtime attribution. Both are runtime decisions with product impact.
"ai.model.route": {
"defaultVariant": "baseline",
"variants": {
"baseline": {
"route": "baseline",
"provider": "openai-compatible",
"model": "support-baseline",
"temperature": 0.1,
"max_output_tokens": 800
},
"high-risk-strict": {
"route": "strict",
"provider": "openai-compatible",
"model": "support-strict",
"temperature": 0.0,
"max_output_tokens": 512
},
"staging-canary": {
"route": "canary",
"provider": "openai-compatible",
"model": "support-candidate",
"temperature": 0.1,
"max_output_tokens": 800
}
},
"rules": [
{
"name": "high-risk-requests-use-strict-route",
"when": {
"risk_tier": "high"
},
"variant": "high-risk-strict"
},
{
"name": "internal-staging-uses-canary",
"when": {
"tenant_tier": "internal",
"environment": "staging"
},
"variant": "staging-canary"
}
]
}LaunchDarkly's AgentControl config documentation is a useful signal here because commercial rollout platforms are treating AI model and prompt configuration as runtime-managed product behavior. You do not need that specific vendor to use the pattern. You do need the operational idea: model and prompt choices should be attributable to a request.
Tool Access Is A Flag, But Policy Owns Execution
A feature flag can choose a tool mode. It should not be the only thing that decides whether a tool is safe. The policy engine still checks the selected value against the request risk tier.
risk_tier = str(context.get("risk_tier", "standard"))
max_tokens = int(model_config.get("max_output_tokens", 0))
allowed_tokens = self._max_tokens_for_risk(risk_tier)
if max_tokens > allowed_tokens:
reasons.append(f"max_output_tokens_exceeds_policy:{max_tokens}>{allowed_tokens}")
allowed_tool_modes = self._allowed_tool_modes_for_risk(risk_tier)
if tools_mode not in allowed_tool_modes:
reasons.append(f"tools_mode_not_allowed_for_risk:{tools_mode}:{risk_tier}")
if reasons:
return self._blocked(context, evaluations, reasons)This is the deterministic boundary. The flag system may return write_gated. The request may be high risk. The policy is allowed to say no. That matters because flag systems are configuration systems, not safety systems by themselves.
The tests include a case where a high-risk request receives write-enabled tools from a manipulated evaluation result. The policy blocks it. That is the behavior I want in production code. Configuration can change. Execution policy still has to hold.
Retrieval Corpus Selection Belongs In The Same Contract
Model route and prompt version get most of the attention, but retrieval selection is just as important. If the wrong corpus is active, the answer can be grounded in the wrong jurisdiction, stale product policy, or unapproved incident notes.
"ai.retrieval.corpus": {
"state": "ENABLED",
"valueType": "string",
"owner": "knowledge-platform",
"description": "Selects the retrieval corpus version.",
"defaultVariant": "global-current",
"variants": {
"global-current": "support-global-2026-08",
"eu-current": "support-eu-2026-08",
"frozen-incident": "frozen-support-knowledge-2026-08"
},
"rules": [
{
"name": "incident-mode-uses-frozen-corpus",
"when": {
"incident_mode": true
},
"variant": "frozen-incident"
},
{
"name": "eu-region-uses-eu-corpus",
"when": {
"region": "eu"
},
"variant": "eu-current"
}
]
}This connects directly to the last issue. A data contract decides what is allowed to become AI context. A runtime flag decides which approved corpus this request is allowed to use. Those are different boundaries, and production systems need both.
Telemetry Makes Rollout Explainable
When a user reports a bad answer, the team should not have to guess which model, prompt, corpus, and tool mode were active. The runtime decision should be visible in telemetry.
The companion repo writes one feature_flag.evaluation event for each required flag and one ai.runtime.decision event for the final policy result. The event names and attribute keys are local JSONL, but they follow the shape of OpenTelemetry feature-flag conventions where it matters.
{
"name": "feature_flag.evaluation",
"attributes": {
"feature_flag.key": "ai.model.route",
"feature_flag.result.variant": "high-risk-strict",
"feature_flag.result.value": {
"route": "strict",
"provider": "openai-compatible",
"model": "support-strict",
"temperature": 0,
"max_output_tokens": 512
},
"feature_flag.context.id": "7dd0a0f4afdbb597",
"feature_flag.evaluation.reason": "TARGETING_MATCH",
"ai.request.id": "REQ-002",
"ai.workflow": "claims_summary",
"ai.environment": "prod",
"ai.risk_tier": "high"
}
}The raw targeting key is not written to telemetry. The repo hashes it before persistence. That does not make the telemetry perfect, but it makes the example honest: control observability should not casually leak user or tenant identifiers.
A Local Run Tells The Story
The repo runs with no model and no external service:
python run.pyThe sample workload contains five requests:
- a normal enterprise support request
- a high-risk EU claims summary request
- a request during incident mode
- an internal staging request for a canary route
- a blocked restricted tenant
The output is deliberately plain:
AI runtime flag evaluation
Requests: 5
Allowed: 3
Blocked: 2
REQ-001 | ALLOW | route=baseline | reasons=all_runtime_controls_satisfied
REQ-002 | ALLOW | route=strict | reasons=all_runtime_controls_satisfied
REQ-003 | BLOCK | route=none | reasons=runtime_disabled_by_flag
REQ-004 | ALLOW | route=canary | reasons=all_runtime_controls_satisfied
REQ-005 | BLOCK | route=none | reasons=runtime_disabled_by_flag
Report: data/reports/runtime-decisions.json
Events: data/reports/otel-events.jsonlThat output gives a reviewer enough information to ask useful questions. Why did REQ-002 use the strict route? Why did REQ-004 receive canary behavior? Why did REQ-003 block? Which exact flag produced that decision? Which policy rule accepted or rejected it?
The Tests Protect The Control Layer
The tests do not need a feature flag service. They test the control contract directly.
python -m unittest discover -s testsThe current suite covers:
- strict model routing for high-risk requests
- runtime disablement during incident mode
- invalid flag-rule validation
- policy enforcement for blocked tenants
- policy rejection of unsafe tool modes
- OpenTelemetry-style audit event fields
That is the right test boundary for this issue. A real provider can be swapped in later. The runtime contract still needs stable behavior when a flag is missing, a variant is unsafe, a request is high risk, or a kill switch is active.
The Deterministic Boundary
The probabilistic layer may draft an answer, summarize retrieved evidence, classify a support request, or produce a proposed action. It does not own runtime authority.
The deterministic layer owns:
- which flags are required
- how request context is evaluated
- which model route is active
- which prompt package is active
- which retrieval corpus is active
- which tool authority is exposed
- whether the runtime kill switch blocks execution
- which telemetry is written for later review
This keeps AI runtime behavior out of the prompt and out of invisible deployment assumptions. The model can still be useful. It just cannot decide the operating envelope it runs inside.
Why This Architecture Works
This architecture works because it turns AI runtime behavior into explicit, testable control decisions instead of hidden assumptions inside prompts, deployments, or model calls.
- Runtime behavior can change without redeploying the service.
- Every important AI runtime choice has an owner and a flag key.
- Request context makes rollout targeted instead of global by default.
- The kill switch runs before model execution, not after damage is observed.
- Policy code can reject unsafe flag combinations.
- Telemetry makes a later incident review concrete.
- The local implementation is small enough to inspect, test, and replace.
Potential Enhancements
The obvious next step is to replace the local JSON evaluator with a real OpenFeature provider and run the same policy tests against provider-backed evaluations. The point would not be to trust the provider blindly. The point would be to keep the same deterministic runtime contract while moving flag storage and targeting to production infrastructure.
I would also add signed flag bundles, change approval records, stale-flag detection, per-tenant rollout budgets, automatic rollback thresholds, and correlation with GenAI request metrics. OpenTelemetry's GenAI semantic conventions are relevant here because feature flag events become more useful when they can be correlated with latency, token use, failures, and answer-quality signals.
For agent systems, I would split tool access into more granular flags: read-only lookup, ticket annotation, customer message draft, customer message send, refund proposal, refund execution, deployment rollback proposal, and deployment rollback execution. A single tools_enabled boolean is too blunt for serious systems.
Final Notes
A production AI system should be able to change its operating envelope without a redeploy. It should also be able to explain which envelope was active for a specific request. Feature flags are a practical way to get there, but only when they are treated as runtime contracts rather than convenience toggles.
The practical lesson is simple: put model route, prompt version, tool authority, retrieval corpus, reasoning mode, and kill switches behind explicit runtime controls. Then enforce the final decision in deterministic policy code before the model gets a chance to act.
Explore the companion repository at the GitHub repository.
See you in the next issue.
Stay curious.