Most LLM demos still use free-form text. That is fine for chat, but weak for software systems. Production systems need machine-readable outputs for routing, escalation, workflow branching, and controls.
If your app must interpret prose, behavior drifts. Structured inputs and outputs remove that ambiguity. In this issue, we build a local-first app with structured JSON input, structured JSON output, typed parsing, deterministic validation, and policy-based routing.
What You Are Building
A production-shaped structured LLM pipeline:
- Load runtime config with environment overrides
- Validate endpoint, model, and key before execution
- Build incident context as JSON
- Send explicit JSON instructions and schema context
- Use typed response formatting when supported
- Fallback to prompt-only JSON when needed
- Normalize and validate parsed output
- Route with deterministic policy
The scenario is incident triage. The model proposes. The system decides.
Why Free-Form Output Breaks
Prompt:
Analyze this production incident and provide severity, cause, and recommended actions.Typical output:
The issue appears related to the database cluster. The severity is high since users cannot access checkout. Engineers should investigate slow queries and scale replicas.Humans can read this. Machines must guess:
- Where severity appears
- How severity maps to allowed values
- Where cause starts and ends
- How actions are separated
- Whether format changed across calls
Structured outputs solve this. The model fills a contract. Code validates and routes.
System Structure
This is a deterministic shell around a probabilistic component.
- Load and validate config
- Build structured incident input
- Request structured output
- Normalize and validate output
- Apply deterministic routing policy
The diagram below shows the control flow and the key decision points in the system:
Runtime Config First
Validate runtime settings before any model call.
This is where reliability starts. Bad endpoint or model settings should fail fast.
public sealed class LlmAppConfig
{
public string Provider { get; init; } = "lmstudio";
public string BaseUrl { get; init; } = "http://localhost:1234/v1";
public string ApiKey { get; init; } = "not-needed";
public string ModelId { get; init; } = "deepseek/deepseek-r1-0528-qwen3-8b";
public float Temperature { get; init; } = 0.0f;
public bool UseTypedResponseFormat { get; init; } = true;
public void Validate()
{
if (string.IsNullOrWhiteSpace(BaseUrl))
throw new InvalidOperationException("Llm:BaseUrl is required.");
if (!Uri.TryCreate(BaseUrl, UriKind.Absolute, out _))
throw new InvalidOperationException("Llm:BaseUrl must be an absolute URI.");
if (string.IsNullOrWhiteSpace(ModelId))
throw new InvalidOperationException("Llm:ModelId is required.");
if (string.IsNullOrWhiteSpace(ApiKey))
throw new InvalidOperationException("Llm:ApiKey is required.");
}
}Fail early. Do not debug runtime config through model behavior.
Structured Input and Output Contracts
Input contract:
The input object carries operational facts as typed fields, not loose prose.
public sealed class IncidentContextEnvelope
{
public IncidentContext Incident { get; init; } = new();
}
public sealed class IncidentContext
{
public string IncidentId { get; init; } = string.Empty;
public string Service { get; init; } = string.Empty;
public string Region { get; init; } = string.Empty;
public int ErrorRatePercent { get; init; }
public int P95LatencyMs { get; init; }
public int ActiveCheckoutSessions { get; init; }
public List<string> Symptoms { get; init; } = [];
public List<string> RecentChanges { get; init; } = [];
public string DetectedAtUtc { get; init; } = string.Empty;
}Output contract:
This output shape is the strict boundary between model generation and deterministic code.
public sealed class IncidentAnalysis
{
public string Severity { get; set; } = string.Empty;
public string PrimaryCause { get; set; } = string.Empty;
public List<string> RecommendedActions { get; set; } = [];
public List<string> MissingData { get; set; } = [];
}Free-form text is not a contract. Typed output is.
Schema and Prompt Constraints
The schema bounds values and data types. The prompt enforces behavior rules.
{
"type": "object",
"properties": {
"Severity": {
"type": "string",
"enum": [
"P1",
"P2",
"P3",
"P4"
]
},
"PrimaryCause": {
"type": "string"
},
"RecommendedActions": {
"type": "array",
"items": {
"type": "string"
}
},
"MissingData": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"Severity",
"PrimaryCause",
"RecommendedActions",
"MissingData"
]
}var prompt = $"""
You are a production incident analysis system.
Rules:
1. Never invent metrics not provided.
2. Use severity only: P1,P2,P3,P4.
3. Provide at least 3 recommended actions.
4. Put unknowns in MissingData.
5. Return JSON only and follow the schema.
Output JSON schema:
{responseSchema}
Incident context:
{incidentContextJson}
""";The schema constrains values. The prompt constrains behavior.
Typed Output with Fallback
Use typed response mode when available. It usually improves consistency.
var typedSettings = new OpenAIPromptExecutionSettings
{
Temperature = config.Temperature
};
if (config.UseTypedResponseFormat)
typedSettings.ResponseFormat = typeof(IncidentAnalysis);Fallback keeps the pipeline running when a provider does not support typed response format.
static async Task<string> InvokeWithTypedFallbackAsync(
Kernel kernel,
string prompt,
OpenAIPromptExecutionSettings typedSettings,
LlmAppConfig config)
{
try
{
var result = await kernel.InvokePromptAsync(prompt, new KernelArguments(typedSettings));
return result.ToString();
}
catch when (config.UseTypedResponseFormat)
{
var fallbackSettings = new OpenAIPromptExecutionSettings { Temperature = config.Temperature };
var fallback = await kernel.InvokePromptAsync(prompt, new KernelArguments(fallbackSettings));
return fallback.ToString();
}
}Prefer typed mode. Keep a reliable fallback path.
Normalize, Validate, Decide
Some models still wrap JSON with extra text. Normalize first.
The extractor below strips fences and extra text, then returns the JSON object slice.
static string NormalizeJson(string payload)
{
var trimmed = payload.Trim();
if (trimmed.StartsWith("```", StringComparison.Ordinal))
{
var firstNewLine = trimmed.IndexOf('\n');
var lastFence = trimmed.LastIndexOf("```", StringComparison.Ordinal);
if (firstNewLine >= 0 && lastFence > firstNewLine)
trimmed = trimmed[(firstNewLine + 1)..lastFence].Trim();
}
var firstBrace = trimmed.IndexOf('{');
var lastBrace = trimmed.LastIndexOf('}');
if (firstBrace >= 0 && lastBrace > firstBrace)
return trimmed[firstBrace..(lastBrace + 1)];
return trimmed;
}Then normalize fields and enforce deterministic rules:
Parsing alone is not enough. Validation is the control point for downstream actions.
public static List<string> Validate(IncidentAnalysis analysis)
{
var errors = new List<string>();
if (!AllowedSeverity.Contains(analysis.Severity))
errors.Add("Severity must be one of P1/P2/P3/P4.");
if (string.IsNullOrWhiteSpace(analysis.PrimaryCause))
errors.Add("PrimaryCause is required.");
if (analysis.RecommendedActions.Count < 3)
errors.Add("RecommendedActions must include at least 3 actions.");
return errors;
}Do not let model output execute directly. Route through deterministic policy.
This decision object is what automation should consume.
public sealed class IncidentExecutionDecision
{
public string TicketPriority { get; init; } = string.Empty;
public string TicketQueue { get; init; } = string.Empty;
public string EscalationTarget { get; init; } = string.Empty;
public int MitigationSlaMinutes { get; init; }
public string ExecutionMode { get; init; } = string.Empty;
public bool FreezeDeployments { get; init; }
public List<string> DeterministicActions { get; init; } = [];
}The model helps classify. The system owns control.
Structured Outputs vs JSON Mode
JSON mode usually guarantees valid JSON syntax. Structured outputs aim to enforce a schema.
Use both layers:
- Prefer typed response formatting when available
- Include schema in the prompt for fallback
- Normalize and validate after parsing
Final Notes
Structured inputs and outputs are not cosmetic. They are control contracts for software systems.
When model output is typed, validated, and routed by deterministic policy, behavior becomes predictable.
Explore the source code at the GitHub repository.
See you in the next issue.
Stay curious.