Most multimodal demos look the same: base64 a photo into a chat message, ask the model to describe the damage, print the JSON. It works on the first try, which is exactly why teams ship it as the intake path.
That is the wrong shape for production. A photo is not a trusted input. It can be blurred, rotated, a resend, a forwarded thumbnail, or an edited file captured weeks before the incident, and the model will still return confident JSON about it. When the vision model is the only boundary, every one of those problems becomes a prompt problem, and prompts do not reject files.
In this issue, we build a Python photo gate on Microsoft Foundry, with motor insurance claims as the worked example. The same boundaries apply to receipts, product returns, property inspections, or any workflow that accepts photos from people. The gate admits or rejects each of a claim's six photos with deterministic checks, strips metadata, budgets image tokens, asks a Foundry vision model for strict JSON three times, keeps what the samples agree on, checks the result against the policy record, and routes the claim through a fixed table. The model observes. Code decides.
What You Are Building
The companion repository is a small Python application with one dependency, Pillow, and two front ends, a CLI and a local web app, that both call Foundry. The sample claim uses two real photographs of a crash-damaged Suzuki Swift from Wikimedia Commons, published under CC BY-SA 4.0. The pieces:
- An admission gate that rejects unusable photos before any bytes leave the process
- A preparation step that fixes orientation, strips EXIF, and budgets image tokens
- A strict structured-output call to a Foundry vision deployment, made three times and reduced by majority vote
- A contract gate that enforces what the schema cannot express
- A routing table where integrity beats evidence and evidence beats convenience
Issue #49 kept an LLM away from first-pass document extraction because OCR returns inspectable evidence. A crumpled bumper has no OCR equivalent, so the engineering moves to the boundaries around the model.
System Structure
The architecture is a straight line with two record-keeping exits. Admission evaluates each photo, preparation budgets the survivors, Foundry is asked three times, a consensus vote keeps what the samples agree on, the contract gate checks the result against the claim, and the routing table picks one of four outcomes. Every decision lands in a JSON report and an HTML review page.
The model never sees a rejected photo, never sees a file's metadata, and never gets to fast-track a claim. Those three facts carry most of the design.
Runtime Configuration First
Every threshold is explicit configuration with CPX_ environment overrides. The two hard ceilings come from Microsoft's vision-enabled chat models guide: 20 MB per image and ten images per request, and the validator refuses anything above them at startup. The same guide requires image URLs to be publicly reachable, so private claim photos travel as base64 data URIs. The model capability tables list gpt-5-mini, gpt-5, and the gpt-4.1 family with both image input and structured outputs. An excerpt of the live configuration:
{
"foundry": {
"model": "gpt-5.4-mini",
"image_detail": "high",
"reasoning_effort": "medium",
"samples": 3
},
"admission": {
"max_file_megabytes": 20,
"min_shortest_side_px": 640,
"min_sharpness_score": 60
},
"preparation": {
"max_longest_side_px": 1536,
"max_image_tokens_per_claim": 9000
},
"contract": {
"min_damage_confidence": 0.6,
"min_sample_agreement": 2
}
}Admission Runs Before Upload
Before deciding anything, the gate reads facts from the file: format, size, EXIF orientation, capture time, software tag, and whether a GPS block exists. It computes three more itself: mean brightness, a Laplacian-variance sharpness score, and a 64-bit difference hash. OpenAI's images and vision guide says the model does not see file metadata, may misread rotated images, and counts only approximately.
if min(facts.width, facts.height) < self._config.min_shortest_side_px:
reasons.append("resolution_too_low")
if facts.sharpness_score < self._config.min_sharpness_score:
reasons.append("blur_detected")
for other_id, other_hash in admitted_hashes.items():
if hamming_distance(facts.dhash_hex, other_hash) <= self._config.duplicate_hamming_threshold:
reasons.append(f"duplicate_of:{other_id}")
breakPhotos are evaluated in claimant priority order, so the primary photo survives and the resend is rejected. Softer checks produce flags, not rejections. A missing capture time is normal after a messaging app forwards a photo, so it is only noted. A capture time outside the incident window, or an editing application in the software tag, is an integrity signal the routing table acts on later. On the sample claim, three of six photos survive:
P1 admit flags: []
P4 admit_flagged flags: [capture_time_missing]
P5 admit_flagged flags: [gps_metadata_present, orientation_normalised]
P2 reject reasons: [blur_detected, duplicate_of:P5]
P3 reject reasons: [duplicate_of:P1]
P6 reject reasons: [resolution_too_low, duplicate_of:P4]The blurred close-up scored 19 on sharpness against a threshold of 60, while the usable photos scored 930 to 1,321. It is also the same shot as P5, so it carries both reasons. None of the rejected files was ever encoded for upload.
Prepare, Strip, and Budget
Admitted photos are turned upright using their EXIF orientation, bounded to 1,536 pixels, and re-encoded as JPEG with no EXIF block, so the GPS coordinates found in P5 are recorded but never travel. Then each photo is priced. The vision guide documents two token methods: patch-based models such as gpt-5.4-mini count 32-pixel patches and apply a model multiplier, while tile-based models such as gpt-4.1 count 512-pixel tiles. The application implements both, and the tests pin them to the guide's worked examples.
def estimate_patch_tokens(width: int, height: int, multiplier: float, patch_budget: int) -> int:
patches = math.ceil(width / PATCH_SIZE) * math.ceil(height / PATCH_SIZE)
if patches > patch_budget:
shrink = math.sqrt((PATCH_SIZE**2 * patch_budget) / (width * height))
width, height = int(width * shrink), int(height * shrink)
patches = math.ceil(width / PATCH_SIZE) * math.ceil(height / PATCH_SIZE)
return math.ceil(patches * multiplier)The three sample photos cost an estimated 5,192 of the claim's 9,000-token image budget. Photos that do not fit are excluded with a reason. The estimate is a budgeting tool, not a bill: the report stores the endpoint's real usage beside it so the multiplier can be calibrated.
The Schema Fixes the Vocabulary
The model must answer with one strict JSON schema. Microsoft's structured outputs guide requires additionalProperties: false and every property listed as required, and it does not support keywords such as minimum or pattern. So the schema fixes the vocabulary, a closed list of 33 parts, damage types, and severities, and code enforces the rest, such as confidence staying between zero and one. A test walks the schema and fails if anyone adds an unsupported keyword.
damage = {
"type": "object",
"properties": {
"part": {"type": "string", "enum": PARTS},
"damage_type": {"type": "string", "enum": DAMAGE_TYPES},
"severity": {"type": "string", "enum": SEVERITIES},
"confidence": {"type": "number"},
},
"required": ["part", "damage_type", "severity", "confidence"],
"additionalProperties": False,
}The prompt withholds the answer it is meant to check. It names the vehicle and the photo ids to expect, but not the policy plate. If the prompt said the plate should be LM64 LVG, an unsure model would agree. When the model reads LM64 LVG from the bumper unprompted, the application has a fact to compare. The system prompt also defines left and right as the vehicle's own, gives a three-level severity rubric, and allows a part to be reported for a photo only if it is visible in that photo.
The Foundry client sends one chat completion to the endpoint's /openai/v1/ base, with an image part per photo and the request parameters each model family expects. A refusal routes to a person, a truncated answer is never parsed optimistically, and if too few photos survive admission the model is not called at all.
Consensus Before the Contract
A vision model asked the same question twice does not always name the same parts. That is not a defect to prompt away. With samples set to 3, the engine sends the identical request three times and votes before the contract gate sees anything.
for part, votes in part_votes.most_common():
if votes < min_agreement:
continue
records = part_records[part]
severity = _ordinal_vote([record["severity"] for record in records], SEVERITY_ORDER)
matching = [record for record in records if record["severity"] == severity] or records
damages_out.append(
{
"part": part,
"damage_type": _type_vote([record["damage_type"] for record in matching]),
"severity": severity,
"confidence": round(sum(float(record["confidence"]) for record in records) / len(records), 3),
}
)A part survives only if two of three samples reported it, and its confidence is the mean. Severity takes the majority, or the median when the samples split three ways, so one alarmed sample cannot outvote two calmer ones; an even split rounds up. The damage type comes only from samples that agree on that severity. Plate readings, after normalisation, and quality warnings need the same two votes. The result has exactly the schema shape, and the vote counts travel beside it to the review page.
The Contract Gate Decides Meaning
The endpoint guarantees the shape; the contract gate decides the meaning. It checks that the returned photo ids equal the ids sent, that confidences are in range, that a legible plate matches the policy, and that the colour is compatible. Damage below 0.6 confidence is recorded as unconfirmed, so it can neither satisfy a reported zone nor trigger an escalation. Then it compares zones:
for zone in candidate_unreported:
zone_damages = [record for record in confirmed if record["zone"] == zone]
if self._corroborated(zone_damages):
findings.append(ContractFinding("unreported_zone_damaged", "escalate", ...))
else:
findings.append(ContractFinding("possible_unreported_damage", "warn", ...))A reported zone with no confirmed damage means better photos are needed. Damage in a zone the claimant did not report means a person should look, but only if it is corroborated: moderate or worse at 0.8 confidence, or the same part seen in two photos. Anything weaker is a warning, not a reason to reroute the claim.
Routing is then a function with a fixed order. Integrity signals come first: an editing flag, an out-of-window capture time, or a plate, colour, or vehicle mismatch sends the claim to investigation before anything the model said is considered. Evidence comes next: too few photos or an uncovered reported zone means resubmission. Model failures, refusals, and escalations go to an adjuster. Only confirmed, consistent, unescalated damage is fast-tracked. When the model is down, nothing is auto-approved.
Running It in a Browser
A small local web app runs the same engine against your Foundry deployment: upload photos, fill in the claim, press run. It owns no logic, so the browser cannot disagree with the CLI or the tests.
claim-photo-web
Running with Foundry
For a live run, set CPX_FOUNDRY_BASE_URL, CPX_FOUNDRY_API_KEY, and CPX_FOUNDRY_MODEL. The runs below used the configured gpt-5.4-mini deployment. It accepted image parts with the strict schema on the first call, returned exactly the photo ids sent, and read LM64 LVG from both wide photos, matching the policy.
Single calls at low reasoning effort disagreed in ways that matter. Three runs agreed on the crumpled wing, the bumper, and the door, but the wing's severity flipped between runs, one run reported a broken mirror, and another found a scratch on the right wing, a zone the claimant never reported. The mirror is intact. Police tape is wrapped around it, and one sample read the tape as damage.
With consensus on, at three samples and medium reasoning effort, every finding that survived was unanimous:
The wing, door, and bumper scored three of three in P1 and the close-up. The door got no votes in the head-on photo, where it is barely visible. Nothing was found outside the reported zones. All three samples rated the damage severe, the contract gate turned that into high_severity, and the claim went to an adjuster. A crumpled, torn wing is not a fast-track claim, and the table knew that without asking the model.
That run was also the first after a bug was fixed. P5 is stored sideways with an EXIF orientation tag, the way phones save photos. The sample generator rotated it the wrong way before writing the tag, so the gate applied the tag correctly and turned the close-up upside down for three earlier consensus runs. Nothing in the model's answers said so; the bug surfaced only when the uploads were laid out side by side. A test now asserts that the normalised P5 matches the upright framing of the blurred P2. Orientation is part of the input contract.
Because the wide photos never changed, all four consensus runs still work as a control:
- Left wing: kept in all twelve photo-runs, always with three of three votes
- Bumper: kept in eleven of twelve
- Door: the borderline; in P1, a photo that never changed, it scored three, one, zero, and three votes
- Hood: one sample in one run reported it in all three photos; one vote per photo, so it was dropped
- Mirror and right wing: never voted for
Every run routed to adjuster review for high_severity. Consensus does not make a borderline observation certain. It makes the borderline visible, keeps it out of the decision, and leaves the decision unchanged when the strong evidence repeats. One sample repeating itself across three photos is still one witness.
Consensus has a price. Each sample billed about 3,372 prompt tokens against an estimate of 5,192 image tokens, so the patch formula is conservative here. Medium effort spent about 1,400 reasoning tokens per sample, against 120 to 211 at low effort. A claim cost about 15,700 tokens, roughly four times a single low-effort call. That is a business decision, so the sample count and reasoning effort are configuration, not constants.
Tests Lock the Control Layer
The tests need no credentials and make no network calls: a hand-authored model response stands in for Foundry so the gates around it can be checked for free. They cover every admission outcome, both token formulas, schema strictness, consensus and corroboration, each contract check, routing, the review page, and the orientation regression. The model's visual judgement belongs in a separate evaluation against labelled photos. The rules around it should be fast, local, and repeatable.
python -m pytest
68 passed in 34.99sWhy This Architecture Works
The model is surrounded rather than trusted. Photos are evidence with facts attached, read by code before and after the only probabilistic step.
- Unusable and duplicate photos never reach the model
- Metadata is checked by code and stripped before upload
- Majority vote removes findings the model cannot repeat
- The model never receives the answer it is meant to verify
- A fixed table routes, and a missing model means a person, not an approval
Potential Enhancements
A production version would extend the same boundaries rather than add new ones.
- Tune the sample count, agreement, and corroboration thresholds against a labelled photo set
- Index perceptual hashes across claims to catch photos reused from earlier claims
- Add an image-forensics model as a second integrity signal
- Shadow-test model upgrades on the same claims, as in Issue #29
Final Notes
Vision models made photo understanding cheap. They did not make photos trustworthy, and they did not make their own output a decision.
A production multimodal system admits the input, strips what should not travel, budgets what will, constrains what comes back, checks it against what it already knows, and routes with a table it can explain. The model does one job in the middle, and does it better because everything around it is deterministic.
Explore the source code at the GitHub repository.
See you in the next issue.
Stay curious.