A lot of AI systems treat moderation as either a prompt instruction or a single classifier call. The model is told to be safe, the input is checked once, and the rest of the workflow assumes the content boundary has been handled.
That is too thin for production AI engineering. Toxicity, threats, harassment, identity attacks, sexual content, and self-harm signals are not all the same risk. Some inputs should be allowed. Some should be flagged. Some should be blocked. Some should be escalated to a human or crisis-aware support process. If those decisions live only inside model text, the system cannot explain why a case moved forward or stopped.
In this issue, we build a Python moderation gate that uses Detoxify as the first scoring layer, a local Ollama model as a second structured reviewer, and deterministic policy code as the final authority. The goal is not to make a universal safety system. The goal is to show the system shape: measurable classifier signals, bounded LLM review, explicit thresholds, separate sensitive-category escalation, and auditable decisions.
What You Are Building
You are building a production-shaped content moderation gate for text flowing into an AI application. It replays a small frozen case set, scores each message with Detoxify, applies deterministic policy thresholds, optionally asks a local Ollama model for one structured review, and persists the full decision path as JSON.
The gate has four possible outcomes:
allow: no moderation threshold was crossedflag: content can continue, but should be labeled or queued for reviewblock: content should not continue through the product workflowescalate: the case needs a human or specialized safety process
That last category matters. Self-harm or crisis-language signals should not be treated as ordinary toxicity. They are routed through escalation because the operational response is different from blocking an abusive comment.
System Structure
The architecture is intentionally small. A JSON case file feeds the moderation engine. Detoxify returns label scores. Deterministic policy checks input shape, threshold crossings, and sensitive phrases. The LLM reviewer runs only when configured policy says the case needs a second look. The final gate keeps the stricter result and writes an audit record.
The important boundary is simple: the classifier and the LLM provide signals. They do not own the moderation policy. Code owns thresholds, escalation rules, final decision ranking, exit codes, and audit persistence.
Runtime Configuration First
The sample starts from a typed JSON configuration:
{
"detoxify": {
"model_name": "unbiased",
"device": "cpu"
},
"llm": {
"enabled": true,
"base_url": "http://localhost:11434/v1",
"api_key": "ollama",
"model": "llama3.2:3b",
"timeout_seconds": 45,
"review_mode": "ambiguous"
},
"policy": {
"max_input_chars": 4000,
"flag_thresholds": {
"toxicity": 0.45,
"severe_toxicity": 0.25,
"threat": 0.3
},
"block_thresholds": {
"toxicity": 0.85,
"severe_toxicity": 0.6,
"threat": 0.7
},
"ambiguous_margin": 0.08,
"self_harm_terms": [
"kill myself",
"end my life",
"suicide"
]
}
}This is not decoration. Thresholds, review mode, maximum input length, local model identity, timeout, and audit location are operational controls. They should be visible before a moderation run starts.
The config model also rejects remote LLM endpoints:
@field_validator("base_url")
@classmethod
def require_local_endpoint(cls, value: HttpUrl) -> HttpUrl:
host = value.host
if host not in {"localhost", "127.0.0.1", "::1"}:
raise ValueError("LLM base_url must point to a local Ollama endpoint")
return valueThat keeps the second layer local by default. A moderation gate can contain sensitive user text. Accidentally changing the reviewer endpoint from local Ollama to a hosted service should be a configuration error, not a quiet network call.
The Case Set Is Replayable
The repo uses a frozen JSON case set rather than a chat loop. That makes policy behavior repeatable and testable:
[
{
"case_id": "MOD-001",
"text": "Thanks for the update. I disagree with the policy, but I appreciate the explanation.",
"source": "customer_chat"
},
{
"case_id": "MOD-003",
"text": "I feel like I might hurt myself tonight and I do not know who to call.",
"source": "support_chat"
}
]The examples are deliberately plain. The repository does not need a fixture full of slurs to prove the control model. The tests use fake classifier scores, and the sample data uses policy-relevant language that is safe to inspect.
Detoxify Is the First Signal
Detoxify is a good fit for the first layer because it gives the application numeric toxicity-related scores. The repo wraps it behind a small classifier interface so the policy engine does not depend directly on model loading, device selection, or package-specific label names.
class ToxicityClassifier(Protocol):
def assess(self, text: str) -> DetoxifyAssessment:
"""Return toxicity scores for one text input."""
class DetoxifyClassifier:
def __init__(self, settings: DetoxifySettings):
self._settings = settings
self._model = None
def assess(self, text: str) -> DetoxifyAssessment:
if self._model is None:
from detoxify import Detoxify
self._model = Detoxify(self._settings.model_name, device=self._settings.device)
raw_scores = self._model.predict(text)
scores = {_normalize_label(name): float(value) for name, value in raw_scores.items()}
return DetoxifyAssessment(model_name=self._settings.model_name, scores=scores)The adapter lazily loads Detoxify so tests can exercise the policy layer without downloading classifier weights. That keeps the production integration real while keeping the deterministic tests fast.
Policy Owns the First Decision
The first decision is deterministic. It checks empty input, maximum length, self-harm phrases, block thresholds, flag thresholds, and the ambiguous band around a threshold.
if self._contains_self_harm_signal(text):
return (
Decision.ESCALATE,
["self_harm_signal"],
["Self-harm or crisis language requires escalation."],
True,
)
for label, threshold in self._settings.block_thresholds.items():
score = assessment.score_for(label)
if score >= threshold:
decision = Decision.BLOCK
triggered.append(label)
reasons.append(f"{label} score {score:.3f} crossed block threshold {threshold:.3f}.")This is the core control boundary. A classifier score can trigger a policy rule, but the classifier does not decide what the product should do. A threat threshold may block. A lower toxicity score may flag. A self-harm phrase may escalate even when toxicity is low.
Ambiguous Cases Get a Second Layer
The sample does not call the LLM for every input by default. The configured mode is ambiguous, which means the reviewer runs only when deterministic policy says the case needs extra judgment. That includes scores near a threshold, flagged content, blocked content, and sensitive escalation cases.
def should_review(self, mode: ReviewMode, needs_review: bool) -> bool:
if mode == ReviewMode.OFF:
return False
if mode == ReviewMode.ALWAYS:
return True
return needs_reviewThis gives the LLM a narrow role. It can inspect context and recommend a category, but it is not the first line of defense and it is not the final authority.
The LLM Reviewer Returns Structured JSON
The Ollama reviewer uses the local OpenAI-compatible chat completions endpoint and asks for one small JSON object:
payload = {
"model": self._settings.model,
"temperature": 0,
"response_format": {"type": "json_object"},
"messages": [
{
"role": "system",
"content": (
"You are a content-risk reviewer for a moderation pipeline. "
"Return only JSON with keys: category, recommended_decision, "
"confidence, rationale. recommended_decision must be one of "
"allow, flag, block, escalate."
),
},
{
"role": "user",
"content": json.dumps(
{
"text": text,
"detoxify_triggered_labels": triggered_labels,
"allowed_decisions": [decision.value for decision in Decision],
},
ensure_ascii=True,
),
},
],
}The structured output is intentionally small. The reviewer does not write the user response, call tools, or change policy settings. It returns a recommendation that the deterministic gate can compare against the current decision.
Reviewer Failure Fails Closed
The local model may be unavailable. The response may not parse. The server may time out. The sample handles that path explicitly:
except Exception as exc:
return LlmReview(
reviewed=True,
category="review_failed",
recommended_decision=Decision.ESCALATE,
confidence=1.0,
rationale=f"LLM review failed closed: {type(exc).__name__}",
)That is the right default for a moderation boundary. If the system was about to rely on a second reviewer and that reviewer fails, the safe operational result is escalation, not silent approval.
The Final Gate Preserves the Stricter Result
The LLM can raise the decision. It cannot lower a deterministic block into an allow decision.
rank = {
Decision.ALLOW: 0,
Decision.FLAG: 1,
Decision.BLOCK: 2,
Decision.ESCALATE: 3,
}
if rank[review.recommended_decision] > rank[initial]:
return review.recommended_decision, ["LLM reviewer raised the moderation decision."]
return initial, ["Deterministic policy decision was retained after LLM review."]This is where the two-layer design stays inspectable. The LLM is allowed to be useful at the margin, especially for context-sensitive or sensitive cases, but the policy engine keeps the decision lattice explicit.
Walking a Real Local Run
Create the Python environment:
python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"Start Ollama with the configured model:
ollama pull llama3.2:3b
ollama serveRun the moderation gate:
moderation-gate --config config/local-ollama.json --cases data/moderation_cases.jsonA representative run keeps the console output intentionally compact:
MOD-001: allow | labels=none
MOD-002: flag | labels=toxicity
MOD-003: escalate | labels=self_harm_signal
MOD-004: block | labels=threat
Audit written to reports/moderation-run-20260718T090000Z.jsonThe command returns exit code 2 when any case is blocked or escalated. That makes the gate suitable for CI-style checks when the input case set represents content that should remain acceptable.
Audit Records Make the Decision Reviewable
Every result carries the classifier scores, triggered labels, optional LLM review, final decision, and reasons:
{
"case_id": "MOD-003",
"source": "support_chat",
"decision": "escalate",
"triggered_labels": [
"self_harm_signal"
],
"detoxify": {
"model_name": "unbiased",
"scores": {
"toxicity": 0.012,
"threat": 0.004
}
},
"llm_review": {
"reviewed": true,
"category": "self_harm",
"recommended_decision": "escalate",
"confidence": 0.91,
"rationale": "The message describes possible self-harm and asks for help."
},
"reasons": [
"Self-harm or crisis language requires escalation.",
"LLM reviewer recommended escalation."
]
}That record is the difference between a moderation guess and an operational decision. A reviewer can see which layer fired, which threshold mattered, whether the LLM participated, and why the final outcome was selected.
Tests Lock the Control Layer
The test suite does not require Detoxify weights or an Ollama server. It uses fake classifier and reviewer adapters to test the policy contract directly:
def test_self_harm_terms_escalate_even_when_toxicity_is_low() -> None:
policy = ModerationPolicy(settings())
decision, labels, reasons, needs_review = policy.initial_decision(
"I might hurt myself tonight.", assessment(toxicity=0.01, threat=0.01)
)
assert decision == Decision.ESCALATE
assert labels == ["self_harm_signal"]
assert needs_review is True
assert "Self-harm" in reasons[0]That is the right test boundary. The real classifier and local model can change scores or wording. The application policy still needs stable behavior for empty inputs, threshold crossings, ambiguous bands, self-harm escalation, reviewer failure, and final decision ranking.
Why This Architecture Works
- The classifier gives measurable first-pass risk signals
- The policy engine owns thresholds and decision semantics
- The LLM reviewer has a narrow structured role
- Self-harm is treated as escalation, not toxicity
- Reviewer failure fails closed
- Audit records preserve the full decision path
- Tests verify the control layer without requiring model infrastructure
The model remains probabilistic. Threshold evaluation, review routing, final decision ranking, local endpoint validation, exit codes, and audit persistence remain deterministic.
Potential Enhancements
- Add a calibration report for chosen thresholds against a labeled internal dataset
- Separate policy profiles for support chat, public comments, and internal workplace messages
- Add redaction before audit persistence for emails, phone numbers, and account identifiers
- Route escalated cases into an approval or incident workflow instead of only writing JSON
- Record model version, classifier weight identity, and prompt hash in every audit record
- Add batch metrics for false-positive review and reviewer disagreement
Final Notes
Moderation is not one prompt and not one score. It is a runtime gate with inputs, thresholds, review policy, failure behavior, and evidence. The probabilistic parts are useful, but they need a deterministic control surface around them.
The most important design choice in this sample is not the exact threshold value. It is the separation of responsibilities. Detoxify scores. Ollama reviews selected cases. Code decides, records, and fails closed.
Explore the source code at the GitHub repository.
See you in the next issue.
Stay curious.