Moderation and Visual Content Safety
On this page
Updated November 3, 2026 based on OpenAI’s Image generation guide: A moderation refusal is identified by
error.code == "moderation_blocked", with an optionalmoderation_detailsobject, not by the presence of the wordssafetyorcontent_policyin the error message; added themoderationparameter (auto,low). Source: Image generation.
November 2, 2026 update — Image models.
gpt-image-1, on which this course was originally written, was discontinued on October 23, 2026, andgpt-image-1.5,gpt-image-1-mini, andchatgpt-image-latestwill be retired on December 1, 2026. Examples now usegpt-image-2.5-sunburst, the most precise for editing;gpt-image-2.5-flareis faster for standard generation. Both also acceptxhighandmaxqualities. GPT Image models always return the image in base64: theresponse_formatparameter from older DALL·E models is not accepted and has been removed from the code. Source: Image generation (OpenAI).
Moderation and Visual Content Safety
As long as you generate images for yourself, moderation is a matter of personal judgment. Once you open generation to users, it becomes a legal and ethical responsibility that binds you. This lesson covers OpenAI’s moderation tools, how to chain them into successive filters, and the measures that protect both your application and the people who use it.
The Built-in Moderation System
GPT Image refuses certain content on its own—violence, sexually explicit material, among others—and signals this refusal with a specific error. Your code must therefore distinguish a content policy refusal from a technical failure, because the two call for opposite reactions: the former results in a message to the user, the latter in an alert to your team. OpenAI’s documentation specifies the discriminator to use: a moderation refusal arrives as a BadRequestError with error.code equal to moderation_blocked (type image_generation_user_error), and the error body may contain a moderation_details object that specifies the stage (input, output, or unknown) and general categories (harassment, sexual, violence…). Do not search for words in the error message: its text is not a contract, the code is. The function below re-raises the exception only if it is not a moderation refusal.
from openai import OpenAI, BadRequestError
client = OpenAI()
def generate_with_moderation(prompt: str, output: str) -> dict:
"""Generates an image while handling moderation refusals."""
try:
response = client.images.generate(
model="gpt-image-2.5-sunburst",
prompt=prompt,
size="1024x1024",
quality="medium",
)
import base64
data = base64.b64decode(response.data[0].b64_json)
with open(output, "wb") as f:
f.write(data)
return {"status": "ok", "path": output}
except BadRequestError as e:
if e.code != "moderation_blocked":
raise # technical failure, quota, invalid request: handle elsewhere
body = e.body if isinstance(e.body, dict) else {}
details = body.get("moderation_details") or {}
return {
"status": "refused",
"reason": "Content refused by safety policy",
"stage": details.get("moderation_stage"), # input, output or unknown
"categories": details.get("categories") or [], # for your logs
}
result = generate_with_moderation(
"A kitten playing with a ball of yarn",
"kitten.png"
)
print(result)OpenAI advises keeping a generic message for the end user and reserving moderation_details for your logs and support. These errors do not resolve on retry: without modifying the prompt or provided images, the same request will be refused again. Note finally that filtering severity is set by the moderation parameter of GPT Image models: auto (standard filtering, default) or low (less restrictive).
Pre-filtering Prompts
Waiting for GPT Image to refuse a prompt before discovering it was problematic is unsatisfactory: you have already consumed the API call, and you learn nothing specific about the reason. The Moderation API, however, is free, fast, and returns named categories. Calling it before generation allows you to refuse cleanly, explain to the user what was flagged, and keep this information for your audit. The verify_prompt function extracts the list of triggered categories, and generate_safe_image simply chains the two steps: check first, generate second.
from openai import OpenAI
client = OpenAI()
def verify_prompt(prompt: str) -> dict:
"""Checks a prompt with the Moderation API before generation."""
moderation = client.moderations.create(
model="omni-moderation-latest",
input=prompt
)
result = moderation.results[0]
if result.flagged:
categories_flagged = [
cat for cat, flagged in result.categories.__dict__.items()
if flagged
]
return {
"allowed": False,
"categories": categories_flagged,
"message": "This prompt was flagged by the moderation system."
}
return {"allowed": True}
def generate_safe_image(prompt: str, output: str):
"""Complete pipeline: moderation + generation."""
# Step 1: pre-filtering
check = verify_prompt(prompt)
if not check["allowed"]:
print(f"Prompt refused: {check['message']}")
print(f"Categories: {check['categories']}")
return None
# Step 2: generation
print("Prompt validated, generating...")
return generate_with_moderation(prompt, output)
# Test with a harmless prompt
generate_safe_image(
"A mountain landscape with a crystal clear lake",
"mountain.png"
)Also Check Generated Images
An innocent prompt can yield an image that is not. A roundabout phrasing, the ambiguity of a term, or simply an unfortunate model interpretation is enough to produce visual content you would not want displayed under your logo. The omni-moderation-latest model accepts images as input, allowing you to close the loop: the same category mechanics apply to the generated file. The complete pipeline below chains the three phases—prompt check, generation, image check—and crucially, in case of a flag, deletes the file from disk before returning its verdict. A problematic image left on the server “while we look” is an image that will eventually leak.
from openai import OpenAI
import base64
client = OpenAI()
def moderate_generated_image(image_path: str) -> dict:
"""Analyzes a generated image to detect problematic content."""
with open(image_path, "rb") as f:
image_b64 = base64.b64encode(f.read()).decode()
moderation = client.moderations.create(
model="omni-moderation-latest",
input=[
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{image_b64}"
}
}
]
)
result = moderation.results[0]
if result.flagged:
categories_flagged = [
cat for cat, flagged in result.categories.__dict__.items()
if flagged
]
return {
"safe": False,
"categories": categories_flagged
}
return {"safe": True}
def secure_pipeline(prompt: str, output: str):
"""Complete pipeline: prompt moderation + generation + image moderation."""
# 1. Check the prompt
check_prompt = verify_prompt(prompt)
if not check_prompt["allowed"]:
return {"status": "prompt_refused", "details": check_prompt}
# 2. Generate the image
response = client.images.generate(
model="gpt-image-2.5-sunburst",
prompt=prompt,
size="1024x1024",
quality="medium",
)
data = base64.b64decode(response.data[0].b64_json)
with open(output, "wb") as f:
f.write(data)
# 3. Check the generated image
check_image = moderate_generated_image(output)
if not check_image["safe"]:
import os
os.remove(output)
return {
"status": "image_deleted",
"reason": "Generated image was flagged",
"categories": check_image["categories"]
}
return {"status": "ok", "path": output}When Users Write the Prompts
User-generated content changes the nature of the problem: you are no longer facing clumsiness but, at times, deliberate attempts at circumvention. The PromptModerator class stacks four defenses of different natures, which is the very principle of serious protection. Cleaning removes control characters through which injections transit and caps length at five hundred characters, cutting short flood prompts designed to drown instructions. Local verification via a forbidden word list—left empty, to be filled according to your policy and sector—intercepts at zero cost what you already know. The Moderation API then takes over for everything your list did not anticipate. Finally, the security prefix concatenated before the final prompt explicitly reminds the model of the expected framework, shifting the boundary in your favor for edge cases.
from openai import OpenAI
import re
client = OpenAI()
class PromptModerator:
"""Multi-layer filter for user prompts."""
def __init__(self):
self.forbidden_words = [] # Fill according to your policy
self.security_prefix = (
"Safe and appropriate image for all audiences. "
"No violent, sexual, or discriminatory content. "
)
def clean(self, prompt: str) -> str:
"""Cleans and normalizes a user prompt."""
# Remove potential injections
prompt = prompt.strip()
# Limit length
prompt = prompt[:500]
# Remove control characters
prompt = re.sub(r'[\x00-\x1f\x7f-\x9f]', '', prompt)
return prompt
def check_words(self, prompt: str) -> bool:
"""Checks for absence of forbidden words."""
prompt_lower = prompt.lower()
for word in self.forbidden_words:
if word in prompt_lower:
return False
return True
def secure(self, prompt: str) -> str:
"""Adds a security prefix to the prompt."""
return f"{self.security_prefix}{prompt}"
def pipeline(self, raw_prompt: str) -> dict:
"""Complete moderation pipeline for a user prompt."""
# Step 1: cleaning
prompt = self.clean(raw_prompt)
# Step 2: local verification
if not self.check_words(prompt):
return {"allowed": False, "reason": "Forbidden word detected"}
# Step 3: API moderation
check = verify_prompt(prompt)
if not check["allowed"]:
return {"allowed": False, "reason": check["message"]}
# Step 4: add security prefix
final_prompt = self.secure(prompt)
return {"allowed": True, "prompt": final_prompt}
moderator = PromptModerator()
# Simulate a user prompt
result = moderator.pipeline("A tabby cat sleeping on a red cushion")
if result["allowed"]:
print(f"Secure prompt: {result['prompt']}")
else:
print(f"Refused: {result['reason']}")Logging for Accountability
The day a problematic image resurfaces, the question you will be asked is: who requested it, when, and what did your system do. Without logs, you have no answer. AuditLogger writes one JSON line per generation to a .jsonl file—an append-only format that supports concurrent writing and can be read line by line without loading everything into memory. Each entry links a user ID to a truncated prompt, the generation outcome, and, if applicable, the reason for refusal. The report method then derives the acceptance rate over the last hundred generations and details refusals: this is the dashboard that will alert you if a particular account accumulates flags.
import json
import os
from datetime import datetime
class AuditLogger:
"""Audit log for image generations."""
def __init__(self, log_file: str = "audit_images.jsonl"):
self.log_file = log_file
def log(
self,
prompt: str,
user_id: str,
result: str,
image_path: str = None,
refusal_reason: str = None
):
"""Logs an audit entry."""
entry = {
"timestamp": datetime.now().isoformat(),
"user_id": user_id,
"prompt": prompt[:200],
"result": result, # "ok", "prompt_refused", "image_deleted"
"image_path": image_path,
"refusal_reason": refusal_reason,
}
with open(self.log_file, "a") as f:
f.write(json.dumps(entry, ensure_ascii=False) + "\n")
def report(self, last_n: int = 100):
"""Generates a report of recent generations."""
if not os.path.exists(self.log_file):
print("No logs found.")
return
entries = []
with open(self.log_file, "r") as f:
for line in f:
entries.append(json.loads(line))
recent = entries[-last_n:]
total = len(recent)
ok = sum(1 for e in recent if e["result"] == "ok")
refused = sum(1 for e in recent if e["result"] != "ok")
print(f"Report of last {total} generations:")
print(f" Accepted: {ok} ({100*ok/total:.1f}%)")
print(f" Refused : {refused} ({100*refused/total:.1f}%)")
if refused > 0:
print(f"\nRefusal details:")
for e in recent:
if e["result"] != "ok":
print(f" [{e['timestamp'][:10]}] {e['user_id']} : {e['refusal_reason']}")
logger = AuditLogger()
logger.report()Copyright and Trademarks
GPT Image knows how to produce images that evoke an artist’s style or a brand’s visual identity, and this capability is exactly what exposes you. Technical moderation does not protect you on this ground: it does not detect that a poster looks very much like a competitor’s. Protection is therefore contractual and organizational, taking the form of a written, enforceable usage policy that your users accept and your team applies. The PRODUCTION_RULES block gives the minimal formulation, to be displayed in your documentation as well as recalled in your terms.
# Best practices to integrate into your usage policy
PRODUCTION_RULES = """
1. Do not request imitation of a living artist's style by name
2. Do not reproduce logos or registered trademarks
3. Do not generate faces of real people
4. Add an "AI-generated image" notice when required
5. Retain generation logs for at least 12 months
6. Allow users to report problematic content
"""
def usage_warning():
"""Displays production usage rules."""
print("=" * 60)
print("PRODUCTION RULES — AI Image Generation")
print("=" * 60)
print(PRODUCTION_RULES)GDPR Compliance
For a European company, retention periods and consent are not configuration options but obligations whose respect must be demonstrable. Recording them in an explicit configuration object, rather than scattering them through the code, gives you a single place to show during an audit. ComplianceConfig thus sets log retention to one year, image retention to ninety days, and makes the AI notice and consent mandatory. The add_ai_metadata method deserves special attention: it writes into the PNG itself, via PngInfo, the fact that the image is generated, the tool that produced it, and the date. These metadata travel with the file, including outside your platform, which no notice displayed on a web page does.
class ComplianceConfig:
"""Compliance configuration for image generation."""
def __init__(self):
self.log_retention_days = 365
self.image_retention_days = 90
self.ai_notice_mandatory = True
self.consent_required = True
def verify_consent(self, user_id: str) -> bool:
"""Verifies that the user has consented to AI generation."""
# Implement with your consent management system
return True
def add_ai_metadata(self, image_path: str):
"""Adds metadata indicating AI generation."""
from PIL import Image
from PIL.PngImagePlugin import PngInfo
img = Image.open(image_path)
metadata = PngInfo()
metadata.add_text("AI-Generated", "true")
metadata.add_text("Generator", "GPT Image via OpenAI API")
metadata.add_text("Date", datetime.now().isoformat())
img.save(image_path, pnginfo=metadata)
print(f"AI metadata added: {image_path}")Before Opening the Floodgates
Eight provisions must be in place the day your system becomes accessible, and it is more honest to ask them before rather than after. Moderation first, in its two phases: pre-filtering prompts and post-verifying images, as secure_pipeline does. Logging next, complete, associating each generation with its prompt, author, and outcome—this is your only proof in case of dispute, and it must be saved independently of the images themselves, otherwise a storage purge would take your traceability with it.
Then comes the user side. A rate limit per account prevents a single subscriber from consuming your monthly budget in one night, voluntarily or via a poorly written script. Information about the generated nature of images relates to both fairness and, increasingly, regulatory constraint; it is doubled by technical marking in the file’s PNG or EXIF metadata. The right to erasure assumes you can find and delete all of a user’s data on simple request, which is only possible if your audit entries carry a user_id. Finally, open a reporting channel: your users will always see what is problematic before you do, and taking away their means to tell you does not make the problem disappear.
Now assemble the whole into a single image generation service: pre-filtering prompts via Moderation API and word list, security prefix, post-verification of the produced image, audit log, and AI metadata on every file. Test it with five varied prompts, at least one of which must be refused—this is the only way to verify that your guardrails actually work rather than just being present in the code.
Test Your Knowledge
Visual generation via API: validate reflexes before production.
1. What do inpainting and outpainting allow?
Answer: Inpainting modifies a specific area of an existing image (replace, correct); outpainting extends the image beyond its original frame—two targeted edits without regenerating everything.
2. What makes a good advanced visual prompt?
Answer: Describe subject, composition, lighting, and perspective—not just the theme: the more the prompt frames the scene like a director of photography, the more the image obeys.
3. How to work effectively by iterations?
Answer: Start with a base generation, then request targeted variations (style, framing, detail) rather than rewriting the prompt from scratch—keep what works, only tweak the gap.
4. What levers to optimize generation costs?
Answer: Adapt quality and resolution to the real need (drafts in low, finals in high), limit iterations with better prompts, and cache/reuse validated visuals.
5. Why is visual content moderation essential?
Answer: Because the API and your terms impose limits (forbidden content, real people, brands): filtering prompts and controlling outputs protects your users and your application.
Frame, iterate, master costs and compliance: visual production is a pipeline—not a roll of the dice.