Data Quality: Cleaning and Validation

Update November 2, 2026 — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: as of May 7, 2026, organizations that had never run a fine-tuning job can no longer create one; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.

Why cleaning is critical

A technically valid JSONL file doesn’t guarantee good fine-tuning. Noisy, inconsistent, or poorly formatted data will degrade model performance rather than improve it—and the worst part is that it won’t trigger any errors. The job will succeed, the model will be ready, and you’ll discover the problem only in production. This lesson covers the cleaning and validation techniques you should run before uploading.

Technical file validation

Start with the mechanics: verify that your file is syntactically correct and that every line respects the expected contract. The function below walks through the file line by line, accumulating errors instead of stopping at the first one (you want the full list, not a back-and-forth for each issue) and returns some useful statistics along the way, such as the presence of system messages and multi-turn conversations.

import json

def validate_jsonl(path: str) -> dict:
    """Validates a JSONL file and returns statistics."""
    errors = []
    stats = {"total": 0, "with_system": 0, "multi_turn": 0}
    valid_roles = {"system", "user", "assistant"}

    with open(path, "r", encoding="utf-8") as f:
        for i, line in enumerate(f, 1):
            stats["total"] += 1
            try:
                obj = json.loads(line.strip())
            except json.JSONDecodeError as e:
                errors.append(f"Line {i}: Invalid JSON — {e}")
                continue

            if "messages" not in obj:
                errors.append(f"Line {i}: Missing 'messages' key")
                continue

            messages = obj["messages"]
            if not isinstance(messages, list) or len(messages) < 2:
                errors.append(f"Line {i}: Minimum 2 messages required")
                continue

            for j, msg in enumerate(messages):
                role = msg.get("role")
                if role not in valid_roles:
                    errors.append(f"Line {i}, msg {j}: Invalid role")
                if not msg.get("content", "").strip():
                    errors.append(f"Line {i}, msg {j}: Empty content")

            if not any(m["role"] == "assistant" for m in messages):
                errors.append(f"Line {i}: No assistant message")

            if messages[0]["role"] == "system":
                stats["with_system"] += 1
            user_turns = sum(1 for m in messages if m["role"] == "user")
            if user_turns > 1:
                stats["multi_turn"] += 1

    return {"errors": errors, "stats": stats}

result = validate_jsonl("training_data.jsonl")
if result["errors"]:
    print(f"Errors: {len(result['errors'])}")
    for err in result["errors"][:10]:
        print(f"  - {err}")
else:
    print(f"File valid — {result['stats']['total']} examples")

Content cleaning

Exact or near-duplicate entries bias training by over-representing certain patterns. This happens naturally when you aggregate multiple exports: the same frequent question appears thirty times in your logs, and the model concludes it deserves thirty times more attention than others. A hash of the concatenated message content is enough to eliminate them.

import hashlib

def deduplicate(examples: list[dict]) -> list[dict]:
    """Removes duplicates based on message content."""
    seen = set()
    unique = []

    for ex in examples:
        content = "|".join(
            f"{m['role']}:{m['content']}" for m in ex["messages"]
        )
        h = hashlib.md5(content.encode()).hexdigest()

        if h not in seen:
            seen.add(h)
            unique.append(ex)

    print(f"Duplicates removed: {len(examples) - len(unique)}")
    return unique

Next comes text normalization, which addresses four distinct issues. Multiple spaces and superfluous line breaks, inherited from copy-pasting, must go. Encoding must be normalized to UTF-8 with control characters removed—invisible but often present in database exports. Punctuation deserves a check: a response truncated in the middle teaches the model to interrupt itself. Finally, ensure language consistency—if you want formal address, verify that no example uses informal address, because the model will faithfully reproduce the mix.

import re
import unicodedata

def normalize_text(text: str) -> str:
    """Normalizes text for training."""
    text = unicodedata.normalize("NFC", text)
    text = re.sub(r"[\x00-\x09\x0b-\x0c\x0e-\x1f\x7f]", "", text)
    text = re.sub(r"[ \t]+", " ", text)
    text = re.sub(r"\n{3,}", "\n\n", text)
    return text.strip()

Consistency checks

If you use a system message, it must be rigorously identical across all your examples. Two variants that differ by only one word are enough to dilute the signal: the model stops learning a specific instruction and starts learning a fuzzy family of instructions. The check is trivial, and the result is often surprising on a corpus built by multiple people.

def check_system_messages(examples: list[dict]):
    """Checks consistency of system messages."""
    system_msgs = set()
    for ex in examples:
        for msg in ex["messages"]:
            if msg["role"] == "system":
                system_msgs.add(msg["content"].strip())

    if len(system_msgs) > 1:
        print(f"Warning: {len(system_msgs)} different system messages")
        for sm in system_msgs:
            print(f"  - {sm[:80]}...")
    elif len(system_msgs) == 1:
        print("System message consistent")
    else:
        print("No system message used")

The distribution of response lengths tells the same story from another angle. Highly heterogeneous responses signal that multiple people wrote the corpus with different ideas of what a good answer looks like, or that an outlier slipped into the batch. The threshold used here—a maximum greater than five times the average—isn’t absolute, but it draws your eye to the right place.

def analyze_lengths(examples: list[dict]):
    """Analyzes the distribution of response lengths."""
    lengths = []
    for ex in examples:
        for msg in ex["messages"]:
            if msg["role"] == "assistant":
                lengths.append(len(msg["content"]))

    if lengths:
        avg = sum(lengths) / len(lengths)
        mini, maxi = min(lengths), max(lengths)
        print(f"Response length — Avg: {avg:.0f} | Min: {mini} | Max: {maxi}")
        if maxi > avg * 5:
            print("High dispersion — check for outliers")

Using the OpenAI validation API

Once you’ve run your own checks, upload the file and let OpenAI validate it as well. It’s free, immediate, and saves you from discovering a format issue when the job starts.

from openai import OpenAI

client = OpenAI()

file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

print(f"File uploaded: {file.id}")
print(f"Status: {file.status}")

If the file contains errors, the status will change to error with a detailed message. Fix the reported issues before retrying.

Pre-fine-tuning checklist

Run through this list in order before creating the job—each item corresponds to a check covered above, except the last one, which no script can replace:

• JSONL file is syntactically valid
• No duplicates
• Text normalized (UTF-8, spaces, punctuation)
• System messages are consistent
• At least 50 examples (ideally 100+)
• Train/validation split performed
• Reasonable distribution of lengths
• Human review of a random sample

Key takeaways

• Always validate JSONL syntax before uploading
• Remove duplicates to avoid overfitting
• System message consistency is essential
• Check the distribution of response lengths
• Have a human review a sample—it’s irreplaceable