DPO (Direct Preference Optimization)

Update November 2, 2026 — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: since May 7, 2026, organizations that have never used it can no longer create jobs; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.

What Is DPO?

DPO (Direct Preference Optimization) is a fine-tuning method that teaches a model to prefer certain responses over others. Instead of showing only the correct answer, as we’ve done so far, you show a pair: a preferred response and a non-preferred response. The model learns to maximize the probability of the first while minimizing that of the second. The difference runs deeper than it appears: in supervised fine-tuning, you define a target; in DPO, you define a boundary.

When to Use It

DPO makes sense first when nuances matter—that is, when the difference between a good and an excellent response is subtle. Two responses can both be accurate, well-formatted, and polite, yet only one matches what you want to deliver; that distinction doesn’t come across well with an isolated example, but works perfectly with a comparison. It also becomes the natural tool when you want to avoid certain behaviors: supervised fine-tuning has no way to express a counter-example, whereas DPO explicitly shows the model what not to do.

Next comes anything related to alignment—safety, compliance, brand tone—areas where acceptable and problematic phrasing often differ by just a few words. Finally, if your teams have already produced human preferences, for example evaluators who ranked response A against response B, you have your DPO corpus without having to build it: those rankings are your pairs.

Data Format

The format uses the dpo method and replaces the messages array with three keys: the shared context in input, then the two compared outcomes. Note below that the non-preferred response isn’t absurd—it’s simply defensive and shuts down the conversation, whereas the preferred one acknowledges the objection and proposes a next step.

{
  "method": "dpo",
  "input": [
    {
      "role": "system",
      "content": "You are a professional sales assistant."
    },
    {
      "role": "user",
      "content": "Your product is too expensive."
    }
  ],
  "preferred_output": [
    {
      "role": "assistant",
      "content": "I understand your concern about the price. Our solution includes premium support, unlimited updates, and a 99.9% availability guarantee. Compared to alternatives that charge for these services separately, our value for money is very competitive. May I show you a comparative analysis?"
    }
  ],
  "non_preferred_output": [
    {
      "role": "assistant",
      "content": "It's not expensive at all, it's the market price. If you can't afford it, we have a basic plan."
    }
  ]
}

Generating the file follows the same logic as for supervised fine-tuning: one pair per line, UTF-8 encoding, ensure_ascii=False. The two pairs below illustrate the same preference criterion applied to two different situations, which is exactly what you want to achieve across the entire corpus.

import json

pairs = [
    {
        "method": "dpo",
        "input": [
            {"role": "system", "content": "Friendly customer support assistant."},
            {"role": "user", "content": "Your service isn't working!"}
        ],
        "preferred_output": [
            {
                "role": "assistant",
                "content": "I'm sorry for the inconvenience. Can you describe the problem you're experiencing? I'll do my best to resolve it quickly."
            }
        ],
        "non_preferred_output": [
            {
                "role": "assistant",
                "content": "It works fine for everyone else. Have you tried restarting your computer?"
            }
        ]
    },
    {
        "method": "dpo",
        "input": [
            {"role": "system", "content": "Friendly customer support assistant."},
            {"role": "user", "content": "I want a refund."}
        ],
        "preferred_output": [
            {
                "role": "assistant",
                "content": "I understand your request. Can you give me your order number? I'll check your refund eligibility and guide you through the process."
            }
        ],
        "non_preferred_output": [
            {
                "role": "assistant",
                "content": "Refunds aren't possible after 14 days. Read the T&Cs."
            }
        ]
    }
]

with open("dpo_training.jsonl", "w", encoding="utf-8") as f:
    for pair in pairs:
        f.write(json.dumps(pair, ensure_ascii=False) + "\n")

print(f"DPO file created: {len(pairs)} pairs")

Launching the Job

The upload doesn’t change; it’s the method parameter passed when creating the job that switches training to DPO mode.

from openai import OpenAI

client = OpenAI()

# Upload the DPO file
file = client.files.create(
    file=open("dpo_training.jsonl", "rb"),
    purpose="fine-tune"
)

# Create the job with the DPO method
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-5.6-luna",
    method={
        "type": "dpo",
        "dpo": {
            "hyperparameters": {
                "beta": 0.1  # Controls the strength of the preference
            }
        }
    },
    suffix="dpo-support-v1"
)

print(f"DPO job: {job.id}")

The beta controls the strength of DPO regularization, that is, the freedom you give the model to deviate from its original behavior. A low beta, between 0.05 and 0.1, allows the model to move further away from the base model to follow your preferences: the change is noticeable, but you risk taking along qualities you didn’t intend to touch. A high beta, between 0.2 and 0.5, keeps the model closer to the base model and produces conservative changes—the right setting when you’re correcting a tone inflection on a model that otherwise suits you.

Building Usable Pairs

The quality of a DPO corpus is judged more on the non-preferred responses than on the preferred ones. Both responses must be plausible: if the non-preferred one is absurd, the model learns to avoid absurdity, which it already knew how to do, and you’ve gained nothing. The difference must therefore be clear without being caricatured, and focus on the criterion you really want to teach. Then vary the types of situations covered, otherwise your model will apply the learned preference to one context and one only.

Consistency of judgments is the other half of the work, and it’s prepared before the first pair. Explicitly define your preference criteria, have several people evaluate, and keep only the pairs on which they agree—a disagreement among evaluators signals an ambiguous pair, from which the model will draw only a contradictory signal. Finally, keep a written record of these criteria. At the next iteration, led by other reviewers, you’ll judge on the same basis instead of replaying the discussion from scratch.

One last remark on sequencing methods. The most effective strategy often consists of not choosing: supervised fine-tuning first, to teach the basic format and style, then DPO applied to the already fine-tuned model to refine its nuances. Each does what it does best, in the order that’s useful.

Key Takeaways

• DPO teaches the model to prefer certain responses over others
• The format uses preferred/non_preferred pairs
• The beta parameter controls the balance between preferences and stability
• DPO is ideal for tone nuances, safety, and alignment
• Combining SFT then DPO often yields the best results