Self-consistency: Multiple Reasoning Paths

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (Responses API) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

Self-consistency: Multiple Reasoning Paths

Self-consistency involves generating multiple independent responses to the same prompt, then selecting the most frequent or coherent one. The principle is voting: a single call exposes you to the randomness of a bad draw, while five calls give you a distribution you can query. This technique significantly improves response reliability, especially for logical reasoning and mathematical tasks.

The Problem It Solves

An LLM can give a different answer on every call, even with a strictly identical prompt. Some of these answers are correct, others are not, and nothing in the output signals which one you are reading: a wrong answer is stated with the same confidence as a right one. Self-consistency turns this variability to your advantage. If the model reaches the same conclusion via several different reasoning paths, that conclusion is likely correct; an error, by contrast, rarely reproduces identically from one draw to the next because it depends precisely on the path taken.

Basic Implementation

The mechanism has three steps: generate, extract the final answer, vote. The marker >>> ANSWER: enforced in the instructions is not cosmetic—it is what makes the output parsable without having to interpret the reasoning that precedes it. Without this anchor, you would have to write a fragile heuristic to guess where the reflection ends and the conclusion begins.

from openai import OpenAI
from collections import Counter

client = OpenAI()

def self_consistency(prompt: str, system: str = "",
                     n: int = 5, model: str = "gpt-5.6-terra") -> dict:
    """Generates n responses and returns the most frequent one."""
    responses = []

    for _ in range(n):
        response = client.responses.create(
            model=model,
            reasoning={"effort": "none"},  # temperature requires effort none
            instructions=system + "\nReason step by step, then give "
                         "your final answer after '>>> ANSWER:'",
            input=prompt,
            temperature=0.7  # Temperature > 0 for diversity
        )
        responses.append(response.output_text)

    # Extract final answers
    answers = []
    for r in responses:
        if ">>> ANSWER:" in r:
            answer = r.split(">>> ANSWER:")[-1].strip()
            answers.append(answer)

    # Majority vote
    if answers:
        counter = Counter(answers)
        best_answer, count = counter.most_common(1)[0]
        return {
            "answer": best_answer,
            "confidence": count / len(answers),
            "all_answers": dict(counter),
            "total_responses": len(answers)
        }
    return {"answer": None, "confidence": 0}

What the Vote Tells You

Applied to sentiment classification, the function returns not just a label but a distribution, and that distribution is often more valuable than the label itself. For the review below, which is genuinely ambiguous—the product is fine, but the delivery was disastrous—four calls vote “negative” and three vote “neutral.” The 57% confidence tells you this case may deserve special handling: queueing for human review, asking the user for clarification, or simply excluding it from aggregated statistics. A single call would have displayed “negative” without nuance, and you would have treated this ticket exactly like a unanimously furious review.

system = """You are a sentiment classifier.
Analyze the text and reason step by step.
Give your final answer after '>>> ANSWER:' choosing from:
positive, negative, neutral"""

text = "The product is fine but the delivery took 3 weeks."

result = self_consistency(text, system, n=7)
print(f"Sentiment: {result['answer']}")
print(f"Confidence: {result['confidence']:.0%}")
print(f"Distribution: {result['all_answers']}")
# Example output:
# Sentiment: negative
# Confidence: 57%
# Distribution: {'negative': 4, 'neutral': 3}

Parallel Version with asyncio

The cost of this technique is not just financial: five sequential calls also mean five times the latency, which rules out the approach in any interface where someone is waiting at their screen. Yet the calls are perfectly independent; there is no justification for waiting for them one after another. asyncio.gather brings the total time down to that of the slowest call, making self-consistency compatible with online use, with an overhead that becomes purely monetary.

import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI()

async def self_consistency_async(prompt: str, system: str = "",
                                  n: int = 5) -> dict:
    """Asynchronous version of self-consistency."""

    async def single_call():
        response = await async_client.responses.create(
            model="gpt-5.6-terra",
            reasoning={"effort": "none"},  # temperature requires effort none
            instructions=system + "\nReason then give your answer "
                         "after '>>> ANSWER:'",
            input=prompt,
            temperature=0.7
        )
        return response.output_text

    # Launch all requests in parallel
    tasks = [single_call() for _ in range(n)]
    results = await asyncio.gather(*tasks)

    # Extract and vote
    answers = []
    for r in results:
        if ">>> ANSWER:" in r:
            answers.append(r.split(">>> ANSWER:")[-1].strip())

    counter = Counter(answers)
    best, count = counter.most_common(1)[0] if answers else (None, 0)

    return {
        "answer": best,
        "confidence": count / len(answers) if answers else 0,
        "distribution": dict(counter)
    }

When to Use Self-consistency

The decision criterion is simple to state: voting only makes sense if the answer is discrete and comparable. Two well-written summaries cannot be tallied, since they will never be character-for-character identical; two category labels can.

Tuning the Cost/Reliability Ratio

Three calls are the minimum viable number for a majority vote to mean anything; five offers the best compromise for most applications; seven to eleven are justified for critical decisions, where an error costs far more than a few extra calls. However, the real gain comes from how you use the confidence score rather than the sample count. Below 60%, consider that there is no consensus and escalate, either to a human operator or to a more powerful model with more samples, as shown below.

result = self_consistency(question, system, n=5)

if result["confidence"] < 0.6:
    # No clear consensus, escalate
    result = self_consistency(question, system, n=9, model="gpt-5.6-sol")

The remaining question is whether the juice is worth the squeeze in your specific case, and that must be measured, not assumed. Implement the asynchronous version, run it on a dozen sentiment classification questions where you know the expected answer, then compare the accuracy obtained with a single call versus five calls. The figure you obtain will settle the question of overhead for your specific case, something no general recommendation can do for you.

Key Takeaways

• Self-consistency generates multiple responses and votes for the best one
• Use a temperature > 0 to achieve diversity
• Effective for tasks with a discrete answer (classification, calculation)
• Useless for creative tasks without a single correct answer
• Parallelize calls with asyncio to limit latency