Tree-of-Thought: Structured Exploration

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (Responses API) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

Tree-of-Thought: Structured Exploration

Tree-of-Thought (ToT) is a prompting technique that structures the model’s reasoning as a decision tree. Instead of following a single, linear path of thought, the model explores multiple branches, evaluates each one, and keeps only the most promising. The resemblance to a search algorithm is direct: you generate candidates, score them, and prune the rest. This makes it highly effective for problems requiring planning or backtracking.

Difference from Chain-of-Thought

Chain-of-Thought (CoT) follows linear reasoning: step 1, step 2, step 3. Its weakness is structural. If the model heads down the wrong path at step 2, everything that follows inherits that error, and there is no mechanism to go back. The reasoning may be perfectly articulated, but it starts from a false premise. Tree-of-Thought explores multiple directions at each step and discards dead ends before investing further development in them.

# Chain-of-Thought (linear)
# Step 1 → Step 2 → Step 3 → Answer

# Tree-of-Thought (tree-based)
#            Step 1
#          /    |    \
#     Opt A  Opt B  Opt C    ← evaluation
#       |      |      ✗      ← pruning
#    Opt A1  Opt B1
#      |       |
#   Answer  Answer            ← final selection

Single-Prompt Implementation

The simplest version fits in a single call: the prompt forces the model to follow an exploration structure—generate three approaches, evaluate them, pick one, and develop it. This is inexpensive and often sufficient to tackle an open-ended question. Be aware of one limitation before relying on it: the model proposes and evaluates in the same breath, making it lenient toward its own ideas and resulting in less severe pruning than necessary.

from openai import OpenAI

client = OpenAI()

def tree_of_thought_simple(problem: str) -> str:
    """ToT in a single prompt."""
    prompt = f"""Solve this problem using the Tree-of-Thought approach.

Problem: {problem}

Step 1 - Generate 3 different approaches:
- Approach A: ...
- Approach B: ...
- Approach C: ...

Step 2 - Evaluate each approach (strengths and weaknesses):
- Approach A: ...
- Approach B: ...
- Approach C: ...

Step 3 - Select the best approach and develop it:
Selected approach: ...
Detailed development: ...

Step 4 - Final answer:
..."""

    response = client.responses.create(
        model="gpt-5.6-sol",
        reasoning={"effort": "none"},  # temperature requires effort none
        input=prompt,
        temperature=0.7
    )
    return response.output_text

Multi-Call Implementation

For complex problems, separate each step into distinct calls. This breakdown gives you two things a single prompt cannot. First, step-by-step temperature control: 0.8 for generation, where diversity of proposals is the whole point of the method, then 0.3 for evaluation and development, where you want rigor and reproducibility instead. Second, intermediate results you can inspect, log, or even submit for human validation before letting the chain continue.

def tree_of_thought(problem: str, breadth: int = 3,
                    depth: int = 3) -> dict:
    """ToT with multi-level exploration."""
    system = "You are an expert in structured problem solving."

    # Step 1: Generate initial branches
    gen_prompt = (f"Problem: {problem}\n\n"
                  f"Propose {breadth} different approaches to solve "
                  f"this problem. For each approach, provide a title and "
                  f"a 2-3 sentence description. Number them 1 to {breadth}.")

    branches_response = client.responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        instructions=system,
        input=gen_prompt,
        temperature=0.8
    )
    branches = branches_response.output_text

    # Step 2: Evaluate each branch
    eval_prompt = (f"Problem: {problem}\n\n"
                   f"Here are {breadth} proposed approaches:\n{branches}\n\n"
                   f"Evaluate each approach on 3 criteria (score /10):\n"
                   f"- Feasibility\n- Efficiency\n- Simplicity\n\n"
                   f"Rank them from best to worst with justification.")

    eval_response = client.responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        instructions=system,
        input=eval_prompt,
        temperature=0.3
    )
    evaluation = eval_response.output_text

    # Step 3: Develop the best branch
    dev_prompt = (f"Problem: {problem}\n\n"
                  f"Evaluation of approaches:\n{evaluation}\n\n"
                  f"Develop the highest-ranked approach in detail. "
                  f"Provide a complete, actionable solution.")

    solution = client.responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        instructions=system,
        input=dev_prompt,
        temperature=0.3
    )

    return {
        "branches": branches,
        "evaluation": evaluation,
        "solution": solution.output_text
    }

Worked Example: Software Architecture

The problem below is exactly the kind where ToT adds value. Several strategies are defensible—caching, read replicas, extracting the search service, blue-green deployment—and the choice depends on constraints the prompt states explicitly, especially team size. A direct prompt would give you the solution most often cited in technical literature, regardless of these constraints; ToT pits it against the others and must defend its ranking based on feasibility criteria.

problem = """
Our monolithic REST API (Django, PostgreSQL) is hitting its limits:
- Max 500 requests/second
- Response time > 2s on search endpoints
- Deployment = 30 minutes of downtime
- 3 developers on the team

How can we improve performance and availability
without rewriting the entire application?
"""

result = tree_of_thought(problem, breadth=4)
print(result["solution"])

Where ToT Wins, Where It Wastes Your Time

The technique shines on planning problems—architecture, roadmaps, strategy—on constrained problems like optimization or resource allocation, on complex debugging where multiple causes remain plausible simultaneously, and on technical decisions like choosing a framework or database. These situations share two traits: multiple valid solutions exist, and an explicit criterion exists to distinguish between them.

Conversely, skip it for simple factual questions where there is nothing to explore and you would pay for three calls to get an obvious answer. It also fails on open-ended creative tasks due to the lack of a defensible evaluation criterion: ranking three advertising slogans on a feasibility scale makes no sense. Finally, avoid it when latency is critical, since the multi-call version mechanically chains several API requests before returning anything.

Combining with Reasoning Settings

At high reasoning settings, GPT-5.6 models already perform structured internal exploration. Explicit ToT then becomes largely redundant and charges you twice for the same work. Reserve it for situations where reasoning is disabled or low, as in the comparison below.

# With low reasoning: explicit ToT is useful
result = tree_of_thought(problem, breadth=3)

# With high reasoning: the model already reasons deeply
response = client.responses.create(
    model="gpt-5.6-sol",
    input=problem,
    reasoning={"effort": "high"}
)

To decide in your own context, take an architecture or design problem you actually have on your plate this week, run it through multi-call ToT with breadth=3, then submit it as-is in a direct prompt. Compare the two solutions and, above all, the token cost of each. For a high-stakes decision, the quality gap more than pays for the extra cost; for a secondary question, it does not, and knowing this will keep you from applying the technique everywhere out of habit.

Key Takeaways

• ToT explores multiple reasoning paths and selects the best one
• Simple single-prompt version or advanced multi-call version
• Ideal for planning, architecture, and complex debugging
• At high reasoning settings, GPT-5.6 models already have implicit ToT
• The extra token cost is justified for high-impact decisions