Aqbeż għall-kontenut ewlieni
Malti— Agħżel lingwa
Menu

Irkupru: Tqassim f'Blokok u Klassifikazzjoni

Lezzjoni 8 / 20 Zviluppatur
F'din il-paġna
  1. Irkupru: Tqassim f’Blokok u Klassifikazzjoni
  2. Il-Pipeline tal-Irkupru
  3. Konfigurazzjoni tal-Istrateġija ta’ Chunking
  4. Limitu ta’ Rilevanza u Klassifikazzjoni
  5. Ottimizzazzjoni tal-Kwalità tal-Irkupru
  6. Kejl tal-Kwalità tal-Irkupru
  7. Strateġiji Avvanzati
  8. Punti Ewlenin

Irkupru: Tqassim f’Blokok u Klassifikazzjoni

It-Tiftix tal-Fajls jiddependi fuq pipeline sħiħ ta’ rkupru: id-dokumenti tiegħek jinqasmu f’blokok (chunks), isiru vetturi, jiġu indiċjati, u mbagħad ikunu kklassifikati skont ir-rilevanza meta tasal mistoqsija. Li tifhem dan il-pipeline jippermettilek tottimizza l-kwalità tar-risponsi.

Il-Pipeline tal-Irkupru

Meta ttella’ fajl f’ħażna vettorjali (vector store), OpenAI awtomatikament iwettaq:

  1. Parsing: Jiestra t-test raw mill-format tas-sors
  2. Chunking: Jaqsam it-test f’segmenti ta’ daqs ikkontrollat
  3. Embedding: Jagħmel kull blokka f’vettur
  4. Indexing: Jaħżen il-vetturi fl-indiċi vettorjali

Fil-ħin tal-mistoqsija:

  1. Query Embedding: Il-mistoqsija ssir vettur
  2. Vector Search: Jidentifika l-blokok l-aktar qrib
  3. Ranking: Jerġa’ jikklassifika r-riżultati skont ir-rilevanza
  4. Injection: Iddaħħal l-aqwa blokok fil-kuntest tal-mudell

Konfigurazzjoni tal-Istrateġija ta’ Chunking

Il-Chunking huwa s-setting li jmexxi kollox ieħor, u l-kompromess huwa sempliċi. Blokok qosra jagħtu riżultati ta’ tiftix preċiżi iżda jipprovdu estratti mingħajr kuntest—issib is-sentenza t-tajba iżda titlef lil xiex tirreferi. Blokok twal iżommu l-kuntest iżda midfun informazzjoni rilevanti f’test mhux rilevanti, li jdgħajjef l-iskor tas-similarità.

M’hemm l-ebda valur universali: is-setting it-tajjeb jiddependi mill-istruttura tad-dokument tiegħek, eżattament kif riassunt fit-tabella hawn taħt.

from openai import OpenAI

client = OpenAI()

# Default strategy: auto
vector_store = client.vector_stores.create(
    name="legal-base",
    chunking_strategy={"type": "auto"}
)

# Custom strategy: smaller chunks for higher precision
vector_store = client.vector_stores.create(
    name="support-faq",
    chunking_strategy={
        "type": "static",
        "static": {
            "max_chunk_size_tokens": 400,
            "chunk_overlap_tokens": 100
        }
    }
)

Għażla tad-Daqs Tajjeb tal-Blokka

Tip ta’ Kontenutmax_chunk_sizeoverlapGħaliex
FAQ, Mistoqsijiet u Tweġibiet200-40050Tweġibiet qosra u awtosuffiċjenti
Dokumentazzjoni Teknika600-800200Paragrafi bil-kuntest
Kuntratti, Testi Legali800-1200300Klawsoli twal u interkonnessi
Kodiċi Sors400-600100Funzjonijiet u klassijiet

Impatt tal-Overlap

L-Overlap isolvi problema speċifika: informazzjoni li taqsam il-fruntiera ta’ blokka inkella tkun maqsuma f’nofs, li jfisser li l-ebda nofs ma jaqbel mal-mistoqsija. Billi tagħmel overlap tal-blokok, tiggarantixxi li kull sentenza tidher sħiħa f’mill-inqas segment wieħed. L-ispiża hija r-ridondanza—overlap ta’ 300 token fuq blokok ta’ 800 token jaħżen kważi 40% test duplikat.

# Original text (simplified)
# "Clause 5.1 states the supplier must deliver within 30 days.
#  In case of delay, penalties of 1% per day apply.
#  Clause 5.2 specifies penalties are capped at 15%."

# Without overlap: Clause 5.1 and penalties might end up in different chunks
# With overlap: the link between clause and penalties is preserved

Limitu ta’ Rilevanza u Klassifikazzjoni

Il-Klassifikazzjoni (Ranking) tiddetermina liema blokok jinżammu. Tista’ tikkonfigura filtru:

response = client.responses.create(
    model="gpt-5.6-terra",
    input="What is the return policy?",
    tools=[{
        "type": "file_search",
        "vector_store_ids": [vector_store.id],
        "max_num_results": 10,
        "ranking_options": {
            "ranker": "auto",
            "score_threshold": 0.5
        }
    }]
)

Is-score_threshold jiffiltra r-riżultati taħt limitu ta’ rilevanza. Valur ta’ 0.0 jirritorna kollox, filwaqt li 1.0 ikun restrittiv wisq. Fil-prattika, bejn 0.3 u 0.6 jagħti riżultati tajbin.

Ottimizzazzjoni tal-Kwalità tal-Irkupru

Preparazzjoni tad-Dokumenti

Din hija l-aktar leva effiċjenti f’termini ta’ spiża f’din il-lezzjoni kollha, u l-unika waħda li ma tispija xejn f’settings: dokument strutturat sew jiġi indiċjat tajjeb. Uża titoli espliċiti, sentenza introduttorja għal kull taqsima, u evita tabelli konvertiti f’firda ta’ test. Bil-kontra, l-ebda setting ta’ chunking ma jista’ jsalva PDF li l-estrazzjoni tiegħu tipproduċi kolonni mħallta.

# Bad: Scanned PDF with approximate OCR
# -> Incoherent chunks, poor-quality embeddings

# Good: Structured document with headings and sections
# -> Chunks aligned with logical sections

# Pattern: Pre-process before upload
def prepare_document(raw_text: str) -> str:
    """Cleans and structures text before indexing."""
    # Remove repetitive headers/footers
    lines = raw_text.split("\n")
    lines = [l for l in lines if not is_header_footer(l)]

    # Normalize spaces
    text = "\n".join(lines)
    text = re.sub(r"\n{3,}", "\n\n", text)

    return text

Arrikkiment bil-Metadata

Il-Metadata jippermetti pre-filtru qabel it-tiftix vettorjali, li jtejjeb il-preċiżjoni:

# Upload with rich metadata
documents = [
    {"file": "hr-policy-2026.pdf", "department": "hr", "type": "policy", "year": 2026},
    {"file": "supplier-contract-a.pdf", "department": "procurement", "type": "contract", "year": 2026},
    {"file": "it-security-guide.pdf", "department": "it", "type": "guide", "year": 2025},
]

for doc in documents:
    with open(doc["file"], "rb") as f:
        file_obj = client.files.create(file=f, purpose="assistants")

    client.vector_stores.files.create(
        vector_store_id=vector_store.id,
        file_id=file_obj.id,
        attributes={
            "department": doc["department"],
            "document_type": doc["type"],
            "year": doc["year"]
        }
    )

Kejl tal-Kwalità tal-Irkupru

Mingħajr kejl, kull setting ta’ chunking huwa biss intuwizzjoni. Sett ta’ għoxrin mistoqsija fejn tkun taf it-tweġiba mistennija u d-dokument li fih huwa biżżejjed biex toġġettivizza l-proċess: ibdel parametru, erġa’ mexxi s-sett tat-test, u qabbel. Dan huwa xogħol żgħir meta mqabbel ma’ sigħat mitlufa tiddibatti dwar id-daqs tal-blokka mingħajr data.

def evaluate_retrieval(test_questions: list[dict], vs_id: str) -> dict:
    """Evaluates retrieval quality on a test set.

    Each element in test_questions contains:
    - question: the question to ask
    - expected_answer: keywords or phrases expected in the response
    """
    results = {"total": 0, "relevant": 0, "irrelevant": 0}

    for test in test_questions:
        response = client.responses.create(
            model="gpt-5.6-terra",
            input=test["question"],
            tools=[{
                "type": "file_search",
                "vector_store_ids": [vs_id],
                "max_num_results": 5
            }]
        )

        answer = response.output_text.lower()
        found_keywords = sum(
            1 for word in test["expected_answer"]
            if word.lower() in answer
        )

        results["total"] += 1
        if found_keywords >= len(test["expected_answer"]) * 0.5:
            results["relevant"] += 1
        else:
            results["irrelevant"] += 1

    results["relevance_rate"] = results["relevant"] / results["total"]
    return results

Strateġiji Avvanzati

Multi-Vector-Store

Ikkonsulta bażijiet tad-dokumenti multipli b’mod parallel:

response = client.responses.create(
    model="gpt-5.6-terra",
    input="Compare our HR policy with legal obligations",
    tools=[{
        "type": "file_search",
        "vector_store_ids": [
            vs_internal_policies,
            vs_legal_texts
        ],
        "max_num_results": 15
    }]
)

Mistoqsijiet Dekomposti

Mistoqsija li fiha talbiet multipli tipproduċi vettur medju li ma jixbah lil ebda wieħed minnhom, u twassal għal riżultati fqar ta’ tiftix għal kollox. Li taqsamha f’mistoqsijiet sekondarji separati jagħti riżultati ferm aħjar—dan huwa l-istess prinċipju tat-tiftix fuq il-web: mistoqsija preċiża taqbeż waħda eżawrjenti.

def decomposed_search(complex_question: str, vs_id: str) -> str:
    """Decomposes a complex question into sub-searches."""
    # Step 1: Decompose the question
    decomposition = client.responses.create(
        model="gpt-5.6-terra",
        input=f"Decompose this question into 2-3 simple sub-questions: {complex_question}",
    )

    # Step 2: Search for each sub-question
    sub_results = []
    for sub_q in decomposition.output_text.split("\n"):
        if sub_q.strip():
            r = client.responses.create(
                model="gpt-5.6-terra",
                input=sub_q.strip(),
                tools=[{"type": "file_search", "vector_store_ids": [vs_id]}]
            )
            sub_results.append(r.output_text)

    # Step 3: Synthesize
    synthesis = client.responses.create(
        model="gpt-5.6-terra",
        input=f"Original question: {complex_question}\n\n"
              f"Intermediate results:\n" + "\n---\n".join(sub_results) +
              "\n\nSynthesize a complete and coherent answer."
    )
    return synthesis.output_text

Punti Ewlenin

  • Il-static chunking b’daqs u overlap personalizzati jtejjeb il-preċiżjoni
  • Blokok żgħar għall-FAQs, blokok kbar għad-dokumenti legali
  • Is-score_threshold jiffiltra r-riżultati mhux rilevanti
  • Naddaf u struttura d-dokumenti tiegħek qabel ma tindiciżżahom
  • Ikk evalwa l-kwalità tal-irkupru b’sett tat-test