جواب «Step 4: With and Without RAG» چالش 4 از جلسهی 72
سؤال
بسنج، فرض نکن. برای هر سؤال آزمون، یک بار ask_plain(question) (مدل بدون زمینه) و یک بار ask_rag(question) (با بهترین تکه) را صدا بزن. جواب درست است اگر کلمهی کلیدی آزمون را (با حروف کوچک) داشته باشد. برای هر سؤال plain ✗ rag ✓ و مانند آن را چاپ کن و در آخر دو امتیاز. خروجی:What is condensation? plain ✗ rag ✓Why does rain fall? plain ✗ rag ✓What is evaporation? plain ✗ rag ✓without RAG 0/3, with RAG 3/3
راهنما
correct یعنی key in answer.lower(). در حلقه: p = correct(ask_plain(question), key) و r = correct(ask_rag(question), key)؛ امتیازها را با += جمع کن؛ علامت: "✓" if p else "✗".
یک جواب درست
STOP = {"the", "a", "an", "is", "are", "and", "or", "of", "in", "on", "to", "into", "it", "this",
"that", "what", "how", "why", "does", "do", "as", "when", "they", "called", "its"}
CHUNKS = [
"Evaporation is when the sun heats water in seas and lakes and the warm water turns into vapour and rises.",
"Condensation is when the vapour cools high in the sky and forms clouds.",
"Precipitation is when the drops in a cloud get heavy and fall as rain or snow.",
"Transpiration is when plants lose water through their leaves.",
]
GUESSES = { # what the model "remembers" without notes: confident and wrong
"What is condensation?": "Condensation is when water turns into ice.",
"Why does rain fall?": "Rain falls because the sky gets tired.",
"What is evaporation?": "Evaporation is when water disappears forever.",
}
TESTS = [("What is condensation?", "clouds"), ("Why does rain fall?", "heavy"), ("What is evaporation?", "vapour")]
def tokens(text):
words = []
for w in text.lower().split():
w = "".join(ch for ch in w if ch.isalpha())
if w and w not in STOP:
words.append(w)
return words
def jaccard(a, b):
a, b = set(a), set(b)
if not a or not b:
return 0
return len(a & b) / len(a | b)
def best_chunk(question, chunks):
q = tokens(question)
best, best_score = None, 0
for i, chunk in enumerate(chunks):
s = jaccard(q, tokens(chunk))
if s > best_score:
best, best_score = i, s
return best, best_score
def ask_plain(question):
return GUESSES.get(question, "I don't know.")
def ask_rag(question):
index, score = best_chunk(question, CHUNKS)
return CHUNKS[index] if index is not None else "Not in the notes."
def correct(answer, key):
return key in answer.lower()
plain_score = rag_score = 0
for question, key in TESTS:
p = correct(ask_plain(question), key)
r = correct(ask_rag(question), key)
plain_score += p
rag_score += r
print(question, "plain", "✓" if p else "✗", "rag", "✓" if r else "✗")
print(f"without RAG {plain_score}/{len(TESTS)}, with RAG {rag_score}/{len(TESTS)}")
اول خودت امتحان کن؛ جوابهای دیگر هم قبول است. برگرد و خودت حلش کن