جواب «Track A: Score the Bot» چالش 2 از جلسهی 79
سؤال
مسیر الف. ربات جلسهی ۷۸ را روی آزمون پنهان اجرا کن. برای هر آزمون، جواب bot را با classify (همان جلسهی ۷۶) دستهبندی کن و چاپ کن، بعد شمارش هر دسته (ترتیب الفبا) و درصد موفقیت = (right + right refusal) ÷ کل. خروجی:right: What happens in evaporation?right: How are clouds made?right: What falls from heavy clouds?right refusal: When did our school open?over-cautious: What is a cloud made of?over-cautious 1, right 3, right refusal 1success 4/5 = 80%
راهنما
kind = classify(bot(question), expected)؛ counts[kind] = counts.get(kind, 0) + 1. درصد: good = counts.get("right", 0) + counts.get("right refusal", 0) و 100 * good / len(HIDDEN_TEST).
یک جواب درست
STOP = {"the", "a", "an", "is", "are", "and", "or", "of", "in", "on", "to", "it", "what", "how", "why", "does", "do", "when"}
CHUNKS = [
"Evaporation is when the sun heats water and it turns into vapour.",
"Condensation is when vapour cools and forms clouds.",
"Precipitation is rain or snow falling from heavy clouds.",
]
HIDDEN_TEST = [ # (question, key word or None)
("What happens in evaporation?", "vapour"),
("How are clouds made?", "cools"),
("What falls from heavy clouds?", "rain"),
("When did our school open?", None),
("What is a cloud made of?", "vapour"),
]
def tokens(text):
words = []
for w in text.lower().split():
w = "".join(ch for ch in w if ch.isalpha())
if w and w not in STOP:
words.append(w)
return words
def jaccard(a, b):
a, b = set(a), set(b)
if not a or not b:
return 0
return len(a & b) / len(a | b)
def bot(question):
scores = [jaccard(tokens(question), tokens(c)) for c in CHUNKS]
if max(scores) == 0:
return "I don't know."
return CHUNKS[scores.index(max(scores))]
def classify(reply, expected):
refused = reply == "I don't know."
if expected is None:
return "right refusal" if refused else "hallucinated"
if refused:
return "over-cautious"
return "right" if expected in reply.lower() else "wrong"
counts = {}
for question, expected in HIDDEN_TEST:
kind = classify(bot(question), expected)
counts[kind] = counts.get(kind, 0) + 1
print(f"{kind}: {question}")
print(", ".join(f"{k} {counts[k]}" for k in sorted(counts)))
good = counts.get("right", 0) + counts.get("right refusal", 0)
print(f"success {good}/{len(HIDDEN_TEST)} = {100 * good / len(HIDDEN_TEST):.0f}%")
اول خودت امتحان کن؛ جوابهای دیگر هم قبول است. برگرد و خودت حلش کن