Track A: Score the Bot چالش 2 از جلسهی 79، آسان
مسیر الف. ربات جلسهی ۷۸ را روی آزمون پنهان اجرا کن. برای هر آزمون، جواب bot را با classify (همان جلسهی ۷۶) دستهبندی کن و چاپ کن، بعد شمارش هر دسته (ترتیب الفبا) و درصد موفقیت = (right + right refusal) ÷ کل. خروجی:right: What happens in evaporation?right: How are clouds made?right: What falls from heavy clouds?right refusal: When did our school open?over-cautious: What is a cloud made of?over-cautious 1, right 3, right refusal 1success 4/5 = 80%
STOP = {"the", "a", "an", "is", "are", "and", "or", "of", "in", "on", "to", "it", "what", "how", "why", "does", "do", "when"}
CHUNKS = [
"Evaporation is when the sun heats water and it turns into vapour.",
"Condensation is when vapour cools and forms clouds.",
"Precipitation is rain or snow falling from heavy clouds.",
]
HIDDEN_TEST = [ # (question, key word or None)
("What happens in evaporation?", "vapour"),
("How are clouds made?", "cools"),
("What falls from heavy clouds?", "rain"),
("When did our school open?", None),
("What is a cloud made of?", "vapour"),
]
def tokens(text):
words = []
for w in text.lower().split():
w = "".join(ch for ch in w if ch.isalpha())
if w and w not in STOP:
words.append(w)
return words
def jaccard(a, b):
a, b = set(a), set(b)
if not a or not b:
return 0
return len(a & b) / len(a | b)
def bot(question):
scores = [jaccard(tokens(question), tokens(c)) for c in CHUNKS]
if max(scores) == 0:
return "I don't know."
return CHUNKS[scores.index(max(scores))]
def classify(reply, expected):
refused = reply == "I don't know."
if expected is None:
return "right refusal" if refused else "hallucinated"
if refused:
return "over-cautious"
return "right" if expected in reply.lower() else "wrong"
counts = {}
for question, expected in HIDDEN_TEST:
kind = "?"
print(f"{kind}: {question}")
راهنما و جواب این چالش · برگشت به درس جلسهی 79: پروژهی آزاد با مربی (۳): آزمودن · چالش بعدی: Track B: The Confusion Matrix
بیشتر بخوان: list comprehension در پایتون: از حلقه for تا یک خط کد