Which Prompt Wins? چالش 6 از جلسهی 76، سخت
دو پرامپت سیستم را روی همان آزمون مقایسه کن. تابع evaluate(system) را بنویس که برای هر آزمون fake_model(system, question) را صدا بزند و دو عدد برگرداند: چند سؤال جوابدار درست جواب داده شد (کلمهی کلیدی در جواب)، و چند سؤال بیجواب درست رد شد (I don't know.). خروجی:A: answered 3/3, refused 0/2, total 3/5B: answered 3/3, refused 2/2, total 5/5winner: B
KNOWN = {
"What is evaporation?": "Water turns into vapour.",
"What is condensation?": "Vapour cools and forms clouds.",
"What is precipitation?": "Rain or snow falls from clouds.",
}
def fake_model(system, question):
# A pretend model: it knows three answers; for the rest it makes things up,
# unless the system prompt allows it to say it does not know.
if question in KNOWN:
return KNOWN[question]
if "i don't know" in system.lower():
return "I don't know."
return "It was invented in 1975 by a famous scientist."
TESTS = [ # (question, key word or None when the notes have no answer)
("What is evaporation?", "vapour"),
("What is condensation?", "clouds"),
("What is precipitation?", "rain"),
("Who was the first head teacher of our school?", None),
("How tall is the school roof?", None),
]
PROMPTS = {
"A": "You are a science helper.",
"B": "You are a science helper. If the notes do not say, answer: I don't know.",
}
def evaluate(system):
answered = refused = 0
return answered, refused
n_answer = sum(1 for _, key in TESTS if key is not None)
n_refuse = len(TESTS) - n_answer
totals = {}
for name, system in PROMPTS.items():
answered, refused = evaluate(system)
totals[name] = answered + refused
print(f"{name}: answered {answered}/{n_answer}, refused {refused}/{n_refuse}, total {totals[name]}/{len(TESTS)}")
راهنما و جواب این چالش · برگشت به درس جلسهی 76: ارزیابی مدل
بیشتر بخوان: تابع در پایتون: آموزش def، پارامتر و return با مثال