بتل کد Battle Codeدارد باز می‌شود…

جواب «Which Prompt Wins?» چالش 6 از جلسه‌ی 76

سؤال

دو پرامپت سیستم را روی همان آزمون مقایسه کن. تابع evaluate(system) را بنویس که برای هر آزمون fake_model(system, question) را صدا بزند و دو عدد برگرداند: چند سؤال جواب‌دار درست جواب داده شد (کلمه‌ی کلیدی در جواب)، و چند سؤال بی‌جواب درست رد شد (I don't know.). خروجی:
A: answered 3/3, refused 0/2, total 3/5
B: answered 3/3, refused 2/2, total 5/5
winner: B

راهنما

در evaluate: reply = fake_model(system, question)؛ if key is None: refused += reply == "I don't know."؛ وگرنه answered += key in reply.lower(). برنده: max(totals, key=totals.get)؛ اگر دو عدد برابر بود winner: tie.

یک جواب درست

KNOWN = {
    "What is evaporation?": "Water turns into vapour.",
    "What is condensation?": "Vapour cools and forms clouds.",
    "What is precipitation?": "Rain or snow falls from clouds.",
}


def fake_model(system, question):
    # A pretend model: it knows three answers; for the rest it makes things up,
    # unless the system prompt allows it to say it does not know.
    if question in KNOWN:
        return KNOWN[question]
    if "i don't know" in system.lower():
        return "I don't know."
    return "It was invented in 1975 by a famous scientist."


TESTS = [   # (question, key word or None when the notes have no answer)
    ("What is evaporation?", "vapour"),
    ("What is condensation?", "clouds"),
    ("What is precipitation?", "rain"),
    ("Who was the first head teacher of our school?", None),
    ("How tall is the school roof?", None),
]
PROMPTS = {
    "A": "You are a science helper.",
    "B": "You are a science helper. If the notes do not say, answer: I don't know.",
}


def evaluate(system):
    answered = refused = 0
    for question, key in TESTS:
        reply = fake_model(system, question)
        if key is None:
            refused += reply == "I don't know."
        else:
            answered += key in reply.lower()
    return answered, refused


n_answer = sum(1 for _, key in TESTS if key is not None)
n_refuse = len(TESTS) - n_answer
totals = {}
for name, system in PROMPTS.items():
    answered, refused = evaluate(system)
    totals[name] = answered + refused
    print(f"{name}: answered {answered}/{n_answer}, refused {refused}/{n_refuse}, total {totals[name]}/{len(TESTS)}")
if totals["A"] == totals["B"]:
    print("winner: tie")
else:
    print("winner:", max(totals, key=totals.get))

اول خودت امتحان کن؛ جواب‌های دیگر هم قبول است. برگرد و خودت حلش کن