Evaluating LLMs
AI & ML: lesson 15 of 32
If you cannot score it, you cannot improve it.
Lesson 15 of 32 · 5 min
Evaluating LLMs
Step 1 of 14
casesfrozen
expectedanswerverdict
123
Paris
·paris·
42
42.
blue
green
An evaluation is three things: fixed inputs, an expected outcome for each, and a scorer.
The Idea
An evaluation is a fixed set of inputs, an expected outcome for each, and a scorer. Deterministic checks come first: normalised exact match, a pattern, does the code run. Softer qualities need a rubric, graded by people or by a model.
Real-World Example
A driving examiner runs the same route with a written scorecard, and a second examiner spot-checks the marking, because two people rarely agree on "smooth enough" until the rubric forces them to.
The Code
def norm(s):
return " ".join(s.lower().strip().split()).rstrip(".")
cases = [("Paris", " paris "), ("42", "42."), ("blue", "green")]
hits = sum(1 for want, got in cases if norm(want) == norm(got))
print(hits, "/", len(cases)) # 2 / 3
raw = sum(1 for want, got in cases if want == got)
print(raw, "/", len(cases)) # 0 / 3 -> unnormalised scoring liesYour turn
What does this print?
def norm(s):
return " ".join(s.lower().strip().split()).rstrip(".")
cases = [("Rome", "rome."), ("7", " 7"), ("cat", "cats")]
print(sum(1 for want, got in cases if norm(want) == norm(got)))Mini quiz
1 / 3