Evaluating LLMs

The Idea

An evaluation is a fixed set of inputs, an expected outcome for each, and a scorer. Deterministic checks come first: normalised exact match, a pattern, does the code run. Softer qualities need a rubric, graded by people or by a model.

Real-World Example

A driving examiner runs the same route with a written scorecard, and a second examiner spot-checks the marking, because two people rarely agree on "smooth enough" until the rubric forces them to.

The Code

def norm(s):
    return " ".join(s.lower().strip().split()).rstrip(".")

cases = [("Paris", " paris "), ("42", "42."), ("blue", "green")]

hits = sum(1 for want, got in cases if norm(want) == norm(got))
print(hits, "/", len(cases))    # 2 / 3

raw = sum(1 for want, got in cases if want == got)
print(raw, "/", len(cases))     # 0 / 3 -> unnormalised scoring lies

def norm(s):
  return " ".join(s.lower().strip().split()).rstrip(".")

cases = [("Rome", "rome."), ("7", " 7"), ("cat", "cats")]
print(sum(1 for want, got in cases if norm(want) == norm(got)))