Skip to content
Cheat sheets

CCAR-F · Domain 5: Context Management & Reliability · 15% of the exam

Task 5.5: Design human review workflows and confidence calibration

Measuring accuracy by document type and field rather than overall, stratified sampling of high-confidence output, field-level confidence calibrated on labeled data, and sending reviewers the cases that need them.

Study it

Sample questions

Try each one before opening the answer. Every option is explained, with the Anthropic documentation page that proves it.

Question 1 · choose 1

Your extraction system reports 97 percent field accuracy on a labeled validation set, and management wants to stop human review of high-confidence extractions. A breakdown shows the dates on handwritten intake forms are correct only 71 percent of the time. What should you do before reducing review?

  1. AReduce review everywhere, since 97 percent overall meets the target that was agreed
  2. BAdd more examples to the prompt and re-measure the overall accuracy before deciding
  3. CMeasure accuracy by document type and field, and keep review where a segment falls short
  4. DRaise the confidence threshold for all document types and remove human review for anything above it
Show the answer and why

  • AReduce review everywhere, since 97 percent overall meets the target that was agreed

    Incorrect

    The average hides a segment where nearly three dates in ten are wrong.

  • BAdd more examples to the prompt and re-measure the overall accuracy before deciding

    Incorrect

    Another aggregate number can again hide how each segment performs.

  • CMeasure accuracy by document type and field, and keep review where a segment falls short

    Correct

    An aggregate figure can hide weak segments; automation decisions need per-segment evidence.

  • DRaise the confidence threshold for all document types and remove human review for anything above it

    Incorrect

    One global threshold does not show whether confidence is reliable for handwritten dates at all.

Aggregate accuracy can mask poor performance on specific document types or fields. Validate by segment before automating, and keep human review where a segment is weak.

Anthropic documentation

Task 5.5Report a problem with this question

Question 2 · choose 1

You tuned your extraction prompt against 200 labeled documents until it scored 98 percent on them. Before reducing human review, you want an honest estimate of accuracy on new documents. What should you measure on?

  1. AThe same 200 documents, rerun several times
  2. BA dozen hand-picked easy documents
  3. CThe model's own rating of its accuracy
  4. DA held-out labeled set kept out of tuning
Show the answer and why

  • AThe same 200 documents, rerun several times

    Incorrect

    Reruns repeat the same optimistic estimate.

  • BA dozen hand-picked easy documents

    Incorrect

    An easy sample overstates accuracy.

  • CThe model's own rating of its accuracy

    Incorrect

    A self-rating is not a measurement.

  • DA held-out labeled set kept out of tuning

    Correct

    Scores on the tuning set are inflated by fitting to it.

Measure against held-out data that mirrors real inputs; tuning-set scores overestimate how the system will perform.

Anthropic documentation

Task 5.5Report a problem with this question

Question 3 · choose 1

Your extraction system auto-approves fields with high calibrated confidence. Audits find a recurring miss: invoices whose own figures disagree, such as a total on page one that differs from the total on page three. Claude extracts one of the totals with high confidence, and nobody sees the conflict. How should routing change?

  1. ARaise the confidence threshold for totals
  2. BSend documents whose own figures contradict each other to human review, whatever the confidence score
  3. CAlways auto-approve the larger of the two totals
  4. DRemove totals from the extraction so that the conflict cannot arise
Show the answer and why

  • ARaise the confidence threshold for totals

    Incorrect

    The extraction is confident because one printed total is read correctly; a higher threshold does not reveal that the document contradicts itself.

  • BSend documents whose own figures contradict each other to human review, whatever the confidence score

    Correct

    A contradictory source needs a human decision about which value is right; that is a routing signal separate from extraction confidence.

  • CAlways auto-approve the larger of the two totals

    Incorrect

    Choosing by a rule hides the contradiction instead of resolving it.

  • DRemove totals from the extraction so that the conflict cannot arise

    Incorrect

    The downstream system needs the total; dropping it hides the problem.

Extraction confidence is not the only routing signal. A document whose own content is ambiguous or contradictory goes to human review, because a confident reading of a self-contradicting document is still a guess.

Anthropic documentation

Task 5.5Report a problem with this question

Question 4 · choose 1

Human reviewers correct a few hundred extractions every week. Your evaluation set was built six months ago and no longer reflects today's documents. How can the corrections help?

  1. AReplace the whole evaluation set with this week's corrections
  2. BUse them only to count reviewer workload
  3. CAdd corrected cases to the labeled set
  4. DDiscard them once the extraction is fixed
Show the answer and why

  • AReplace the whole evaluation set with this week's corrections

    Incorrect

    A set of only corrected failures would skew the measurement.

  • BUse them only to count reviewer workload

    Incorrect

    Workload counts do not improve measurement.

  • CAdd corrected cases to the labeled set

    Correct

    Fresh labeled cases keep evaluation current with real inputs.

  • DDiscard them once the extraction is fixed

    Incorrect

    This throws away up-to-date labeled data.

Feed reviewed cases back into the labeled evaluation set so accuracy is measured on current, representative documents.

Anthropic documentation

Task 5.5Report a problem with this question

Question 5 · choose 1

Before reducing human review, you measure per-field accuracy of your extractions against a labeled set, using exact string comparison. The report says invoice dates are right only 61 percent of the time, but spot checks show most "errors" are labels written as 2026-03-07 and extractions written as 7 March 2026. What should you change?

  1. AGrade dates with an LLM judge that decides whether two dates match
  2. BNormalize both values to one canonical format before the exact comparison
  3. CRemove dates from the accuracy report until the formats agree
  4. DLower the accuracy target for dates to 60 percent
Show the answer and why

  • AGrade dates with an LLM judge that decides whether two dates match

    Incorrect

    A model grader is slower and less consistent than code for a value with a clear canonical form.

  • BNormalize both values to one canonical format before the exact comparison

    Correct

    Exact-match grading is usually run after normalization; once format differences are removed, the comparison measures whether the date is right.

  • CRemove dates from the accuracy report until the formats agree

    Incorrect

    This hides a field you need to measure before reducing review.

  • DLower the accuracy target for dates to 60 percent

    Incorrect

    The measurement is wrong, not the target; lowering it would accept real errors too.

Per-field accuracy only supports a decision about human review if the measurement is sound. For fields with a canonical form, grade in code with an exact or normalized comparison, normalizing format differences first so they do not count as errors.

Anthropic documentation

Task 5.5Report a problem with this question

Question 6 · choose 1

Before you decide whether to reduce human review of your invoice extractions, you need an error rate you can trust. In the time available, two experts can check every field of 150 extractions, or a script can compare every field of 3,000 extractions with records the accounting team later confirmed, which contain occasional errors of their own. Which measurement should you use?

  1. AThe 150 expert checks, since careful grading matters more than the number of documents
  2. BExpert checks of all 3,000 extractions, however many weeks that takes
  3. CExpert checks of the 150 extractions with the lowest confidence scores
  4. DThe 3,000 automated comparisons with the confirmed records
Show the answer and why

  • AThe 150 expert checks, since careful grading matters more than the number of documents

    Incorrect

    150 documents give a wide margin of error on each field's rate and few chances to see rarer mistakes.

  • BExpert checks of all 3,000 extractions, however many weeks that takes

    Incorrect

    Hand grading is slow and expensive, and this measurement would not fit the time available.

  • CExpert checks of the 150 extractions with the lowest confidence scores

    Incorrect

    The least confident results are not representative, so their error rate says little about the rest.

  • DThe 3,000 automated comparisons with the confirmed records

    Correct

    Both options grade the same fields; many cases with a slightly noisier automated grade generally beat a few hand-graded ones.

Anthropic's evaluation guidance says to automate grading where possible and to prioritize volume: more cases with slightly lower-signal automated grading beat fewer hand-graded ones. Here the script grades the same fields as the experts, so the 3,000 comparisons give a tighter estimate and more chances to catch rarer errors.

Anthropic documentation

Task 5.5Report a problem with this question

Practise domain 5 →Practise all domains →