Skip to content
Cheat sheets

CCAR-F · Domain 4: Prompt Engineering & Structured Output · 20% of the exam

Task 4.4: Implement validation, retry, and feedback loops for extraction quality

Retrying with the specific validation error, knowing when a retry cannot help because the data is not in the source, self-checking fields such as calculated versus stated totals, and tracking which patterns produce dismissed findings.

Study it

  • Validation and retry loops, and when a retry cannot help

    Partly covered by: Guardrails

Sample questions

Try each one before opening the answer. Every option is explained, with the Anthropic documentation page that proves it.

Question 1 · choose 1

Your extraction pipeline validates every result with Pydantic. About 4 percent fail, mostly dates written like "3rd of March" where the model must output YYYY-MM-DD, and amounts with currency symbols attached. How should the pipeline retry these documents?

  1. ARoute the failures straight to manual data entry, since automated retries rarely fix extraction errors
  2. BSend the identical request again, since a second sample will often produce a valid result
  3. CSend only the validation error message back to Claude and ask for a corrected result
  4. DSend a follow-up with the original document, the failed extraction and the specific validation errors
Show the answer and why

  • ARoute the failures straight to manual data entry, since automated retries rarely fix extraction errors

    Incorrect

    Format mismatches like these are the kind of error a guided retry usually fixes, so sending them all to people wastes reviewer time.

  • BSend the identical request again, since a second sample will often produce a valid result

    Incorrect

    Without feedback, the same prompt tends to repeat the same formatting mistake.

  • CSend only the validation error message back to Claude and ask for a corrected result

    Incorrect

    Without the document and the failed extraction, Claude cannot see which values to fix or where they came from.

  • DSend a follow-up with the original document, the failed extraction and the specific validation errors

    Correct

    Specific errors next to the source and the failed output give Claude what it needs to correct exactly the fields that failed.

Retry with error feedback: include the original document, the failed output and the specific validation errors so the model can correct format and structural mistakes.

Anthropic documentation

Task 4.4Report a problem with this question

Question 2 · choose 1

Your contract extraction system retries any document whose renewal_terms field fails validation. For one vendor, every retry fails: the contract says only "renewal terms as set out in Addendum B," and the addendum is a separate file the pipeline never receives. What should the pipeline do with these documents?

  1. ARaise the temperature on retries so that the model tries different readings of the contract
  2. BRaise the retry limit to five attempts so that the model has more chances to find the terms
  3. CAdd few-shot examples showing renewal terms extracted from other vendors' contracts
  4. DStop retrying, mark renewal_terms as missing from the source, and flag the contract for follow-up
Show the answer and why

  • ARaise the temperature on retries so that the model tries different readings of the contract

    Incorrect

    More varied sampling cannot recover text that is not there, and it makes a fabricated value more likely.

  • BRaise the retry limit to five attempts so that the model has more chances to find the terms

    Incorrect

    More attempts at a document that lacks the data only add cost and raise the chance of an invented answer.

  • CAdd few-shot examples showing renewal terms extracted from other vendors' contracts

    Incorrect

    Examples help when data is present but laid out differently; here the terms are in a document the model never sees.

  • DStop retrying, mark renewal_terms as missing from the source, and flag the contract for follow-up

    Correct

    A retry cannot produce information that is not in the input; the right outcome is an honest gap and a follow-up.

Retries work for format or structure errors, not for information that is absent from the source. Detect those cases, stop retrying, and surface the gap.

Anthropic documentation

Task 4.4Report a problem with this question

Question 3 · choose 1

Your contract extraction passes schema validation, but audits find a few values, such as a renewal date, that appear nowhere in the document. You want the pipeline to catch these before results go downstream. What should you add?

  1. AAn instruction to leave a field empty when Claude is unsure of its value
  2. BA stricter schema with patterns on every date and amount field
  3. CA source_quote per field, with code that checks each quote appears verbatim in the document
  4. DA higher retry limit for documents that fail validation
Show the answer and why

  • AAn instruction to leave a field empty when Claude is unsure of its value

    Incorrect

    Permission to leave a field empty reduces invented values, but it checks nothing; the ones that still occur go downstream.

  • BA stricter schema with patterns on every date and amount field

    Incorrect

    A pattern checks the format; an invented date in the right format still passes.

  • CA source_quote per field, with code that checks each quote appears verbatim in the document

    Correct

    Requiring a supporting quote grounds each value, and checking it against the document in code catches values with no basis before they go downstream.

  • DA higher retry limit for documents that fail validation

    Incorrect

    These documents pass validation, so more retries never run on them.

Schema validation catches format and structure, not values with no basis in the source. Ask for a supporting quote per field and verify in code that the quote is really in the document; a field whose quote is missing goes back for correction or to review.

Anthropic documentation

Task 4.4Report a problem with this question

Question 4 · choose 1

You changed your extraction prompt to reduce validation failures. A few spot checks look better, but you want evidence before rolling it out. What should you do?

  1. ACompare both prompts on the same labeled test set
  2. BRoll it out and watch for complaints
  3. CCount how often the new prompt is retried in production
  4. DAsk Claude whether the new prompt is better
Show the answer and why

  • ACompare both prompts on the same labeled test set

    Correct

    A fixed test set gives a like-for-like measurement of the change.

  • BRoll it out and watch for complaints

    Incorrect

    Complaints are slow and incomplete evidence.

  • CCount how often the new prompt is retried in production

    Incorrect

    Production traffic changes daily, so the comparison is muddied.

  • DAsk Claude whether the new prompt is better

    Incorrect

    A self-assessment is not a measurement.

Measure prompt changes against defined success criteria on a consistent evaluation set before relying on them.

Anthropic documentation

Task 4.4Report a problem with this question

Question 5 · choose 1

Your invoice extraction checks that line items add up to the stated total and retries failures with the error. About 6 percent still fail after every retry. Review shows most of them carry a shipping or handling charge, and your schema has no field for charges that are not line items. What should you do?

  1. AKeep retrying these invoices with the sum error appended until they pass
  2. BAllow a 10 percent difference in the total check
  3. CSend every invoice with a shipping charge to human review permanently
  4. DAdd a field for other charges to the schema and include it in the total check, then reprocess the failing invoices
Show the answer and why

  • AKeep retrying these invoices with the sum error appended until they pass

    Incorrect

    No retry can fix a missing field: the model can only pass the check by moving the charge into a line item or changing a number.

  • BAllow a 10 percent difference in the total check

    Incorrect

    A looser check hides real extraction errors along with these.

  • CSend every invoice with a shipping charge to human review permanently

    Incorrect

    This spends reviewer time on a schema gap that can be fixed once.

  • DAdd a field for other charges to the schema and include it in the total check, then reprocess the failing invoices

    Correct

    The failures come from a schema with nowhere to put the charge; once the schema can hold it, the check passes on correct extractions.

When a validation failure repeats across documents and survives retries, look for a systematic cause. Here the schema cannot represent part of the source, so the fix is to the schema and the check, not more retries.

Anthropic documentation

Task 4.4Report a problem with this question

Question 6 · choose 1

Your extraction worker counts every failed attempt toward a document's limit of three and appends the failure text to the prompt for the next attempt. During busy hours many attempts end in HTTP 529 overloaded errors, and documents land in human review without ever producing an extraction. How should the worker handle the 529s?

  1. AKeep appending "previous attempt failed: overloaded" to the prompt so Claude knows to answer faster
  2. BAs transport errors: back off and resend the same request, without spending the document's extraction attempts
  3. CLower max_tokens on the retries so that each request is cheaper to serve
  4. DRaise the limit to ten attempts so that fewer documents reach review
Show the answer and why

  • AKeep appending "previous attempt failed: overloaded" to the prompt so Claude knows to answer faster

    Incorrect

    A 529 produced no output to correct, and nothing in the prompt changes server load.

  • BAs transport errors: back off and resend the same request, without spending the document's extraction attempts

    Correct

    529 means the API is temporarily overloaded; the request never produced an extraction, so it is retried as is after a backoff, as the SDK does by default.

  • CLower max_tokens on the retries so that each request is cheaper to serve

    Incorrect

    Response length does not cause overload errors, and shorter outputs risk cutting extractions off.

  • DRaise the limit to ten attempts so that fewer documents reach review

    Incorrect

    The worker would still treat overloads as failed extractions and still edit the prompt for them.

Separate API errors from extraction failures. A 529 or 429 means no response came back: retry the same request with exponential backoff, honoring retry-after (the official SDK does this automatically), and keep extraction attempts and prompt changes for responses that were actually returned.

Anthropic documentation

Task 4.4Report a problem with this question

Question 7 · choose 1

Human reviewers correct a steady stream of extraction mistakes on one supplier's invoices, and the same mistakes keep coming back. How can the pipeline learn from the corrections?

  1. AKeep correcting by hand, since the model cannot learn
  2. BTurn typical corrections into examples for that supplier
  3. CExclude that supplier's invoices from extraction
  4. DRaise the retry limit for that supplier
Show the answer and why

  • AKeep correcting by hand, since the model cannot learn

    Incorrect

    The prompt can change even though the model does not.

  • BTurn typical corrections into examples for that supplier

    Correct

    Corrected cases make targeted examples for exactly the failures you see.

  • CExclude that supplier's invoices from extraction

    Incorrect

    Dropping documents avoids the problem without solving it.

  • DRaise the retry limit for that supplier

    Incorrect

    More retries repeat the same mistake.

Close the loop: use reviewer corrections to build examples and tests for the document types that keep failing, then measure the effect.

Anthropic documentation

Task 4.4Report a problem with this question

Practise domain 4 →Practise all domains →