Task 2.2: Filter extraneous content in source documents that degrades quality of a RAG application; Identify needed source documents that provide necessary knowledge and quality for a given RAG application

• ARaise the chunk overlap so that body text outweighs the boilerplate in every chunk

  Incorrect

  Overlap duplicates neighboring text; the repeated headers and footers stay in every chunk.

• BSwitch the index to hybrid search for exact matches

  Incorrect

  Keyword search matches repeated letterhead terms even more readily, so the noise does not go away.

• CTell the model in the system prompt to ignore letterheads, footers and page numbers

  Incorrect

  The instruction acts after retrieval; the boilerplate chunks still fill the retrieved context and push out the useful ones.

• DDrop elements whose type is page_header, page_footer or page_number

  Correct

  ai_parse_document labels each element by type, including page headers, footers and page numbers, so the boilerplate can be removed before chunking.

Removing headers, footers and other noise during parsing reduces irrelevant content in chunks. ai_parse_document returns structured elements with types such as section_header, page_header, page_footer and page_number.

• ARemove exact duplicates by hashing each file and keep one copy per hash

  Incorrect

  Small edits change the hash, so edited copies of older policies survive and keep mixing rules.

• BFilter retrieval at query time to the most recent ingestion date recorded for each document

  Incorrect

  All copies were ingested on the same day, so ingestion date cannot tell old policy versions from the current one.

• CAdd an instruction to the prompt to prefer the newest policy it sees

  Incorrect

  Chunks rarely state their policy year, so the model cannot tell versions apart, and old text still crowds the context.

• DFind near-duplicates with MinHash in Spark ML and keep the newest effective version

  Correct

  MinHash locality-sensitive hashing finds near-duplicate documents even with small edits, and the keeper can be chosen by authority or recency.

Duplicate and near-duplicate documents create redundant chunks that lower quality. Metadata can remove some duplicates; content-based near-duplicates need locality-sensitive hashing such as MinHash, with a rule for which version to keep.

• AIndex every shared drive and filter later

  Incorrect

  Indexing everything adds noise and duplicates before anyone knows what is needed.

• BPick documents by file size

  Incorrect

  File size says nothing about relevance or quality.

• CUse only public web pages

  Incorrect

  The needed knowledge is usually in internal manuals and guides.

• DInvolve domain experts from the start

  Correct

  Domain experts and stakeholders help identify relevant content and the questions users will ask.

Corpus composition depends on the application's goals. Engage domain experts early to curate content that improves quality and coverage.

• ANowhere; parse it in memory and keep only the chunks

  Incorrect

  Without a raw copy, problems cannot be traced or reprocessed.

• BIn a target table, kept as ingested

  Correct

  Storing raw source data in a target table preserves it and supports traceability and auditing.

• CInside the AI Search index

  Incorrect

  The index serves embeddings and metadata, not raw source records.

• DIn the agent's prompt cache

  Incorrect

  A prompt cache is not durable, governed storage.

Databricks recommends scalable, incremental ingestion and keeping raw source data in a target table as a best practice.

• AReview a sample of parsed output by hand

  Correct

  Regularly reviewing a sample of the parsed output reveals issues and areas to improve.

• BCount the parsed characters per file and compare with file size

  Incorrect

  A character count cannot show whether the right content was kept.

• CCheck index sync status only

  Incorrect

  A successful sync says nothing about parse quality.

• DRely on end-user complaints

  Incorrect

  Waiting for complaints finds problems late and incompletely.

Parsing best practices include cleaning noise, handling errors, customizing logic and evaluating parsing quality by manual review of samples.

• APayroll exports and HR personnel files

  Incorrect

  These are unrelated to support questions and add sensitive data.

• BRaw web server logs from the support website

  Incorrect

  Logs rarely answer customer questions on their own.

• CKnowledge base articles, FAQs and manuals

  Correct

  Databricks lists knowledge base documents, FAQs, product manuals and troubleshooting guides for a customer support bot corpus.

• DMarketing slogans and brand guidelines

  Incorrect

  Slogans hold little factual support knowledge.

The right corpus depends on the application's requirements; for support bots this usually means knowledge base documents, FAQs, product manuals and troubleshooting guides.