CCAR-P Practice Questions: Evaluation, Testing & Optimization Domain

Correct answer (C): The traces show that generation is faithful but the updated evidence never reaches context, so the first investigation belongs in the ingestion, indexing, and caching path. The two-hour freshness requirement is the governing constraint. The CCAR-P principle is to localize the observed failure before changing prompts, models, or evaluation layers. Correcting source freshness removes the cause, whereas generation changes cannot recover information that was never retrieved.

Why the other options are wrong:
- Option A: Prompting for recent information is plausible when multiple dated sources are present, but Claude cannot prioritize updated manuals that are absent from context.
- Option B: Retrieving more chunks can improve recall when relevant indexed content ranks poorly, but the evidence says updated manuals never enter the retrieval results.
- Option D: A date-based evaluator could detect some stale outputs, but it is a compensating control and does not restore the required index freshness.

Correct answer (C): Profiling identifies repeated stable input as the dominant cost and latency source while current model quality already meets the target. Separating stable and volatile content, reusing eligible prompt material, and eliminating irrelevant context directly address the measured bottleneck without intentionally reducing accuracy. The principle is to optimize based on evidence and preserve proven capability. This improves cost per successful task rather than simply making each call cheaper at unknown quality.

Why the other options are wrong:
- Option A: Model right-sizing can reduce cost and latency, but the proposed 90% threshold violates the stated 98% quality target and ignores the measured context bottleneck.
- Option B: Parallel calls may reduce neither per-request latency nor token processing and would approximately double inference cost without evidence of a quality need.
- Option D: Explanations might aid debugging, but a larger output budget adds latency and cost while leaving the repeated input bottleneck untouched.

Correct answer (A): Different properties require different evaluation methods: deterministic checks efficiently validate exact fields, a rubric evaluator scales nuanced helpfulness assessment after calibration, and experts retain oversight of high-risk regulated claims. The constraints are both the 60,000-response scale and the limited expert capacity. The architectural principle is to use a portfolio of evaluators matched to objective, subjective, and consequential failure types. This provides scalable release evidence without delegating regulatory judgment entirely to an automated evaluator.

Why the other options are wrong:
- Option B: A model evaluator can scale nuanced review, but evaluator confidence is not necessarily calibrated and does not replace deterministic validation or targeted expert oversight.
- Option C: Deterministic checks are strong for exact fields and known phrases, but they cannot reliably assess nuanced helpfulness or implied coverage guarantees.
- Option D: Expert review is appropriate for regulated judgment, but a small undifferentiated sample alone does not validate exact fields across 60,000 outputs or efficiently target high-risk cases.

Correct answer (C): The dataset must deliberately include the rare, high-impact failure classes that ordinary successful traffic underrepresents, including indirect prompt injection and conflicting identity evidence. The controlling constraint is the potential disclosure of protected health information. Risk-based evaluation requires negative, adversarial, malformed, and production-derived regression cases even when their natural frequency is low. This ensures future changes are tested against known safety failures rather than only routine success paths.

Why the other options are wrong:
- Option A: More historical traffic improves representation of routine use, but successful conversations are unlikely to supply enough adversarial or privacy-boundary failures.
- Option B: Synthetic data offers controlled coverage, but replacing all historical traffic would lose production realism and still would not ensure inclusion of the identified failures.
- Option D: Automated privacy checks may complement evaluation, but penalizing all appointment details would reject legitimate scheduling responses and would not improve dataset coverage.

Correct answer (A): The requirements have different evaluation characteristics: required disclosures are mechanically verifiable, while groundedness and tone require nuanced judgment. Scale prevents universal human review, but regulation justifies calibrated expert sampling. The architectural principle is to match each quality dimension to an appropriate evaluator and validate automated judgments against qualified humans. This provides scalable coverage without treating model evaluation as a hard compliance control.

Why the other options are wrong:
- Option B: Universal expert review would provide strong oversight, but the stated monthly volume and limited reviewer capacity make it operationally infeasible.
- Option C: Reference answers can help with constrained tasks, but many semantically correct complaint responses can differ in wording while satisfying the policy and tone requirements.
- Option D: Model-based evaluation scales well, but using it alone for exact disclosures and regulated policy judgments omits stronger deterministic checks and human calibration.

Correct answer (B): Correct tool selection and valid arguments make prompt and orchestration defects less likely, while role-specific permission denials point directly to delegated identity, scope propagation, or policy evaluation. Inspecting that boundary finds the root cause without bypassing authorization. The governing principle is to diagnose failures at the deterministic control layer and preserve least privilege rather than compensate with broader credentials or model judgment.

Why the other options are wrong:
- Option A: Examples can improve malformed or incorrectly selected tool calls, but the traces already show correct selection and valid arguments for supervisors.
- Option C: A service account could make calls succeed, but it would bypass the intended user-scoped authorization boundary and weaken least privilege.
- Option D: Model verification might assess semantic appropriateness, but it cannot correct a role-specific authorization-policy or delegated-identity defect.

Correct answer (D): A stable core preserves longitudinal comparison, while a separately versioned regression suite expands coverage to newly observed high-impact failures. Sanitizing incidents protects sensitive operational data, and calibrating the automated judge against qualified experts addresses evaluator disagreement rather than manipulating a threshold. Evaluation assets must evolve with production evidence while remaining traceable and independently validated.

Why the other options are wrong:
- Option A: Expert ratings are valuable, but replacing the full benchmark with three failures destroys representative coverage and historical comparability.
- Option B: An unchanged benchmark preserves trends, but excluding known production failures from release gates allows the same unsafe behavior to recur.
- Option C: Threshold adjustment may force these examples to fail, but it does not establish that the judge's rubric generalizes or aligns with expert judgment.

Correct answer (B): The team must first translate stakeholder concerns into measurable quality, latency, cost, and business criteria and then establish the current baseline. The explicit constraint is that stakeholders do not agree on what constitutes improvement. CCAR-P architecture practice requires defining success before optimization so that a change is not credited for improving one dimension while degrading another. This matters in production because prompt or model changes otherwise lack defensible acceptance criteria.

Why the other options are wrong:
- Option A: A model upgrade could improve some conversations, but it begins optimization before defining the required quality, latency, and cost outcomes.
- Option C: Prompt revision may eventually be appropriate, but treating competing objectives as equally important does not establish measurable priorities or a baseline.
- Option D: Tracing supports later diagnosis, but additional observations alone do not define success or show whether a proposed change is an improvement.

Correct answer (D): The rare document types create disproportionate financial and regulatory risk, so they require deliberate representation and separate quality thresholds. A production-distributed random sample can hide these failures inside a strong aggregate score. The governing principle is risk-weighted, segment-aware evaluation, which ensures low-volume but high-consequence cases can independently block an unsafe release.

Why the other options are wrong:
- Option A: A larger random sample improves statistical precision, but the rare segments can still have too few examples and remain concealed by the overall threshold.
- Option B: Recent data can improve temporal relevance, but the stated problem is inadequate coverage of rare, high-risk document types rather than dataset age.
- Option C: Production failures are valuable regression cases, but using only failures removes representative normal cases and prevents balanced measurement of overall behavior.

Correct answer (D): The traces show that revised documents never reach the model, while the manual-context test demonstrates that Claude can use them correctly. The governing evidence therefore localizes the failure to freshness or retrieval infrastructure, including ingestion, indexing, filtering, or stale cache behavior. CCAR-P diagnosis starts at the earliest layer contradicted by evidence. Fixing that layer restores current source availability instead of compensating through prompting or added model capability.

Why the other options are wrong:
- Option A: A model comparison would be reasonable if the correct sources were present but interpreted poorly; the manual test shows model reasoning is not the primary failure.
- Option B: A recency instruction can help choose between retrieved versions, but it cannot make an absent revised document appear in context.
- Option C: Chunking can cause incomplete retrieval, but the symptom is version staleness and the scenario explicitly identifies refresh and caching mechanisms that should be checked first.