Free CCAR-P Evaluation, Testing & Optimization Practice Questions

The Evaluation, Testing & Optimization domain makes up 16% of the Anthropic CCAR-P exam. Practice free questions covering evals, regression testing, benchmarking, quality gates, and cost and latency optimization — each with the correct answer and a detailed explanation. The real exam mixes multiple-choice and multiple-response items; these practice questions use a single-best-answer format.

CCAR-P Evaluation, Testing & Optimization Sample Questions with Answers

Sample Question 1 — Evaluation, Testing & Optimization

A logistics company has completed a proof of concept in which Claude summarizes shipment exceptions for operations staff. Executives describe the results as promising, but operations values completeness, finance prioritizes cost, and support wants responses within 3 seconds. The team has only anecdotal user feedback and wants to begin prompt optimization. What should the architect recommend FIRST?

  1. A. Test several prompt variants with executives and deploy the variant receiving the highest preference score.
  2. B. Define task-level quality, latency, cost, and business-success measures, then establish a reproducible baseline. (Correct answer)
  3. C. Upgrade the model for difficult exceptions and compare its average response quality with the current model.
  4. D. Deploy the proof of concept to a small user group and use complaint volume as the quality metric.

Correct answer: B

Explanation: Correct answer (B): The organization must first define what successful production behavior means and measure the existing system against it. Conflicting stakeholder priorities and the absence of objective evidence make quality, latency, cost, and business-outcome criteria the governing constraints. The architectural principle is to establish an evaluation contract and reproducible baseline before optimization. In production, this makes changes comparable and prevents teams from improving one dimension while unknowingly degrading another. Why the other options are wrong: - Option A: Executive preference testing can reveal useful qualitative feedback, but it does not capture operations completeness, the 3-second requirement, cost, or reproducible task performance. - Option C: A model comparison may eventually be appropriate, but no agreed success criteria exist yet to determine whether the additional capability or cost produces a meaningful improvement. - Option D: A limited deployment can provide production evidence, but complaint volume alone is a lagging and incomplete measure that does not define acceptable quality, latency, or cost.

Sample Question 2 — Evaluation, Testing & Optimization

A payment processor uses Claude to classify 80,000 merchant disputes per day. An evaluation set sampled randomly from historical traffic reports 96% accuracy. However, regulatory-escalation cases represent only 0.4% of traffic, and production reviews show that 18% of those cases are assigned to ordinary support queues. Missing an escalation can trigger reporting violations. Which evaluation change BEST addresses the readiness gap?

  1. A. Increase the random sample until the overall confidence interval becomes narrower, while retaining accuracy as the release metric.
  2. B. Use a more capable model on every case and compare its overall accuracy against the current production baseline.
  3. C. Ask reviewers to examine only the model outputs with low self-reported confidence before each release.
  4. D. Add stratified escalation examples and difficult negatives, then enforce a separate error threshold for that class. (Correct answer)

Correct answer: D

Explanation: Correct answer (D): The rare class carries disproportionate regulatory risk and is too sparse for aggregate accuracy to establish readiness. Stratified examples, difficult negatives, and a class-specific threshold directly measure the consequential failure. The architectural principle is that evaluation coverage and release criteria should reflect error impact as well as production frequency. This matters because a system can appear strong globally while remaining unsafe for a small but critical segment. Why the other options are wrong: - Option A: A larger random sample improves statistical precision, but regulatory cases remain rare and their unacceptable error rate can still be obscured by overall accuracy. - Option B: A stronger model could improve classification, but changing models before measuring the critical segment does not establish whether the regulatory failure is resolved. - Option C: Targeted human review can be useful, but model self-reported confidence is not necessarily calibrated and may fail to identify confidently incorrect escalations.

Sample Question 3 — Evaluation, Testing & Optimization

An investment firm uses Claude to analyze loan packages. Each response contains exact borrower identifiers, calculated debt ratios, citations to source pages, and a narrative risk summary. Analysts use the result when recommending credit limits, and material errors require expert review. The team needs a scalable pre-release evaluation. Which design is BEST?

  1. A. Use exact match for every output field and reject any response whose narrative differs from the reference wording.
  2. B. Use an uncalibrated model judge for the complete response and manually inspect only outputs receiving low scores.
  3. C. Validate objective fields deterministically, score cited summaries with a rubric, and sample material decisions for expert review. (Correct answer)
  4. D. Measure analyst satisfaction after release and treat positive ratings as evidence that extraction and reasoning are correct.

Correct answer: C

Explanation: Correct answer (C): The output combines objectively verifiable fields with judgment-dependent narrative content and high-impact recommendations. Deterministic validation fits identifiers and calculations, rubric-based review fits grounded summaries and citations, and expert sampling addresses material credit judgments. The governing principle is to match evaluation methods to output type and risk. Layering methods provides stronger production evidence than forcing heterogeneous qualities into one score. Why the other options are wrong: - Option A: Exact match is appropriate for identifiers and some calculations, but it penalizes valid narrative variation and cannot adequately assess grounded risk reasoning. - Option B: A model judge can scale qualitative assessment, but relying on it without calibration or expert review is insufficient where outputs influence material financial decisions. - Option D: Analyst satisfaction captures usability, but it is a lagging measure and does not independently verify identifiers, calculations, citations, or decision quality.

Sample Question 4 — Evaluation, Testing & Optimization

A manufacturer operates a Claude-based knowledge assistant over approved maintenance manuals. Answers grounded in manuals older than one month are accurate, but manuals approved yesterday never appear in responses. The source repository contains the new files, and retrieval traces show no chunks from them. Technicians require current procedures because outdated instructions can damage equipment. What should the architect investigate FIRST?

  1. A. Inspect ingestion completion, index refresh status, metadata filters, and retrieval eligibility for the new manuals. (Correct answer)
  2. B. Strengthen the system prompt by instructing Claude to prioritize the most recently approved maintenance information.
  3. C. Move the assistant to a more capable model with stronger instruction following and a larger context window.
  4. D. Add a response-level warning whenever Claude is uncertain whether the cited maintenance procedure is current.

Correct answer: A

Explanation: Correct answer (A): The traces prove that the new manuals are absent before generation, so the first investigation belongs at the ingestion and retrieval layer. The explicit freshness requirement and missing chunks identify index refresh, filtering, or retrieval eligibility as likely causes. The architectural principle is to localize failure using pipeline evidence and fix the causal layer. A model cannot reliably use source material it never receives. Why the other options are wrong: - Option B: Prompting Claude to prefer recent information is reasonable only when current sources are available in context; it cannot compensate for missing retrieval results. - Option C: A model upgrade could improve reasoning over retrieved material, but greater capability or context cannot recover documents that were not retrieved. - Option D: A warning is a potentially useful compensating control, but it does not restore the current procedures technicians need.

Sample Question 5 — Evaluation, Testing & Optimization

A regulated insurer uses Claude to categorize complaints before writing them to a case-management system. For several privacy complaints, correlated traces show that Claude returned category code P17, the tool call also contained P17, and schema validation passed. After an integration mapping step, the stored record contains G17, a general-service category. Incorrect storage can delay mandatory reporting. Which action should the architect prioritize?

  1. A. Expand the complaint prompt with more privacy examples and rerun the complete classification benchmark.
  2. B. Add a second Claude call to verify the category immediately before the record is committed.
  3. C. Increase trace retention and alert when the model output differs from the final stored category.
  4. D. Correct the integration mapping and enforce pre-commit validation against the original approved tool value. (Correct answer)

Correct answer: D

Explanation: Correct answer (D): The evidence shows that both Claude and the tool call produced the correct value, while the category changed during integration mapping. Correcting that mapping removes the cause, and deterministic pre-commit validation prevents altered values from reaching the regulated system. The principle is to target the demonstrated failure layer and use deterministic enforcement at a consequential persistence boundary. This reduces risk more directly than changing generation or merely observing corruption. Why the other options are wrong: - Option A: Additional examples could improve genuine classification errors, but the traces show that classification was correct and the value changed later. - Option B: A verification call might detect some discrepancies, but it adds latency and probabilistic complexity while leaving the deterministic mapping defect in place. - Option C: More retention and alerting would improve detection and forensics, but the existing evidence already localizes the issue and alerts would not prevent delayed reporting.

Sample Question 6 — Evaluation, Testing & Optimization

A claims processor runs 120,000 Claude extraction requests per day. Quality meets its 98% field-accuracy target, but p95 latency is 3.8 seconds against a 3-second objective. Profiling shows that a 14,000-token instruction, policy, and schema block is identical across 92% of requests and dominates input cost and processing time. Request-specific documents average 2,000 tokens. What should the architect recommend FIRST?

  1. A. Replace the current model with the least costly model and accept any quality change that remains above 90%.
  2. B. Run two extraction calls concurrently and select the response with the greater number of populated fields.
  3. C. Separate stable from request-specific context, enable prompt reuse or caching, and remove irrelevant repeated content. (Correct answer)
  4. D. Increase the output token limit so Claude can explain every extracted field before returning the structured result.

Correct answer: C

Explanation: Correct answer (C): Profiling identifies repeated stable input as the dominant cost and latency source while current model quality already meets the target. Separating stable and volatile content, reusing eligible prompt material, and eliminating irrelevant context directly address the measured bottleneck without intentionally reducing accuracy. The principle is to optimize based on evidence and preserve proven capability. This improves cost per successful task rather than simply making each call cheaper at unknown quality. Why the other options are wrong: - Option A: Model right-sizing can reduce cost and latency, but the proposed 90% threshold violates the stated 98% quality target and ignores the measured context bottleneck. - Option B: Parallel calls may reduce neither per-request latency nor token processing and would approximately double inference cost without evidence of a quality need. - Option D: Explanations might aid debugging, but a larger output budget adds latency and cost while leaving the repeated input bottleneck untouched.

Sample Question 7 — Evaluation, Testing & Optimization

A regulated utility is piloting a Claude-based assistant that drafts responses to customer billing disputes. Support managers want engineers to revise the prompt because several demonstration responses sounded too formal. The team has no baseline dataset, and it has not defined thresholds for factual correctness, prohibited disclosures, response latency, or cost. The production service must respond within 3 seconds and must not expose another customer's account data. What should the architect recommend FIRST?

  1. A. Rewrite the system prompt using the managers' preferred tone and compare several demonstration responses.
  2. B. Upgrade the model and measure whether support managers prefer its responses during another demonstration.
  3. C. Define acceptance metrics, build a representative evaluation set, and baseline the current implementation. (Correct answer)
  4. D. Deploy the current prompt to a small production cohort and optimize from user satisfaction scores.

Correct answer: C

Explanation: Correct answer (C): The team must first define measurable success across correctness, privacy, latency, cost, and tone, then establish the current system's baseline on representative cases. The governing constraint is that any optimization must preserve the 3-second SLA and prevent cross-customer disclosure, neither of which subjective demonstrations measure. CCAR-P architecture favors defining success before optimization so later prompt or model changes can be evaluated against production-relevant release criteria. Why the other options are wrong: - Option A: An architect might refine tone because managers identified it as a visible issue, but doing so before measuring correctness, privacy, latency, and cost could optimize a secondary preference while degrading critical requirements. - Option B: A more capable model might improve some responses, but there is no evidence of a capability limitation and another subjective demonstration would not establish measurable improvement. - Option D: A production cohort can provide useful feedback, but exposing customers before privacy and correctness criteria exist creates unnecessary regulated risk and yields an incomplete satisfaction-only baseline.

Sample Question 8 — Evaluation, Testing & Optimization

A lender uses Claude to extract covenants from 80,000 loan documents per month. A random evaluation sample reports 97% field accuracy. Production incidents show only 72% accuracy for handwritten amendments and 76% for foreign-language disclosures; together these represent 3% of volume but caused most covenant-monitoring losses. Which change would BEST determine whether the system is ready for broader automation?

  1. A. Create a stratified evaluation set that oversamples both segments and applies separate risk-based acceptance thresholds. (Correct answer)
  2. B. Expand the random sample until it contains enough total documents to narrow the aggregate confidence interval.
  3. C. Weight every document equally, retain the current dataset, and require aggregate field accuracy above 98%.
  4. D. Evaluate only the two failing segments until their combined accuracy matches the current global score.

Correct answer: A

Explanation: Correct answer (A): A stratified set with deliberate coverage and segment-specific thresholds makes the rare, high-impact failures visible without discarding common production cases. The decision-driving constraint is that two low-volume segments account for most financial losses, so an aggregate score is not an adequate release gate. The architectural principle is to align evaluation coverage and thresholds with production distribution and business risk, preventing strong common-case performance from concealing unacceptable defects. Why the other options are wrong: - Option B: A larger random sample improves statistical precision, but the rare segments may still contribute too little to the aggregate metric to govern a high-risk release decision. - Option C: A higher aggregate threshold appears stricter, but equal weighting allows performance on common document types to compensate for unacceptable failures in the loss-driving segments. - Option D: Focused testing would help diagnose the two segments, but excluding the remaining production distribution could allow regressions elsewhere and would not establish overall readiness.

Keep Practicing

Take the 10-question CCAR-P quick-start test across all 7 domains, or return to the CCAR-P practice test hub for 200+ questions, exam details, and a study plan. Drill the other domains: Integration, Solution Design & Architecture, Evaluation, Testing & Optimization, Governance, Safety & Risk Management, Stakeholder Communication & Lifecycle Management, Claude Models, Prompting & Context Engineering, and Developer Productivity & Operational Enablement.