CCDV-F Practice Questions: Eval, Testing, and Debugging Domain

Correct answer (B): The failure occurs before Claude returns any model response, and the error explicitly identifies an invalid request payload. A 4xx-style invalid request should be debugged at the client request construction, authentication, parameters, or payload-format boundary before changing prompts, models, or retry policies.

Why the other options are wrong:
- Option A: Changing models may seem useful when an application fails, but the log shows the request is rejected before model output exists.
- Option C: Prompt instructions cannot fix a malformed API payload that is rejected before Claude processes the message.
- Option D: Backoff helps with transient errors, but this deterministic 400 error will continue until the request format is corrected.

Correct answer (A): The regression follows a schema change that added a required field, while the prompt-output contract was not updated to elicit that field. The correct first fix is to align the prompt, schema, validator behavior, and retry or feedback handling so the required escalation_reason is produced and validated rather than weakening the contract or blaming the model.

Why the other options are wrong:
- Option B: Making the field optional avoids the failure but contradicts the new required behavior for escalation cases.
- Option C: Storing invalid JSON removes a guardrail and risks downstream defects instead of fixing the contract.
- Option D: The evidence points to a changed schema without a matching prompt contract, not a model capability issue.

Correct answer (B): The raw output violates the declared schema in a repeatable way, so the fix should preserve validation and repair the contract between the prompt, expected schema, parser, and downstream code. Blind retries or schema loosening would mask the defect rather than ensuring the application receives the required field in the required type.

Why the other options are wrong:
- Option A: Disabling validation may appear to improve throughput, but it hides a contract failure and risks incorrect downstream writes.
- Option C: Retries can help transient failures, but the artifact shows the same deterministic validation failure each time.
- Option D: Loosening the schema may be tempting, but the application explicitly requires total_cents as an integer field.

Correct answer (B): A rate-limit response is a transient operational failure, and the operation is a generation-only task with no external side effect. The best recovery is a bounded retry policy with exponential backoff and jitter. This avoids hammering the service while still allowing recovery from temporary capacity limits.

Why the other options are wrong:
- Option A: Some errors are permanent, but the scenario identifies a rate-limit response, which is commonly handled as transient with bounded backoff.
- Option C: Prompt wording does not address a rate-limit status because the failure is operational rather than model-output quality.
- Option D: Marking the document complete loses work and hides the failure instead of applying an appropriate transient-error policy.

Correct answer (A): Production confidence for Claude applications requires representative evals with expected behavior or grading criteria, especially when risks include varied and difficult cases. Unit tests can verify integration behavior, but model behavior should be covered with regression-oriented eval cases rather than a single manual happy path.

Why the other options are wrong:
- Option B: A single real claim is not representative enough to catch regressions across the stated risk areas.
- Option C: Removing unit tests loses coverage of integration code; evals should complement, not replace, unit tests.
- Option D: Repeating one happy-path claim does not cover the known risk cases or define expected behavior criteria.

Correct answer (D): The timeout happened after a side-effecting payment request may have been sent. Retrying the whole workflow without an idempotency key or status check can duplicate the charge. The correct mitigation is to make recovery idempotency-aware before retrying an ambiguous external action.

Why the other options are wrong:
- Option A: Immediate full-workflow retry may look like a simple recovery, but it risks duplicate external side effects.
- Option B: The prompt cannot determine whether the payment API already processed the request after the timeout.
- Option C: Reducing model cost does not address the correctness risk of duplicating a payment action.

Correct answer (C): The raw model output still matches the old contract, and the prompt has not changed. The failure appeared after a backend refactor that introduced a validator expecting `label` instead of `category`. The best conclusion is an application contract mismatch, not a sudden model-quality or transport problem.

Why the other options are wrong:
- Option A: A larger model might be considered for quality issues, but the evidence points to a schema contract mismatch introduced by the refactor.
- Option B: The prompt is unchanged, and the model output matches the old expected shape, so prompt ambiguity is not the first conclusion.
- Option D: The raw content is well-formed and available to the validator, so malformed transport bytes are not indicated.

Correct answer (B): Useful Claude debugging logs should capture the boundary and context needed for triage while avoiding unnecessary sensitive data exposure. Request ID, model version, prompt version, status, sanitized input or output excerpts, and validation details help determine whether the failure occurred in request handling, model output, or post-processing.

Why the other options are wrong:
- Option A: A user ID may help customer support, but it does not provide enough technical context to diagnose the failure boundary.
- Option C: Full prompts and responses can help debugging, but the scenario explicitly requires avoiding full sensitive prompt storage.
- Option D: The final browser output omits request metadata, raw response clues, and validation details needed for root-cause analysis.

Correct answer (A): The artifact describes a transient timeout with validated requests and prior success for the same payload. For timeout or infrastructure-related failures, bounded retry with backoff and logging is appropriate, unlike deterministic validation failures that require a code or contract fix.

Why the other options are wrong:
- Option B: Validation is not causing the failures, and disabling it would reduce confidence in future requests.
- Option C: Prompt changes are speculative because the evidence points to a network timeout rather than prompt behavior.
- Option D: Dropping failed work hides incomplete processing and does not recover the affected documents.

Correct answer (C): The trace shows that the Claude API request succeeded and the model produced a tool request. The application then dispatched the tool, and the first failure signal is the carrier API HTTP 500. Therefore the failing boundary is external tool execution, not model generation or validation.

Why the other options are wrong:
- Option A: Request construction succeeded because the API returned 200 and a valid tool_use event was produced.
- Option B: The model successfully selected a tool and supplied arguments, so the first failure is not before tool selection.
- Option D: No final assistant continuation occurred, so final response validation is downstream of the observed failure.