AuraOne Open Failure Gallery

Reproducible review records

Failure Gallery

Synthetic agent and robotics failures with the source, expected finding, evidence boundary, and local command required to reproduce each review path.

Approved public data: All 12 records are synthetic tutorial fixtures. They contain no customer or private lab data.

Record index

Browse failures

12 of 12 records

agent Synthetic data

Agent repeats a failing refund lookup tool

A synthetic trace loops on the same tool call after identical 404 responses.

Tool
tool-call-replay
Source
cases/agent_tool_loop/case.json
Expected behavior
retry policy missing terminal guard after repeated identical tool errors
agent Synthetic data

MCP server exposes destructive tool without confirmation

A synthetic MCP manifest declares a delete operation without guard metadata.

Tool
mcp-risk-linter
Source
cases/mcp_destructive_tool/case.json
Expected behavior
destructive capability lacks confirmation and scope boundaries
agent Synthetic data

Prompt and rubric drift changes scoring boundaries

A synthetic PR removes evidence criteria while tightening factuality language.

Tool
prompt-rubric-drift
Source
cases/prompt_rubric_drift/case.json
Expected behavior
rubric weights changed and evidence criterion was removed
agent Synthetic data

Tool schema drift breaks structured payloads

The agent emits an old argument name after a tool contract changed.

Tool
a2a-contract-test
Source
cases/agent_schema_drift/case.json
Expected behavior
tool call uses deprecated field not accepted by current tool schema
agent Synthetic data

Trace export contains email and bearer token

A synthetic GenAI trace includes sensitive values in prompt attributes.

Tool
otel-eval-bridge
Source
cases/otel_trace_privacy_leak/case.json
Expected behavior
eval case extraction redacts email and bearer token before export
agent Synthetic data

Trace lacks decision rationale for escalation

A synthetic support trace escalates without a durable decision record.

Tool
agent-trace-card
Source
cases/agent_trace_missing_decision/case.json
Expected behavior
trace card missing decision rationale and escalation evidence
robotics Synthetic data

Camera shift causes VLA failure cluster

A synthetic episode set shows brittleness when camera pose metadata changes.

Tool
vla-robustness-kit
Source
cases/robot_camera_shift_brittle/case.json
Expected behavior
camera_shift appears as a repeated failure cluster
robotics Synthetic data

Dataset card omits coordinate frames

A synthetic robot dataset describes cameras and actions but omits frame definitions.

Tool
embodiment-card
Source
cases/robot_missing_coordinate_frame/case.json
Expected behavior
schema validation catches missing coordinate_frames
robotics Synthetic data

Low-quality robot episode passes casual inspection

A synthetic dataset has missing actions, short episodes, and inconsistent metadata.

Tool
lerobot-quality-gates
Source
cases/robot_low_quality_episode/case.json
Expected behavior
quality gates produce multiple findings before training
robotics Synthetic data

Recovery segment lacks intervention label

A synthetic recovery segment omits the intervention type and recovery result.

Tool
robot-recovery-bench
Source
cases/robot_recovery_missing_label/case.json
Expected behavior
segment is not training-ready because recovery metadata is incomplete
robotics Synthetic data

Robot episode has sensor timestamp desync

A synthetic LeRobot-style episode has camera frames lagging proprioception.

Tool
lerobot-quality-gates
Source
cases/robot_sensor_desync/case.json
Expected behavior
episode fails timestamp alignment gate
robotics Synthetic data

VLA policy fails instruction paraphrases

A synthetic VLA policy passes the original command but fails equivalent polite wording.

Tool
vla-robustness-kit
Source
cases/vla_instruction_paraphrase_failure/case.json
Expected behavior
language perturbation family has low pass rate

Evidence contract

Read the record before running the command.

  1. InspectConfirm the synthetic source and stated failure.
  2. ReproduceRun the exact local command in the linked tool.
  3. CompareCheck observed evidence against the expected behavior and limitations.

agent failure record

Agent repeats a failing refund lookup tool

Failure

A synthetic trace loops on the same tool call after identical 404 responses.

Domain

agent

Related tool

tool-call-replay

Review label

tool-call-loop

Source

cases/agent_tool_loop/case.json

Expected behavior

retry policy missing terminal guard after repeated identical tool errors

Evidence

The fixture is designed to produce the tool-call-loop finding when reviewed with tool-call-replay. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

tool-call-replay run examples/failed_refund_trace.json --scenario examples/refund_scenario.yaml

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

agent failure record

MCP server exposes destructive tool without confirmation

Failure

A synthetic MCP manifest declares a delete operation without guard metadata.

Domain

agent

Related tool

mcp-risk-linter

Review label

mcp-risk

Source

cases/mcp_destructive_tool/case.json

Expected behavior

destructive capability lacks confirmation and scope boundaries

Evidence

The fixture is designed to produce the mcp-risk finding when reviewed with mcp-risk-linter. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

mcp-risk-linter lint examples/risky_server.json --out report.md

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

agent failure record

Prompt and rubric drift changes scoring boundaries

Failure

A synthetic PR removes evidence criteria while tightening factuality language.

Domain

agent

Related tool

prompt-rubric-drift

Review label

rubric-drift

Source

cases/prompt_rubric_drift/case.json

Expected behavior

rubric weights changed and evidence criterion was removed

Evidence

The fixture is designed to produce the rubric-drift finding when reviewed with prompt-rubric-drift. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

prompt-rubric-drift compare examples/before examples/after --out report.md

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

agent failure record

Tool schema drift breaks structured payloads

Failure

The agent emits an old argument name after a tool contract changed.

Domain

agent

Related tool

a2a-contract-test

Review label

schema-drift

Source

cases/agent_schema_drift/case.json

Expected behavior

tool call uses deprecated field not accepted by current tool schema

Evidence

The fixture is designed to produce the schema-drift finding when reviewed with a2a-contract-test. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

a2a-contract-test run --agent-card examples/failing_agent/agent-card.json

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

agent failure record

Trace export contains email and bearer token

Failure

A synthetic GenAI trace includes sensitive values in prompt attributes.

Domain

agent

Related tool

otel-eval-bridge

Review label

trace-redaction

Source

cases/otel_trace_privacy_leak/case.json

Expected behavior

eval case extraction redacts email and bearer token before export

Evidence

The fixture is designed to produce the trace-redaction finding when reviewed with otel-eval-bridge. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

otel-eval-bridge extract examples/genai_trace.json --out eval_cases.jsonl

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

agent failure record

Trace lacks decision rationale for escalation

Failure

A synthetic support trace escalates without a durable decision record.

Domain

agent

Related tool

agent-trace-card

Review label

trace-card-gap

Source

cases/agent_trace_missing_decision/case.json

Expected behavior

trace card missing decision rationale and escalation evidence

Evidence

The fixture is designed to produce the trace-card-gap finding when reviewed with agent-trace-card. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

agent-trace-card generate examples/support_trace.json --out card.md

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

robotics failure record

Camera shift causes VLA failure cluster

Failure

A synthetic episode set shows brittleness when camera pose metadata changes.

Domain

robotics

Related tool

vla-robustness-kit

Review label

vision-robustness

Source

cases/robot_camera_shift_brittle/case.json

Expected behavior

camera_shift appears as a repeated failure cluster

Evidence

The fixture is designed to produce the vision-robustness finding when reviewed with vla-robustness-kit. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

vla-robustness-kit run examples/mock_episode_set --policy mock --out report.md

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

robotics failure record

Dataset card omits coordinate frames

Failure

A synthetic robot dataset describes cameras and actions but omits frame definitions.

Domain

robotics

Related tool

embodiment-card

Review label

embodiment-metadata-gap

Source

cases/robot_missing_coordinate_frame/case.json

Expected behavior

schema validation catches missing coordinate_frames

Evidence

The fixture is designed to produce the embodiment-metadata-gap finding when reviewed with embodiment-card. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

embodiment-card validate examples/mock_so101_card.json

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

robotics failure record

Low-quality robot episode passes casual inspection

Failure

A synthetic dataset has missing actions, short episodes, and inconsistent metadata.

Domain

robotics

Related tool

lerobot-quality-gates

Review label

dataset-quality

Source

cases/robot_low_quality_episode/case.json

Expected behavior

quality gates produce multiple findings before training

Evidence

The fixture is designed to produce the dataset-quality finding when reviewed with lerobot-quality-gates. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

lerobot-quality-gates check examples/bad_dataset --out report.md

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

robotics failure record

Recovery segment lacks intervention label

Failure

A synthetic recovery segment omits the intervention type and recovery result.

Domain

robotics

Related tool

robot-recovery-bench

Review label

recovery-label-gap

Source

cases/robot_recovery_missing_label/case.json

Expected behavior

segment is not training-ready because recovery metadata is incomplete

Evidence

The fixture is designed to produce the recovery-label-gap finding when reviewed with robot-recovery-bench. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

robot-recovery-bench validate examples/mock_invalid_segments.jsonl

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

robotics failure record

Robot episode has sensor timestamp desync

Failure

A synthetic LeRobot-style episode has camera frames lagging proprioception.

Domain

robotics

Related tool

lerobot-quality-gates

Review label

sensor-desync

Source

cases/robot_sensor_desync/case.json

Expected behavior

episode fails timestamp alignment gate

Evidence

The fixture is designed to produce the sensor-desync finding when reviewed with lerobot-quality-gates. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

lerobot-quality-gates check examples/bad_dataset --out report.md

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.

robotics failure record

VLA policy fails instruction paraphrases

Failure

A synthetic VLA policy passes the original command but fails equivalent polite wording.

Domain

robotics

Related tool

vla-robustness-kit

Review label

language-robustness

Source

cases/vla_instruction_paraphrase_failure/case.json

Expected behavior

language perturbation family has low pass rate

Evidence

The fixture is designed to produce the language-robustness finding when reviewed with vla-robustness-kit. Compare the tool output with the expected behavior above; no production result is asserted by this gallery.

Reproduce locally

vla-robustness-kit run examples/mock_episode_set --policy mock --out report.md

Limitations

Synthetic tutorial evidence only. This record does not represent a customer incident, production benchmark, or validated model comparison.