Agent skill testing
Skill Coverage
A trajectory-based test adequacy metric for reusable agent skills that asks which documented skill behavior constraints become observable in agent executions, and whether the agent satisfies or violates them.
- Source tasks
- 87
- Skill artifacts
- 202
- Unique SBCs
- 4,283
- SBC coverage
- 38.66-45.51%
- Judge agreement
- 88.58%
- Recovery rate
- 16.0%
Benchmark results
SkillsBench coverage and strengthening
Five agent-model configurations, 87 source tasks, three reruns per configuration, validated SBC coverage labels, and failed-constraint guided skill strengthening.
| Agent-model | Task success | SBC coverage | Coverage range | Failed tasks | Recovery rate | Strengthened SBC pass |
|---|
SBC coverage
The share of task-associated skill behavior constraints labeled Pass or Fail by the validated DeepSeek V4 Flash judge.
Coverage range
The minimum-to-maximum SBC coverage observed across the three repeated trajectory slots for the same 87 tasks.
Failed tasks
The diagnostic failed-task count used for coverage-guided skill strengthening in the corresponding agent-model row.
Recovery rate
The average strengthened-skill success rate over three rerun attempts, divided by the failed diagnostic task denominator.
Evidence views
Coverage, skill buckets, and task outcomes
These charts are rendered directly in the page from the current RQ2 data package. The bucket view is supplementary; the outcome view corresponds to the manuscript's task-outcome-conditioned analysis.
Per-skill coverage buckets
202 skill artifactsOutcome-conditioned coverage
successful vs unsuccessfulProject
Skill Coverage: A Test Adequacy Metric for Agent Skills
The paper formulates reusable skills as auditable sets of skill behavior constraints, validates trajectory-level coverage labels against human judgments, and shows that failed constraint labels can guide instruction-emphasis strengthening on selected failed tasks.