NVIDIA SkillEvaluator Measures Whether Agent Skills Actually Improve Performance

NVIDIA is testing agent skills as measurable software components

NVIDIA published SkillEvaluator on August 19, 2026, an open-source evaluation layer designed to answer a practical question for agent developers: does installing a skill actually improve an agent's behavior, or does it merely add more context and tokens?

The first published benchmark snapshot covers more than 300 verified skills across more than 30 NVIDIA products, evaluated through two independent agent harnesses: OpenAI Codex and Claude Code.

The results are NVIDIA's own benchmark measurements, not an independent cross-vendor evaluation. They are still useful because the methodology explicitly compares runs with and without each skill instead of assuming that a packaged instruction set is beneficial.

The evaluator uses three tiers

NVIDIA describes a three-tier process. Tier 1 performs static checks around structure, metadata and safety. Tier 2 checks distinctiveness, using semantic comparisons to determine whether a skill is sufficiently differentiated rather than duplicating existing guidance. Tier 3 performs live evaluation. SkillEvaluator converts cases into Harbor task bundles and runs each case both with and without the skill in isolated environments. Graders then calculate a 'Skill Lift' score across multiple dimensions.

That paired design is important. Agent performance varies from run to run, and a skill can theoretically hurt a model by distracting it, triggering at the wrong time or adding unnecessary tokens. A with/without comparison at least makes the incremental effect observable.

NVIDIA reports large average gains on its specialized tasks

For the August 12 benchmark snapshot cited in NVIDIA's post, macro-averaged results across published skill-harness pairs were Correctness: 46 without the skill vs 87 with it (+41 points), Discoverability: 42 vs 82 (+40), Effectiveness: 39 vs 78 (+39), Efficiency: 43 vs 78 (+35) and Security: 97 vs 98 (+1).

NVIDIA reports an average +31 points across all dimensions, or +39 excluding Security, which already had a very high baseline.

These numbers need careful interpretation. NVIDIA explicitly says the scores are not pass-probability estimates. Discoverability and Efficiency are also scored against the skill itself, so they are better read as evidence that an installed skill was found and used appropriately than as a clean measure of unaided agent competence.

Claude Code and Codex both improved, but product domain mattered more

NVIDIA reports average Skill Lift of +34 points for Claude Code and +29 for Codex across all dimensions; excluding Security, the figures were +42 and +36.

The company says variation across individual products was much larger than the gap between harnesses, with per-product Skill Lift ranging roughly from +2 to +46. That suggests skill quality, task design and domain specificity may matter more than simply choosing a different agent harness.

The benchmark is not a head-to-head model leaderboard. A higher lift can also mean the baseline agent lacked product-specific knowledge that the skill supplied.

More context does not automatically mean fewer tokens

One of the most useful findings is that skills can improve task outcomes without necessarily reducing compute.

NVIDIA highlights two single-attempt examples. Its jetson-optimize-memory skill reduced tokens from 617,306 to 142,540 and execution time from 474.9 to 220.0 seconds in that run, while cuopt-install increased tokens from 25,227 to 55,582 and execution time from 34.0 to 41.1 seconds.

Those examples are not broad averages, but they expose an important optimization lesson: correctness lift and cost lift are separate measurements. A skill that makes an agent more capable can still be inefficient, and agent teams should measure both.

Partner pilots add a security angle

NVIDIA says OpenClaw is piloting SkillEvaluator for official organizations on ClawHub. Nous Research also tested it with Hermes Agent alongside an optional SkillSpector advisory scan that checks issues including PII, Unicode smuggling, script linting, licensing and security findings before installation.

That is relevant because agent skills can contain instructions, scripts and tool guidance with meaningful privileges. Treating skills as installable software artifacts rather than harmless prompt text creates a stronger security model: verify provenance, lint content, isolate execution and measure behavior before broad deployment.

Why this matters

Agent ecosystems are rapidly accumulating skills, plugins, MCP servers and reusable instruction packages. Without evaluation, teams can end up with a catalog that looks capable but is hard to trust.

SkillEvaluator's main contribution is not the headline lift number. It is the paired evaluation pattern: define representative tasks, run with and without the skill, isolate execution, score outcomes and track token/time costs separately.

The current benchmark is NVIDIA-centric and internally generated, so independent reproduction across more products, models and agent frameworks would strengthen the evidence. But the methodology gives agent teams a concrete starting point for turning skill quality from a subjective impression into an auditable engineering metric.