Browse by type
📺 Livestream: Ask the Experts: Evaluating Agent Skills
SkillEvaluator checks AI agent skills and plugins, and measures a skill's effect on task results: start with quality checks, then compare runs with and without a skill.
Agent skills are folders of instructions and supporting files defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline.
Install the CLI in a Python 3.12–3.14 environment:
pip install skillevaluator
Or use uv
uv tool install --python 3.13 skillevaluator
For uv setup or direct GitHub source installation, see Installation.
Then check skills from your terminal, CI, or agent:
After [all-tier setup with NVIDIA Build](#run-all-three-tiers-with-a-free-nvidia-build-key), run the combined pipeline:
|
GitLab details: [repository guide](https://docs.nvidia.com/skills/skillevaluator/repository-integration).
|
Then ask: *"Is my skill any good?"*
See the [agent guide](https://docs.nvidia.com/skills/skillevaluator/agent-skill).
|
Included in SkillEvaluator 1.0.1; see Installation for package availability. No model key or agent run.
skillevaluator harness-check ./my-skill
Illustrative summary of versioned profiles. Full output can include notes and schema diagnostics.
╭─ Harness compatibility · my-skill ──────────────────────────────────────────────────────╮
│ ⚠️ codex degraded Description cut to 1,021 chars + '...' in the skill list │
│ ❌ claude-code not loaded .agents/skills is not scanned │
│ ✅ opencode loads │
│ ✅ hermes loads │
│ ✅ openclaw loads │
│ ❌ copilot-cli not loaded description is 1,028 UTF-16 units, 4 over the 1,024 limit │
╰────────────────────────────────────────────────────────────────── ❌ 2 of 6 not loaded ─╯
validate in 1.0.1 includes it in Tier 1. 🧪 Ready to go deeper? Try a
live agent evaluation (Tier 3).
skillevaluator quality-check ./my-skill
No API key, scanner, GPU or Docker is needed. It prints a score out of 100 with a
grade and the findings behind it; the passing threshold defaults to 70. This is
static quality only; publication evidence stays INCOMPLETE. Security and agent impact need additional checks. See the
keyless walkthrough.
No skill yet? Clone the repository and use the bundled
src/skillevaluator/tier3/reference_skills/text-analyzer. Scores can differ by release.
Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, subject to access terms and rate limits.
pip install "skillevaluator[all]"
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
skillevaluator validate ./my-skill
Without --tiers, validate runs all three. The key covers embeddings and chat
in Tier 2 and supported agents in Tier 3; add --llm for model checks in Tier 1.
For uv or GitHub installations, use the matching all-tier install instead of the pip command above. Tier 3 also
needs Docker running; add --tiers 1,2 to skip it. More in
Run the combined pipeline, the
Build walkthrough and provider configuration.
Each tier is independent. Use the all-tier installation for all three:
| Tier | Question | Example after setup | Additional setup |
|---|---|---|---|
| Tier 1: Validation | Is the skill safe, well-formed and harness-compatible? | skillevaluator tier1 ./my-skill --no-llm |
External scanners for full static coverage; a provider and model clients enable LLM checks |
| Tier 1: Harness Check | Will the agent load the skill? | skillevaluator harness-check ./my-skill |
SkillEvaluator 1.0.1; no API key or agent run |
| Tier 2: Deduplication | Does guidance repeat or overlap? | skillevaluator tier2 ./my-skill |
Embeddings and chat; add --catalog ./skill-catalog.json for inter-skill comparison |
| Tier 3: Live Agent Evaluation | Does the skill help the agent? | skillevaluator tier3 ./my-skill |
Provider access, a supported agent, and a running sandbox (Docker by default) |
Prepare scanners for your installed release; Semgrep is optional.
--no-llm does not remove scanner requirements; a scanner that cannot run is
INCOMPLETE and exits nonzero.
SkillSpector provides specialized skill
security scanning; Harbor, the
open-source agent evaluation framework, powers sandboxed agent runs.
Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses sources or creates one skill-grounded case with limited coverage. See tier guides.
After setting up the provider, scanners, and agent sandbox, run all three tiers:
skillevaluator doctor --env-mode docker
skillevaluator validate ./my-skill
doctor checks readiness, not inference success. validate runs all tiers with
autopilot for skills. Use --tiers 1,2 to skip live evaluation or --tiers 1 for
static checks. --no-autopilot requires an existing source. Other artifact types
retain their applicable stages.
Tier 1 gates validate. Tier 2 gates unless --no-block-on-dedup is set;
Tier 3 gates only with --block-on-agent-eval. Model and managed-sandbox usage
can incur charges; see runtime and cost.
Standalone generation needs substantive evals/EVAL.md guidance, a --prompt
file with task requests and outcomes, or matching retained traces. --force
below replaces the autopilot starter; --refine refines it. Explicit
--grounding skill permits skill-only starters. Review cases; templates omit
unauthored negatives.
skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force
For an editable environment, follow the development setup.
Resources: paper, PDF, DOI, BibTeX, and NVIDIA Developer blog. Cite the paper and record the SkillEvaluator release or commit; see CITATION.cff.
See the documentation for setup, tier guides, reports, CI integration, CLI reference, and performance dashboard.
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
Read CONTRIBUTING.md and run:
make lint && make test && make build
See project governance and Code of Conduct.
Support level: Experimental. Community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement. Report bugs and feature requests through GitHub Issues; see SUPPORT.md.
Report suspected vulnerabilities using the private process in SECURITY.md. Do not disclose security issues in a public GitHub issue.
See CHANGELOG.md and GitHub Releases.
Apache License 2.0: see LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
$ claude mcp add SkillEvaluator \
-- python -m otcore.mcp_server <graph>