MCPcopy Create free account
hub / github.com/NVIDIA/SkillEvaluator

github.com/NVIDIA/SkillEvaluator @v1.0.1

Chat with this repo
repository ↗ · DeepWiki ↗ · release v1.0.1 ↗ · + Follow
26,736 symbols 120,612 edges 874 files 6,932 documented · 26%

Browse by type

⨍ Functions 24,741◇ Types & classes 1,292↳ Endpoints 703
What it actually does AI analysis from the code graph
loading…
README

SkillEvaluator

Evaluate and improve AI agent skills and plugins: quality, security, overlap, compatibility, and real impact across agents

License Python PyPI Documentation Paper Blog Stars Forks

📺 Livestream: Ask the Experts: Evaluating Agent Skills

SkillEvaluator checks AI agent skills and plugins, and measures a skill's effect on task results: start with quality checks, then compare runs with and without a skill.

Agent skills are folders of instructions and supporting files defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline.

Get started

Install the CLI in a Python 3.12–3.14 environment:

pip install skillevaluator

Or use uv

uv tool install --python 3.13 skillevaluator

For uv setup or direct GitHub source installation, see Installation.

Then check skills from your terminal, CI, or agent:

Run the CLI directly Quickly check your skill without an LLM key:
skillevaluator quality-check ./my-skill
After [all-tier setup with NVIDIA Build](#run-all-three-tiers-with-a-free-nvidia-build-key), run the combined pipeline:
skillevaluator validate \
  ./my-skill
Add it to your repo via CI Our CI template checks every skill change, no code to maintain. Run in your GitHub repo to open a setup PR:
skillevaluator ci init
GitLab details: [repository guide](https://docs.nvidia.com/skills/skillevaluator/repository-integration).
Wire it to your agent Add the skill to Claude Code, Codex or another agent:
npx skills add \
  NVIDIA/SkillEvaluator
Then ask: *"Is my skill any good?"* See the [agent guide](https://docs.nvidia.com/skills/skillevaluator/agent-skill).

⚡ Harness check: will your agent load your skill?

Included in SkillEvaluator 1.0.1; see Installation for package availability. No model key or agent run.

skillevaluator harness-check ./my-skill

Illustrative summary of versioned profiles. Full output can include notes and schema diagnostics.

╭─ Harness compatibility · my-skill ──────────────────────────────────────────────────────╮
│  ⚠️ codex        degraded    Description cut to 1,021 chars + '...' in the skill list   │
│  ❌ claude-code  not loaded  .agents/skills is not scanned                              │
│  ✅ opencode     loads                                                                  │
│  ✅ hermes       loads                                                                  │
│  ✅ openclaw     loads                                                                  │
│  ❌ copilot-cli  not loaded  description is 1,028 UTF-16 units, 4 over the 1,024 limit  │
╰────────────────────────────────────────────────────────────────── ❌ 2 of 6 not loaded ─╯

validate in 1.0.1 includes it in Tier 1. 🧪 Ready to go deeper? Try a live agent evaluation (Tier 3).

Quickstart

Check your skill without a key

skillevaluator quality-check ./my-skill

No API key, scanner, GPU or Docker is needed. It prints a score out of 100 with a grade and the findings behind it; the passing threshold defaults to 70. This is static quality only; publication evidence stays INCOMPLETE. Security and agent impact need additional checks. See the keyless walkthrough.

No skill yet? Clone the repository and use the bundled src/skillevaluator/tier3/reference_skills/text-analyzer. Scores can differ by release.

Run all three tiers with a free NVIDIA Build key

Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, subject to access terms and rate limits.

  1. Sign in at build.nvidia.com and complete enrollment.
  2. Open a model page, select Get API Key, and generate your key.
  3. Complete scanner setup, start Docker, and prepare a reviewed dataset. Install every tier, set the provider and key, then validate:
pip install "skillevaluator[all]"
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
skillevaluator validate ./my-skill

Without --tiers, validate runs all three. The key covers embeddings and chat in Tier 2 and supported agents in Tier 3; add --llm for model checks in Tier 1. For uv or GitHub installations, use the matching all-tier install instead of the pip command above. Tier 3 also needs Docker running; add --tiers 1,2 to skip it. More in Run the combined pipeline, the Build walkthrough and provider configuration.

Three-tier overview

Three-tier pipeline: skill, validation, deduplication, live evaluation, reports

Each tier is independent. Use the all-tier installation for all three:

Tier Question Example after setup Additional setup
Tier 1: Validation Is the skill safe, well-formed and harness-compatible? skillevaluator tier1 ./my-skill --no-llm External scanners for full static coverage; a provider and model clients enable LLM checks
Tier 1: Harness Check Will the agent load the skill? skillevaluator harness-check ./my-skill SkillEvaluator 1.0.1; no API key or agent run
Tier 2: Deduplication Does guidance repeat or overlap? skillevaluator tier2 ./my-skill Embeddings and chat; add --catalog ./skill-catalog.json for inter-skill comparison
Tier 3: Live Agent Evaluation Does the skill help the agent? skillevaluator tier3 ./my-skill Provider access, a supported agent, and a running sandbox (Docker by default)

Prepare scanners for your installed release; Semgrep is optional. --no-llm does not remove scanner requirements; a scanner that cannot run is INCOMPLETE and exits nonzero. SkillSpector provides specialized skill security scanning; Harbor, the open-source agent evaluation framework, powers sandboxed agent runs.

Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses sources or creates one skill-grounded case with limited coverage. See tier guides.

Run the combined pipeline

After setting up the provider, scanners, and agent sandbox, run all three tiers:

skillevaluator doctor --env-mode docker
skillevaluator validate ./my-skill

doctor checks readiness, not inference success. validate runs all tiers with autopilot for skills. Use --tiers 1,2 to skip live evaluation or --tiers 1 for static checks. --no-autopilot requires an existing source. Other artifact types retain their applicable stages.

Tier 1 gates validate. Tier 2 gates unless --no-block-on-dedup is set; Tier 3 gates only with --block-on-agent-eval. Model and managed-sandbox usage can incur charges; see runtime and cost.

Standalone generation needs substantive evals/EVAL.md guidance, a --prompt file with task requests and outcomes, or matching retained traces. --force below replaces the autopilot starter; --refine refines it. Explicit --grounding skill permits skill-only starters. Review cases; templates omit unauthored negatives.

skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force

Run from a source checkout

For an editable environment, follow the development setup.

Research and citation

Resources: paper, PDF, DOI, BibTeX, and NVIDIA Developer blog. Cite the paper and record the SkillEvaluator release or commit; see CITATION.cff.

Documentation

See the documentation for setup, tier guides, reports, CI integration, CLI reference, and performance dashboard.

Installation and third-party software

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Contributing

Read CONTRIBUTING.md and run:

make lint && make test && make build

See project governance and Code of Conduct.

Support

Support level: Experimental. Community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement. Report bugs and feature requests through GitHub Issues; see SUPPORT.md.

Security

Report suspected vulnerabilities using the private process in SECURITY.md. Do not disclose security issues in a public GitHub issue.

Releases

See CHANGELOG.md and GitHub Releases.

License

Apache License 2.0: see LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Core symbols most depended-on inside this repo

browse all functions →

Shape

Function 19,112
Method 5,629
Class 1,292
Route 703

Languages

Python96%
TypeScript4%

Modules by API surface

src/skillevaluator/reporting/assets/chart.umd.min.js958 symbols
tests/validators/test_security.py628 symbols
tests/test_harbor_secure_docker_environment.py354 symbols
tests/test_harbor_local_mode.py306 symbols
tests/test_tier3_progress.py297 symbols
tests/test_harbor_runtime_skill_isolation.py241 symbols
src/skillevaluator/tier3/harbor/collector.py235 symbols
src/skillevaluator/tier3/eval_core/plugin_signals.py232 symbols
src/skillevaluator/tier3/eval_core/checks.py224 symbols
src/skillevaluator/tier3/harbor/adapter.py214 symbols
tests/reporting/test_reporters.py181 symbols
tests/validators/test_quality_score.py180 symbols

Dependencies from manifests, versioned

fern-api5.121.0 · 1×
bandit1.9.4 · 1×
jinja23.1 · 1×
packaging24 · 1×
pydantic2.11.0 · 1×
pyyaml6.0 · 1×
regex2024.11.6 · 1×
rich13.5.0 · 1×
uv0.11.27 · 1×

Datastores touched

appDatabase · 1 repos
(mysql)Database · 1 repos
appDatabase · 1 repos
dbDatabase · 1 repos

For agents

$ claude mcp add SkillEvaluator \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact

Ask about this repo answers extend the page