skill-validator

A CLI tool that validates and scores Agent Skill packages.

Spec compliance is table stakes. skill-validator goes further: it checks that links actually resolve, flags files that shouldn't be in a skill directory, reports token counts so you can see how much of an agent's context window your skill will consume, analyzes content quality metrics, detects cross-language contamination, and offers LLM-as-judge scoring to evaluate skill quality across dimensions like clarity, actionability, and novelty. A spec-compliant skill that has broken links or a 60k-token reference file will technically pass the spec but perform poorly in practice.

Table of Contents

• Install
• CLI
  • Homebrew
  • Using Go
  • Pre-commit hook
• As a library
• Command Usage
• validate structure
• validate links
• analyze content
• analyze contamination
• check
• score evaluate
• score report
• Output Formats
• JSON output
• Markdown output
• GitHub Actions annotations
• CI Integration
• CI workflow example
• Multi-skill directories
• Examples
• What it checks & why
• Structure validation
  • Flat skill layouts
  • Allowing non-standard directories
• Link validation
• Content analysis
• Contamination analysis
• LLM scoring
• Stability
• Development

Install

Install CLI

You can install the CLI in three ways:

• Homebrew
• Using Go
• Pre-commit hook

Homebrew

brew tap agent-ecosystem/tap
brew install skill-validator

Using Go

go install github.com/agent-ecosystem/skill-validator/cmd/skill-validator@latest

Or build from source:

git clone https://github.com/agent-ecosystem/skill-validator.git
cd skill-validator
go build -o skill-validator ./cmd/skill-validator

Pre-commit hook

skill-validator supports pre-commit. Platform-specific hooks are provided for all major agent platforms, so the correct skills directory is used automatically. For example, the following configuration runs the skill-validator check command on the ".claude/skills/" path:

repos:
  - repo: https://github.com/agent-ecosystem/skill-validator
    rev: v0.5.0
    hooks:
      - id: skill-validator-claude

Available platform hooks: skill-validator-amp, skill-validator-cline, skill-validator-claude, skill-validator-codex, skill-validator-copilot, skill-validator-cursor, skill-validator-gemini, skill-validator-goose, skill-validator-kiro, skill-validator-mistral-vibe, skill-validator-roo-code, skill-validator-trae, skill-validator-windsurf.

A generic skill-validator hook is also available if you want to specify a custom command override and/or custom path — supply the command and path via args:

hooks:
  - id: skill-validator
    args: ["check", "path/to/skills/"]

As a library

The validation and scoring packages are importable for use in custom tooling, CI pipelines, and enterprise integrations:

import (
    "github.com/agent-ecosystem/skill-validator/orchestrate"
    "github.com/agent-ecosystem/skill-validator/judge"
    "github.com/agent-ecosystem/skill-validator/evaluate"
)

API documentation and runnable examples are on pkg.go.dev.

Custom LLM providers

The built-in clients cover Anthropic, OpenAI-compatible APIs, and the Claude CLI. For other providers, implement the judge.LLMClient interface:

type LLMClient interface {
    Complete(ctx context.Context, systemPrompt, userContent string) (string, error)
    Provider() string
    ModelName() string
}

Your Complete method receives the scoring rubric as the system prompt and the skill/reference content as user content. Return the raw LLM response text; the judge package handles JSON parsing. Provider() and ModelName() are used for cache key generation, so they should return stable, unique values.

For OpenAI-compatible providers (Azure OpenAI, etc.), you can use the built-in client with a custom base URL:

client, err := judge.NewClient(judge.ClientOptions{
    Provider: "openai",
    APIKey:   os.Getenv("AZURE_OPENAI_API_KEY"),
    BaseURL:  "https://your-resource.openai.azure.com/openai/deployments/your-deployment",
    Model:    "gpt-5.2",
})

For providers that need different request formats (AWS Bedrock, Google Vertex AI, local models), implement LLMClient directly and pass it to the scoring functions:

scores, err := judge.ScoreSkill(ctx, skillContent, myClient, judge.DefaultMaxContentLen)

// Or use the evaluate package for full orchestration with caching
result, err := evaluate.EvaluateSkill(ctx, "./my-skill", myClient, evaluate.Options{
    MaxLen: judge.DefaultMaxContentLen,
})

Command Usage

Commands map to skill development lifecycle stages:

Use --version to print the installed version.

Exit codes:

Use --strict on check or validate structure to treat warnings as errors (exit 1 instead of 2). This is useful in CI pipelines where you want a binary pass/fail:

skill-validator check --strict <path>
skill-validator validate structure --strict <path>

For more details about how the commands are implemented and what they provide, refer to What it Checks.

validate structure

skill-validator validate structure <path>
skill-validator validate structure --skip-orphans <path>
skill-validator validate structure --strict <path>
skill-validator validate structure --allow-extra-frontmatter <path>
skill-validator validate structure --allow-flat-layouts <path>
skill-validator validate structure --allow-dirs=evals,testing <path>

Checks spec compliance: directory structure, frontmatter fields, token limits, skill ratio, code fence integrity, internal link validity, and orphan file detection.

Validating skill: my-skill/

Structure
  ✓ SKILL.md found

Frontmatter
  ✓ name: "my-skill" (valid)
  ✓ description: (54 chars)
  ✓ license: "MIT"

Markdown
  ✓ no unclosed code fences found

Tokens
  SKILL.md body:        1,250 tokens
  references/guide.md:    820 tokens
  ─────────────────────────────────────
  Total:                2,070 tokens

Result: passed

validate links

skill-validator validate links <path>

Validates external (HTTP/HTTPS) links in SKILL.md. Internal (relative) links are checked by validate structure.

analyze content

skill-validator analyze content <path>
skill-validator analyze content --per-file <path>

Computes content quality metrics for SKILL.md and reference markdown files:

Content Analysis
  Word count:               1,250
  Code block ratio:         0.32
  Imperative ratio:         0.45
  Information density:      0.39
  Instruction specificity:  0.78
  Sections: 6  |  List items: 23  |  Code blocks: 8

References Content Analysis
  Word count:               820
  ...

References Contamination Analysis
  Contamination level: low (score: 0.00)
  Scope breadth: 0

Metrics include word count, code block count/ratio, code languages, sentence count, imperative sentence ratio, information density, strong/weak language markers, instruction specificity, section count, and list item count. Reference files in references/ are analyzed in aggregate. Use --per-file to see a breakdown by individual reference file.

analyze contamination

skill-validator analyze contamination <path>
skill-validator analyze contamination --per-file <path>

Detects cross-language contamination — skills where code examples in one language could cause incorrect code generation in another context. Analyzes both SKILL.md and reference markdown files:

Contamination Analysis
  Contamination level: medium (score: 0.35)
  Primary language category: javascript
  ⚠ Language mismatch: python (1 category differ from primary)
  ℹ Multi-interface tool detected: mongodb
  Scope breadth: 4

References Contamination Analysis
  Contamination level: low (score: 0.00)
  Scope breadth: 0

Contamination scoring considers three factors: multi-interface tools (0.3 weight), application language mismatch across code blocks (0.4 weight), and scope breadth (0.3 weight). Auxiliary languages (shell, config formats, query languages, markup) are excluded from the mismatch calculation since they don't cause syntactic confusion with application languages. Reference files in references/ are analyzed in aggregate. Use --per-file to see a breakdown by individual reference file.

check

skill-validator check <path>
skill-validator check --only structure,links <path>
skill-validator check --skip contamination <path>
skill-validator check --per-file <path>
skill-validator check --skip-orphans <path>
skill-validator check --strict <path>
skill-validator check --allow-extra-frontmatter <path>
skill-validator check --allow-flat-layouts <path>
skill-validator check --allow-dirs=evals,testing <path>

Runs all checks (structure + links + content + contamination).

Valid check groups: structure, links, content, contamination.

score evaluate

Uses an LLM-as-judge approach to score skill quality across multiple dimensions. This is based on findings from the agent-skill-analysis research project, which identified novelty as a key predictor of skill value — skills that provide genuinely novel information are more likely to improve LLM outputs, while skills that restate common knowledge can potentially degrade performance.

export ANTHROPIC_API_KEY=your-key-here
skill-validator score evaluate <path>
skill-validator score evaluate --skill-only <path>
skill-validator score evaluate --refs-only <path>
skill-validator score evaluate --display files <path>
skill-validator score evaluate path/to/references/api-guide.md

# Or use the Claude CLI (no API key needed if already authenticated)
skill-validator score evaluate --provider claude-cli <path>

Provider support: Use --provider to select the backend: