MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

↓ 14 callersFunction_source
(root: Path, config: str = "")
tests/tier3/test_interactions.py:32
↓ 14 callersFunction_stub_provider
Answer the SDK's async httpx requests in order: a response, an error status, "cut-off", or a 200 JSON body.
tests/tier3/test_ragas_judge_evidence.py:217
↓ 14 callersFunction_tier3_page
(html: str, page: str, next_page: str)
tests/reporting/test_unified_tier3_report.py:153
↓ 14 callersMethod_tok
(self, index: int)
src/skillevaluator/evaluation/dataset_case_removal.py:267
↓ 14 callersFunction_traj
(*steps: dict[str, Any], agent: str = "claude-code")
tests/tier3/test_f90_mcp_execution.py:47
↓ 14 callersFunction_type_name
JSON type name of a parsed value, for messages (``null``, ``array``, ``object``, ...).
src/skillevaluator/plugin_format_schema.py:68
↓ 14 callersFunction_validate_compose_service_name
(service: str)
src/skillevaluator/tier3/harbor/secure_docker_environment.py:553
↓ 14 callersFunction_write_complete_job_result
(job_dir: Path, trial_names: list[str])
tests/test_harbor_collector_runtime_failures.py:604
↓ 14 callersFunction_write_described_skill
(tmp_path: Path, description: str)
tests/validators/test_quality_score.py:1784
↓ 14 callersFunction_write_skills
(write_skill, root: Path, *names: str)
tests/embedding/test_catalog.py:653
↓ 14 callersFunction_write_task_toml
( task_dir: Path, entry: dict[str, Any], has_skill: bool, mcp_servers: list[dict[str, Any]] |
src/skillevaluator/tier3/harbor/adapter.py:2212
↓ 14 callersFunctionaccept_traces
(skill: Path, review_dir: Path, *, dry_run: bool = False, allow_native_harbor: bool = False)
src/skillevaluator/tier3/trace_import.py:437
↓ 14 callersMethodadmit_many
Reserve a logical unit, waiting for measured in-flight work when necessary.
src/skillevaluator/tier3/harbor/run_budget.py:168
↓ 14 callersMethodanalyze
(_client, cluster)
tests/deduplication/intra_skill/test_intra_skill_validator.py:1516
↓ 14 callersFunctionbuild_native_task_staging
Build a bundle after checking native component and MCP policy.
src/skillevaluator/tier3/harbor/native_staging.py:143
↓ 14 callersFunctioncheck_activation
Check whether the agent activated the expected skill. Supports multiple activation patterns: 1. Claude Code ``Skill`` tool (via *skill_tool
src/skillevaluator/tier3/eval_core/checks.py:7002
↓ 14 callersMethodclose
(self)
src/skillevaluator/model_catalog.py:136
↓ 14 callersMethodembed
Generate embeddings for a batch of texts. Args: texts: Strings to embed. Empty list returns empty list. Returns:
src/skillevaluator/embedding/client.py:180
↓ 14 callersMethodemit
(self, event: ProgressEvent)
src/skillevaluator/tier3/harbor/progress.py:472
↓ 14 callersFunctioni
(i)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 14 callersFunctioninventory
(root: Path, **kwargs)
tests/test_plugin_inventory.py:29
↓ 14 callersFunctioninvoke
(api, directory)
tests/ci/test_comment_report.py:91
↓ 14 callersFunctionllm_failure_diagnostic
Explain SDK failures without importing optional SDKs or formatting exceptions.
src/skillevaluator/inference/diagnostics.py:58
↓ 14 callersFunctionload_catalog
(modules, tmp_path, *, client=None, version=2, skills=(), plugins=(), **kwargs)
tests/deduplication/plugin/test_private_dedup.py:141
↓ 14 callersFunctionload_plugin_profile
Discover fresh effective members and project only bounded comparison fields. Calling again after provider work intentionally obtains a new snapsh
src/skillevaluator/deduplication/plugin/profile.py:76
↓ 14 callersFunctionmanifest
(platform="github")
tests/ci/test_feedback.py:31
↓ 14 callersFunctionmetric_is_not_applicable
Return whether *reward* explicitly records *metric* as not applicable. Only the metrics in ``NOT_APPLICABLE_ELIGIBLE_METRICS`` can be N/A, only
src/skillevaluator/tier3/harbor/metrics.py:326
↓ 14 callersFunctionmigrate_dataset
Return a lossless object preview; write only an explicitly new JSON/YAML file. Case IDs retain their authored numeric/string types. Legacy unlabe
src/skillevaluator/tier3/dataset_migration.py:146
↓ 14 callersFunctionoverall_score_from_metrics
Aggregate one metric set using its versioned score policy. Metrics in *not_applicable* (N/A for every attempt of the arm, so not scored) are
src/skillevaluator/tier3/harbor/metrics.py:561
↓ 14 callersFunctionparse_environment_kwarg_overrides
Parse repeatable CLI ``KEY=VALUE`` arguments using Harbor's value rules.
src/skillevaluator/tier3/evals_config.py:730
↓ 14 callersFunctionpii_evidence_sha256
Return the digest that identifies a PII value in a waiver without storing the value. The value is case-folded because PII findings group values c
src/skillevaluator/validators/pii_waivers.py:78
↓ 14 callersMethodpoll
(self)
tests/test_harbor_runner_status.py:741
↓ 14 callersFunctionr
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:7
↓ 14 callersFunctionread_combined
(reports, row)
tests/ci/test_benchmark_reports.py:114
↓ 14 callersFunctionredact_sensitive_data
Recursively redact structured data using secret-looking key names.
src/skillevaluator/utils/redaction.py:344
↓ 14 callersFunctionstage_native_bundle
Write the bundle into ``env_dir/skilleval`` and return the Dockerfile COPY lines.
src/skillevaluator/tier3/harbor/native_staging.py:243
↓ 14 callersFunctionsummarize_skill_activation
Count logical trials, preserving uncertainty and excluding negative controls. A multi-step trial activated the skill if any step did. Missing or
src/skillevaluator/tier3/harbor/activation.py:65
↓ 14 callersFunctiontree
(root: Path, files: dict)
tests/test_plugin_inventory.py:21
↓ 14 callersFunctionvalidate_inventory_schema
Validate every declared format and compose its inventory findings. The additional manifests remain audit sources even when another client is
src/skillevaluator/validators/private_plugin_schema.py:34
↓ 14 callersFunctionvalidate_tier3_source
Resolve and validate the Tier 3 source selected by a skill.
src/skillevaluator/tier3/evals_spec.py:395
↓ 14 callersFunctionwhatwg_url
The absolute URL a WHATWG client resolves ``url`` (against ``base``) to, in a form urllib reads the same way. For http(s), ws(s), and ftp URLs th
src/skillevaluator/validators/url_policy.py:560
↓ 13 callersFunction_action_args
(tool_call: dict[str, Any])
src/skillevaluator/tier3/eval_core/checks.py:878
↓ 13 callersFunction_agent_eval_safe_text
(value: object)
src/skillevaluator/reporting/base.py:772
↓ 13 callersFunction_annotate_security_attribution
Annotate with-skill security findings with baseline-aware attribution.
src/skillevaluator/tier3/harbor/collector.py:6007
↓ 13 callersFunction_assert_hook_critical
One accurate Critical and manifest gate beside any independent completeness gates.
tests/validators/test_skillspector_hook_surface.py:342
↓ 13 callersFunction_bounded
(blocks: list[str], limit: int, footer: str)
ci/report_views.py:255
↓ 13 callersFunction_canary_operands
Arguments that are not options, redirections, or redirection targets.
src/skillevaluator/tier3/eval_core/plugin_security.py:863
↓ 13 callersFunction_claude
(*steps: dict[str, Any])
tests/tier3/test_f90_signals_handoff.py:46
↓ 13 callersFunction_collection
(root: Path, count: int)
tests/embedding/test_similarity_limits.py:23
↓ 13 callersFunction_dataset
(run: Path, *, negative: bool = True, ids: tuple[str, ...] = _IDS)
tests/test_tier3_next_steps.py:25
↓ 13 callersFunction_dataset
(skill: Path, *, suffix: str = ".json")
tests/tier3/test_workflow.py:30
↓ 13 callersFunction_entry_id
Resolve the canonical case ID for a collected reward dictionary.
src/skillevaluator/tier3/harbor/collector.py:4645
↓ 13 callersFunction_error_recovery
(trajectory: dict, extractor)
tests/tier3/test_codex_tool_call_normalization.py:1459
↓ 13 callersMethod_find_all_skills
Recursively find all skill directories containing SKILL.md files. Manifests beneath an excluded directory below ``root_path`` (e.g. `
src/skillevaluator/validators/base.py:205
↓ 13 callersMethod_find_skill_manifest
Locate the skill manifest, preferring canonical exact casing.
src/skillevaluator/validators/base.py:187
↓ 13 callersFunction_finding
( evidence: str, *, finding_type: str = "destructive_command", message: str = DESTRUCTIVE_REAS
tests/reporting/test_tier3_failure_reasons.py:49
↓ 13 callersMethod_get_client
Lazily construct the SDK client for the selected provider.
src/skillevaluator/inference/client.py:619
↓ 13 callersFunction_hook_plugin
(root: Path, command: str, event: str, matcher: str | None, files: dict[str, str])
tests/validators/test_plugin_component_risk.py:840
↓ 13 callersFunction_install_ragas
Install the RAGAS 0.4 modules the verifier imports; the metric sends one request through the OpenAI SDK.
tests/tier3/test_ragas_judge_evidence.py:146
↓ 13 callersFunction_llm_findings
(result)
tests/validators/test_rubric_eval.py:1273
↓ 13 callersFunction_openclaw
(root: Path, files: dict[str, Any] | None = None, **fields: Any)
tests/tier3/test_openclaw_prepare.py:41
↓ 13 callersFunction_paired_pass_comparison
Compare pass@k outcomes for matching cases across both evaluation arms.
src/skillevaluator/tier3/harbor/collector.py:4991
↓ 13 callersFunction_pass_summary
Summarize pass@k using SkillEvaluator continuous reward scores. *scorer* replaces the trial's own overall score; the cross-arm pass compariso
src/skillevaluator/tier3/harbor/collector.py:5126
↓ 13 callersFunction_provider
(name: str = "openai-compatible", model: str = "google/gemini-3.8-flash")
tests/test_harbor_gke_mode.py:43
↓ 13 callersFunction_reference_caveat_warning
The warning SkillSpector 2.13.3 reference caveats (``reference_caveat_metadata``) carry instead.
tests/validators/test_security.py:9363
↓ 13 callersFunction_render
(results: list[ValidationResult])
tests/reporting/test_verdict_evidence.py:100
↓ 13 callersFunction_render
(agents: dict[str, dict[str, Any]])
tests/reporting/test_tier3_incomplete_arms.py:99
↓ 13 callersMethod_resolve_path
(self, path: str | Path | None)
src/skillevaluator/tier3/harbor/local_environment.py:2039
↓ 13 callersFunction_reward
(**overrides)
tests/reporting/test_tier3_expectations.py:21
↓ 13 callersFunction_scan_expected
( tmp_path: Path, card: str, expected: benchmark_gate.ExpectedSource, *, require: bool = T
tests/test_benchmark_source_provenance.py:536
↓ 13 callersFunction_stage_verifier
(tmp_path: Path, entry: dict[str, Any] | None = None)
tests/tier3/test_judge_contracts.py:709
↓ 13 callersFunction_summary
(skills: list[dict], error: str | None = None)
ci/run_tier1.py:246
↓ 13 callersFunction_tier3_result
(verdict: str, *, legacy: bool = False)
tests/test_public_benchmark_gate.py:846
↓ 13 callersFunction_validate
(target: Path, reports: Path, workers: int = 1)
tests/test_catalog_discovery.py:38
↓ 13 callersMethod_validate_folder_or_skill
Template method for folder-level or single-skill validation. Eliminates duplicated folder detection logic across validators. Args:
src/skillevaluator/validators/base.py:225
↓ 13 callersFunction_weakens_requirements
True when advice removes or weakens a dataset requirement or takes a dataset case out of the dataset. ``case_ids`` are the ids of the dataset cas
src/skillevaluator/evaluation/insights_judge.py:482
↓ 13 callersFunction_write
(path: Path, data: object)
tests/tier3/test_dataset_migration.py:19
↓ 13 callersFunctionagent_eval_report_serialization_issue
Return why Tier 3 cannot be emitted losslessly within report limits.
src/skillevaluator/reporting/base.py:513
↓ 13 callersFunctionaudit
(tmp_path, text, monkeypatch, *, exit_code=0, output=None, available=True)
tests/validators/test_target_dependencies.py:14
↓ 13 callersMethodbind
(self, _source_address)
tests/test_model_catalog.py:361
↓ 13 callersFunctioncustom_metric_contract_error
Return a fixed diagnostic when one reward exceeds the custom-metric contract.
src/skillevaluator/tier3/harbor/metrics.py:248
↓ 13 callersFunctionensure_local_runtimes
Check requested local runtimes are installed and can report a version. Purely diagnostic — never installs. Probes run with a bounded timeout and
src/skillevaluator/tier3/harbor/local_runtime.py:681
↓ 13 callersFunctionevaluation
(monkeypatch: pytest.MonkeyPatch, tmp_path: Path)
tests/test_version_comparison.py:31
↓ 13 callersFunctionexercise
()
tests/test_harbor_local_environment.py:120
↓ 13 callersMethodfileno
(self)
src/skillevaluator/model_catalog.py:133
↓ 13 callersFunctionformat_evidence_volume
(value: object)
src/skillevaluator/tier3/evidence_volume.py:136
↓ 13 callersMethodgetContext
(t,e,i)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 13 callersFunctionhas_secret_shape
True when a literal value is shaped like a known token or key, or is an inline ``Bearer``/``Basic`` credential. A reference (:func:`is_env_refere
src/skillevaluator/validators/url_policy.py:251
↓ 13 callersFunctionis_generated_output_root
Return whether *path* has a valid marker bound to its canonical location.
src/skillevaluator/tier3/output_provenance.py:577
↓ 13 callersFunctionmark_generated_output_root
Claim or validate an output root before any generated child is replaced.
src/skillevaluator/tier3/output_provenance.py:673
↓ 13 callersMethodmonotonic
(self)
tests/tier3/test_judge_policies.py:35
↓ 13 callersFunctionparse_frontmatter
Parse YAML frontmatter from a markdown file. Centralizes the frontmatter extraction logic used by multiple validators. Args: file_pa
src/skillevaluator/validators/frontmatter_parser.py:37
↓ 13 callersFunctionparse_json_output
Parse JSON output from external tool. Args: output: Raw stdout from tool on_error: Optional message to check for in non-JSON outp
src/skillevaluator/utils/tool_runner.py:470
↓ 13 callersFunctionplugin_facts
Facts about a plugin folder: manifest formats, manifest validity, name and component counts.
src/skillevaluator/harness/plugin.py:93
↓ 13 callersFunctionresult_publication_evidence
Return one result's recognized built-in producer identity, if present.
src/skillevaluator/publication_evidence.py:92
↓ 13 callersFunctionrun_fixture
( tmp_path, monkeypatch, *, missing=False, steps=1, options=None, variants=("with"
tests/tier3/test_plugin_evidence.py:107
↓ 13 callersFunctionsafe_url
A URL for reports: no userinfo, query, or fragment; bounded (also for a URL that does not parse). An http(s), ws(s), or ftp URL is shown the way
src/skillevaluator/validators/url_policy.py:834
↓ 13 callersMethodsave
(self, results: list[ValidationResult], target: Path)
tests/reporting/test_unified_tier3_report.py:769
↓ 13 callersFunctionskill_root_for
Return the nearest ancestor of *path* that contains ``SKILL.md`` or ``skill.md``. Deleted skills and paths that never sit under a skill are ignor
src/skillevaluator/precommit.py:68
↓ 13 callersFunctionunit
(name, size=2)
tests/tier3/test_budget_reservation.py:17
← previousnext →601–700 of 24,741, ranked by callers