MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

↓ 10 callersFunction_skill
(name: str)
tests/tier3/test_openclaw_prepare.py:29
↓ 10 callersFunction_skill
(path: Path)
tests/tier3/test_decoy_identity.py:18
↓ 10 callersFunction_skill
(name: str)
tests/deduplication/plugin/test_openclaw_profile.py:56
↓ 10 callersFunction_skillspector_provider_outage_report
Return a SkillSpector 2.13.3 LLM-stage report whose only fatal cause is provider retry exhaustion. ``old_scanner`` adds what SkillSpector can als
tests/validators/test_security.py:6889
↓ 10 callersFunction_skip_baseline_agent
(with_skill: dict[str, float], *, pass_rate: float = 0.5)
tests/test_tier3_result_display.py:692
↓ 10 callersFunction_stat_is_link_or_reparse
(metadata: os.stat_result)
src/skillevaluator/embedding/registry.py:1334
↓ 10 callersFunction_step
(call_id: str, fn: str, args: dict[str, Any], content: str = "ok", **result: Any)
tests/tier3/test_f90_signals_conflict.py:49
↓ 10 callersFunction_trajectory
(agent: str, *steps: dict[str, Any])
tests/tier3/test_f90_signals_conflict.py:57
↓ 10 callersFunction_waiver_checks
(result)
tests/validators/test_pii_waivers.py:79
↓ 10 callersFunction_write
(root: Path, relative: str, text: str)
tests/test_select_docs_tests.py:15
↓ 10 callersFunctionanthropic_to_chat_request
Translate an Anthropic Messages request to a Build Chat Completions request.
src/skillevaluator/tier3/harbor/nvidia_build_bridge.py:1136
↓ 10 callersFunctionappend_character_facts
Append bounded facts for the evidence's prompt representation, when enabled.
src/skillevaluator/tier3/eval_core/character_facts.py:68
↓ 10 callersFunctionarchive
(payload=None, extra=None, prefix="")
tests/ci/test_feedback.py:49
↓ 10 callersFunctionbaseline_checkout
(tmp_path, manifest=None)
tests/ci/test_run_tier1.py:747
↓ 10 callersFunctionbuild_dataset_snapshot
Build immutable, digest-backed dataset truth owned by one evaluation run.
src/skillevaluator/tier3/harbor/report_data.py:142
↓ 10 callersMethodchat
Send a multi-turn conversation and return the reply text. ``messages`` is an OpenAI-style history: optional leading ``system`` messag
src/skillevaluator/inference/client.py:701
↓ 10 callersMethodcommunicate
(self, input_text: str, timeout: int)
tests/test_harbor_local_mode.py:5966
↓ 10 callersFunctioncontains_credential_value
Return whether text contains credential material, including embedded tokens.
src/skillevaluator/utils/redaction.py:319
↓ 10 callersFunctioncreate_skill_with_declared_license_and_file
Create a skill with both SKILL.md license and LICENSE file content.
tests/validators/test_license.py:250
↓ 10 callersMethodcreate_user_prompt
(self, **_kwargs)
tests/inference/test_llm_client.py:1816
↓ 10 callersFunctioncustom_metric_name_is_publishable
Return whether a custom metric name is bounded and safe to publish.
src/skillevaluator/tier3/harbor/metrics.py:222
↓ 10 callersMethoddownload_artifact
(self, artifact_id)
tests/ci/test_publish_reports.py:1395
↓ 10 callersFunctionevaluate_harnesses
Evaluate one skill (and the catalog it is installed in) against the selected profiles.
src/skillevaluator/harness/overlay.py:1004
↓ 10 callersMethodfinish
(self, return_code: int)
tests/test_harbor_secure_docker_environment.py:223
↓ 10 callersMethodgetDataVisibility
(t)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 10 callersFunctiongit
(repo, *arguments)
tests/ci/test_select_changed_skills.py:20
↓ 10 callersFunctionis_tier3_result
Return whether a result belongs to live Tier 3 evaluation.
src/skillevaluator/reporting/base.py:2034
↓ 10 callersFunctionload_dataset_entries
Load normalized dataset entries from a JSONL, JSON, YAML, or YML file. Legacy SkillEvaluator datasets use a flat list of entries with ``question`
src/skillevaluator/tier3/dataset_utils.py:67
↓ 10 callersMethodmatch
See :func:`match_declared_mcp_server`.
src/skillevaluator/tier3/eval_core/plugin_signals.py:2168
↓ 10 callersMethodname
(self)
src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py:937
↓ 10 callersFunctionnormalize_declared_path
Normalize one manifest path to a contained root-relative POSIX path. Claude Code requires ``./``-relative paths (``"."``/``"./"`` names the root)
src/skillevaluator/plugin_paths.py:75
↓ 10 callersFunctionnormalize_ref
Normalize string and selector refs without fetching remote content.
src/skillevaluator/plugin_refs.py:23
↓ 10 callersFunctionnormalize_report_path
Return a scanner location relative to *scan_root*, or ``unknown``. The location must name an existing regular file inside the scanned artifac
src/skillevaluator/validators/skillspector_scope.py:278
↓ 10 callersFunctionplugin_case_spec
Return the valid, normalized plugin-signal fields of one case entry. Invalid fields are dropped (the dataset validators report them), so a ma
src/skillevaluator/tier3/eval_core/plugin_signals.py:1228
↓ 10 callersMethodprint_all
Print all results directly to console.
src/skillevaluator/reporting/cli.py:181
↓ 10 callersMethodquery_entry
Compare one extracted target entry against every catalog entry. ``exclude_entry_id`` skips one catalog entry, such as the target's own
src/skillevaluator/embedding/registry.py:517
↓ 10 callersMethodread
(self, max_bytes, timeout=None)
tests/tier3/test_mcp_proof_sdk_compat.py:284
↓ 10 callersFunctionrender_case_summary
Print the case grid and fixes; False when the result has no per-case rows to show.
src/skillevaluator/tier3/case_summary.py:778
↓ 10 callersFunctionrenderers
()
tests/reporting/test_private_static_reports.py:28
↓ 10 callersFunctionreport_basename
Return the report basename for a report *kind* (e.g. ``"quality"``). ``report_basename()`` returns the default full-run basename.
src/skillevaluator/reporting/naming.py:24
↓ 10 callersFunctionrepository_commit
(workspace)
tests/ci/test_run_tier1.py:730
↓ 10 callersMethodresolveDataElementOptions
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 10 callersFunctionselect_docs_tests
Return the test files under ``root/tests`` that read documentation, as sorted POSIX paths.
scripts/select_docs_tests.py:71
↓ 10 callersMethodservice_download_dir
( self, source_dir: str, target_dir: Path | str, *, service: str | Non
src/skillevaluator/tier3/harbor/secure_docker_environment.py:2481
↓ 10 callersMethodsettimeout
(self, timeout: float)
tests/test_model_catalog.py:358
↓ 10 callersFunctionstatic_report
()
tests/reporting/test_plugin_evidence_report.py:21
↓ 10 callersFunctionsummarize_tier2
Return (ran, passed, rows, skip_reason) for the dedup results.
src/skillevaluator/reporting/console_ui.py:1456
↓ 10 callersFunctionsynthetic_trajectory_from_opencode_json
Parse OpenCode ``run --format=json`` JSONL (tee'd to ``opencode.txt``).
src/skillevaluator/tier3/eval_core/log_converters.py:493
↓ 10 callersFunctiontext
(value: object)
src/skillevaluator/reporting/plugin_evidence.py:61
↓ 10 callersMethodvalidate
Validate a skill directory and produce quality scores.
src/skillevaluator/validators/quality_score.py:347
↓ 10 callersFunctionvalidate_mcp_server_declarations
Validate MCP server declarations against operator secret exfiltration and unapproved auth headers.
src/skillevaluator/tier3/harbor/adapter.py:2032
↓ 9 callersFunction_agent
(frontmatter: str)
tests/validators/test_plugin_component_risk.py:47
↓ 9 callersFunction_assert_failed_with_incomplete_scan
The scan is recorded as incomplete, but the check is reported as failed. A confirmed failure, such as the Critical, takes precedence over missing
tests/validators/test_skillspector_hook_surface.py:362
↓ 9 callersMethod_assert_work_limit_incomplete
A tripped work limit is missing coverage, never a duplicate-content finding.
tests/deduplication/intra_skill/test_intra_skill_validator.py:260
↓ 9 callersFunction_attempt
(case: str, trial_id: str, accuracy: float | None, judge: str | None)
tests/test_judge_lift_integrity.py:424
↓ 9 callersFunction_attempts
(verifier_dir: Path)
tests/tier3/test_judge_contracts.py:758
↓ 9 callersFunction_autopilot_skill
(tmp_path: Path)
tests/test_evaluation_service.py:327
↓ 9 callersMethod_build_register_mcp_servers_command
Build MCP registration command supporting streamable-http and headers from mcp_servers.json.
src/skillevaluator/tier3/harbor/local_agents.py:318
↓ 9 callersFunction_canary_command_words
Return ``(name, args, assignments)`` past reserved words, prefix assignments and redirections, and wrappers.
src/skillevaluator/tier3/eval_core/plugin_security.py:877
↓ 9 callersFunction_catalog
(root: Path, path: Path, client: EmbeddingClient)
tests/tier3/test_decoys.py:48
↓ 9 callersFunction_checks
(result: ValidationResult)
tests/validators/test_plugin_hook_risk_bounds.py:56
↓ 9 callersFunction_client
(vectors: list[list[float]] | None = None)
tests/tier3/test_decoys.py:39
↓ 9 callersFunction_collect
( root: Path, *, native: bool = False, all_mixed: bool = False, judge_free: bool = False,
tests/test_judge_lift_integrity.py:103
↓ 9 callersFunction_condition_execution_summary
Describe whether a Harbor condition produced complete logical attempts. Native multi-step tasks may emit several reward rows for one Harbor trial
src/skillevaluator/tier3/harbor/collector.py:6907
↓ 9 callersFunction_conflict
( trajectory: dict[str, Any], *probes: dict[str, Any], context: Any = CONTEXT, arm: str = ARM_WITH_SKILL )
tests/tier3/test_f90_signals_conflict.py:66
↓ 9 callersMethod_container_reasons
(tmp_path: Path, revision: str)
tests/test_benchmark_source_provenance.py:965
↓ 9 callersFunction_copy_custom_grader
Copy user custom grader support into a generated task, if configured.
src/skillevaluator/tier3/harbor/adapter.py:2455
↓ 9 callersFunction_count
(value: object)
src/skillevaluator/reporting/plugin_static.py:46
↓ 9 callersFunction_counts
(block: dict[str, Any])
tests/tier3/test_f90_mcp_execution.py:34
↓ 9 callersFunction_custom_payload
(**updates)
tests/reporting/test_native_overview_comparison.py:39
↓ 9 callersFunction_detect_from_directory
Detect content type from directory contents.
src/skillevaluator/cli_core.py:51
↓ 9 callersFunction_entry_file_refs
(entry: dict[str, Any])
src/skillevaluator/tier3/harbor/adapter.py:4043
↓ 9 callersFunction_environment_kwarg_prerequisite_errors
Validate constructor requirements Harbor cannot check in ``preflight``.
src/skillevaluator/tier3/harbor/runner.py:2289
↓ 9 callersMethod_exec_env
(self, env: dict[str, str] | None)
src/skillevaluator/tier3/harbor/local_environment.py:1541
↓ 9 callersFunction_execution_unavailable
(agent: Mapping[str, Any])
src/skillevaluator/tier3/case_summary.py:128
↓ 9 callersFunction_failed_result
(validator_name: str, *, severity: Severity | None = Severity.HIGH)
tests/reporting/test_reporters.py:84
↓ 9 callersFunction_failed_run
Write a run whose ``arm`` failed; ``legacy_ids`` leaves ``trial_id`` out of the failed arm's rewards. ``scored_trials`` are the failed arm's scor
tests/reporting/test_failed_arm_trials.py:28
↓ 9 callersFunction_finite_number
(value: object)
src/skillevaluator/tier3/result_display.py:55
↓ 9 callersFunction_html_tab
(output: str, tab_id: str)
tests/reporting/test_reporters.py:74
↓ 9 callersFunction_info
(with_rewards: list[dict], baseline_rewards: list[dict])
tests/test_judge_lift_integrity.py:433
↓ 9 callersFunction_install
(folder: Path, dirname: str, name: str | None = None)
tests/harness/test_overlay.py:399
↓ 9 callersFunction_invoke
(args: list[str], user_input: str | None = None)
tests/evaluation/test_tier3_feedback_chat.py:132
↓ 9 callersFunction_is_execution_action
(action: str)
src/skillevaluator/tier3/eval_core/checks.py:1224
↓ 9 callersMethod_is_skill_directory
Check if path is a skill directory (contains SKILL.md).
src/skillevaluator/validators/base.py:221
↓ 9 callersFunction_is_vertex_openapi_endpoint
Return whether the base URL points to a Vertex AI Agent Platform OpenAPI endpoint.
src/skillevaluator/provider_config.py:357
↓ 9 callersFunction_mapping
(value: object)
src/skillevaluator/reporting/tier3_token_usage.py:81
↓ 9 callersFunction_matrix
(tmp_path: Path, scores=(0.50, 0.75, 0.70, 0.65), *, custom_only=False)
tests/tier3/test_interactions.py:106
↓ 9 callersFunction_merge_attempt_jobs
Merge per-attempt Harbor jobs into the job directory shape collection expects. Trial directories are copied under attempt-qualified names and the
src/skillevaluator/tier3/harbor/runner.py:4480
↓ 9 callersFunction_page
(html, name, next_name)
tests/reporting/test_report_clarity.py:62
↓ 9 callersFunction_passes
(trajectory: dict[str, Any], **handoff: Any)
tests/tier3/test_f90_signals_handoff.py:89
↓ 9 callersMethod_peek
(self, offset: int = 0)
src/skillevaluator/plugin_component_risk.py:1264
↓ 9 callersFunction_plugin
( root: Path, manifest: str | None, data: dict | None = None, components: tuple[str, ...] = ("skills",) )
tests/harness/test_plugin_compat.py:19
↓ 9 callersMethod_process_skillspector_cli_result
Convert a validated SkillSpector JSON report into ValidationResult entries. With a ``content_type``, issues outside agent control paths are
src/skillevaluator/validators/security.py:5083
↓ 9 callersFunction_provider
(name="nv_build")
tests/test_hermes_agent.py:56
↓ 9 callersFunction_report
(run: Path)
tests/test_judge_lift_integrity.py:153
↓ 9 callersFunction_resolver
(found: dict[str, str])
tests/utils/test_scanner_install.py:20
↓ 9 callersFunction_routed
Answer each judge by its prompt, so a full verifier run can use one fake.
tests/tier3/test_judge_contracts.py:742
↓ 9 callersFunction_run_template
(tmp_path, extra_env, *, scanner_exit=0, existing_report=False)
tests/ci/test_gitlab_template.py:22
↓ 9 callersFunction_scorer
The host or bundled-verifier scorer that pairs with ``extractor``.
tests/tier3/test_codex_tool_call_normalization.py:1454
↓ 9 callersFunction_script
(responses: list[tuple[str | None, str | None]], calls: list[dict[str, Any]])
tests/tier3/test_judge_contracts.py:60
← previousnext →901–1,000 of 24,741, ranked by callers