MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

↓ 58 callersFunction_process
Run one SkillSpector report through the Tier 1 result converter.
tests/validators/test_skillspector_scope.py:49
↓ 58 callersFunction_reference_report
Load a captured SkillSpector ``--no-llm`` report of the notes writer, the guide reader or the setup helper.
tests/validators/test_security.py:9316
↓ 58 callersMethodstart
(self)
src/skillevaluator/reporting/console_ui.py:577
↓ 57 callersFunction_deterministic_results
()
tests/golden/test_benchmark_card.py:69
↓ 57 callersFunction_skill
( root: Path, extra: str = "", desc: str = "Use when checking harness compatibility.", body: str = BODY )
tests/harness/test_overlay.py:27
↓ 56 callersFunctionload_evals_config
Load and validate ``evals/config.yml`` or ``evals/config.yaml``. Missing config is not an error. Returned dictionaries contain only keys sup
src/skillevaluator/tier3/evals_config.py:155
↓ 55 callersFunction_signals
(traj: dict[str, Any], case: dict[str, Any] | None = None, **kwargs: Any)
tests/tier3/test_plugin_signals.py:72
↓ 55 callersMethodread
(self, path: Path)
src/skillevaluator/tier3/loose_parts.py:85
↓ 54 callersFunction_checks
The most severe finding per check.
tests/validators/test_plugin_static_risk_review_fixes.py:59
↓ 54 callersMethodmonotonic
(self)
tests/test_retry.py:386
↓ 54 callersFunctionparse_dashboard_report
Normalize canonical Tier 3, validate-report, and embedded catalog JSON.
src/skillevaluator/reporting/dashboard_data.py:306
↓ 54 callersFunctionretained_results
Rehydrate the JSONReporter's public contract using explicit field maps. Results own deep copies of the payload's metadata by default. A caller th
src/skillevaluator/reporting/retained.py:171
↓ 53 callersFunctionchecker
(request)
tests/tier3/test_error_recovery.py:22
↓ 52 callersMethodpayload
(self)
tests/reporting/test_private_runtime_render.py:35
↓ 52 callersFunctionredact_sensitive_text
Best-effort masking for credentials before writing logs or artifacts.
src/skillevaluator/utils/redaction.py:262
↓ 52 callersMethodrun
Execute tool with given arguments. Args: args: Command arguments (tool path prepended automatically) cwd: Working dir
src/skillevaluator/utils/tool_runner.py:343
↓ 51 callersMethodload
(cls, path: Path)
src/skillevaluator/tier3/harbor/run_budget.py:48
↓ 49 callersFunction_html_report_data
(output: str)
tests/reporting/test_advisory_skip_verdict.py:33
↓ 49 callersFunctionexercise
()
tests/test_harbor_local_mode.py:314
↓ 49 callersMethodfeed
(self, text: str)
src/skillevaluator/tier3/harbor/stream_redaction.py:287
↓ 49 callersMethodjoinpath
(self, *_parts)
tests/reporting/test_chart_dependency.py:285
↓ 48 callersFunction_baseline_fairness_warnings
Explain dataset text the no-skill baseline is graded on but cannot satisfy. ``entry`` is a normalized dataset entry; ``skill_scripts`` are the fi
src/skillevaluator/tier3/evals_spec.py:963
↓ 48 callersFunction_call
(cmd: str, observation: str)
tests/tier3/test_error_recovery.py:32
↓ 48 callersFunction_claude
(root: Path, files: dict, manifest: dict | None = None)
tests/validators/test_plugin_static_risk_review_fixes.py:42
↓ 48 callersFunctioninvoke
(repo, release, *args)
tests/test_ci_onboarding_all_tiers.py:74
↓ 48 callersFunctionoverall_score
Compute pass@k/lift overall score for a reward payload. Default rewards use the equal-weight mean of the five canonical dimensions. Legacy me
src/skillevaluator/tier3/harbor/metrics.py:598
↓ 47 callersMethodclose
(self)
src/skillevaluator/tier3/harbor/progress.py:147
↓ 46 callersFunction_checks
(result: ValidationResult)
tests/validators/test_plugin_component_risk.py:43
↓ 46 callersFunctionexercise
()
tests/test_harbor_secure_docker_environment.py:299
↓ 45 callersFunction_plan
(progress: Any)
tests/test_tier3_progress.py:68
↓ 45 callersFunction_plugin_finding
( severity: Severity, check_name: str, message: str, file_path: str, suggestion: str,
src/skillevaluator/plugin_paths.py:245
↓ 45 callersFunction_trial
(run_dir: Path, agent: str, name: str, reward: dict, observation: str = "")
tests/test_tier3_case_summary.py:37
↓ 45 callersMethodadd_message
Add an informational message. For backward compatibility with existing validators.
src/skillevaluator/models/result.py:687
↓ 45 callersFunctioncomparison_rows
Compute with-minus-without differences only within an identical run.
src/skillevaluator/reporting/dashboard_data.py:497
↓ 45 callersMethodsleep
(self, seconds: float)
tests/test_retry.py:389
↓ 45 callersFunctionstat_is_link_or_reparse
Return whether metadata identifies a symlink or Windows reparse point.
src/skillevaluator/utils/secure_fs.py:113
↓ 44 callersFunction_write_reward
( job_dir: Path, trial_name: str, reward: dict[str, object], *, sidecar: dict[str, object]
tests/test_issue55_collector_truth.py:90
↓ 44 callersFunctionextract_tool_calls_as_dicts
Convert ATIF trajectory to the ``{"action", "action_input", "observation"}`` format consumed by ``eval_core.checks``.
src/skillevaluator/tier3/eval_core/atif_helpers.py:1413
↓ 44 callersFunctionpublication_target_from_path
Return the canonical publication target for one source snapshot.
src/skillevaluator/publication_identity.py:696
↓ 44 callersFunctionrender_agent_eval_html_report
Render a standalone Tier 3 HTML report and optional standard benchmark. ``saved_feedback`` is the report feedback a completed run saved in ``
src/skillevaluator/evaluation/tier3_report.py:686
↓ 43 callersMethodclose
(self)
tests/test_tier3_progress.py:1080
↓ 43 callersMethodmetadata
(self)
src/skillevaluator/tier3/harbor/run_budget.py:329
↓ 43 callersFunctionrun
(trajectory: dict)
tests/tier3/test_error_recovery.py:923
↓ 42 callersFunction_report
()
tests/reporting/test_dashboard_data.py:23
↓ 42 callersMethodparse
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 42 callersFunctionskill
(workspace, path)
tests/ci/test_run_tier1.py:45
↓ 41 callersFunction_joined
(*parts: str)
tests/test_oss_boundary.py:36
↓ 41 callersMethodfind
(self, sub, start=None, end=None)
tests/deduplication/utils/test_chunker.py:74
↓ 40 callersFunction_fake_gym_skill
(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, config: str = "")
tests/test_nemo_gym_backend.py:628
↓ 40 callersFunction_validate_agent_provider_credentials
Reject provider-to-agent combinations that cannot use the selected API.
src/skillevaluator/tier3/harbor/runner.py:2088
↓ 40 callersMethodadd_structured_finding
Add a structured finding with full context for CI reporting. This method stores the Finding object for detailed failure analysis and
src/skillevaluator/models/result.py:534
↓ 40 callersMethoddisplay
(self, rel: PurePosixPath | str)
src/skillevaluator/plugin_paths.py:137
↓ 40 callersFunctionenv_for
(consumer, **updates)
tests/ci/test_run_tiers.py:31
↓ 40 callersMethodexec
( self, command: str, env: dict[str, str] | None = None, **_kw
tests/test_harbor_local_mode.py:1400
↓ 40 callersFunctionload_quality_config
Load and validate the packaged config, or a custom file merged over it.
src/skillevaluator/quality_config.py:306
↓ 40 callersFunctiontrajectory_timing
Return ``{"llm_seconds", "tool_seconds", "source"}`` or ``None``.
src/skillevaluator/tier3/harbor/trajectory_timing.py:63
↓ 40 callersMethodvalidate
Validate skill(s) at path for schema and structure compliance.
src/skillevaluator/validators/schema.py:247
↓ 39 callersFunction_initialized_local_environment
( tmp_path: Path, *, persistent_env: dict[str, str] | None = None, )
tests/test_harbor_local_mode.py:142
↓ 39 callersFunction_live_tier3_result
()
tests/golden/test_benchmark_card.py:130
↓ 39 callersFunction_plugin
(root: Path, manifest: dict | None = None, files: dict[str, str | dict | list] | None = None)
tests/validators/test_plugin_component_risk.py:33
↓ 39 callersFunction_run_fake_gym
(skill: Path, tmp_path: Path, **overrides)
tests/test_nemo_gym_backend.py:682
↓ 39 callersFunction_validate_with
(skill_dir: Path, tmp_path: Path, config_body: str | None = None)
tests/validators/test_quality_score.py:1526
↓ 39 callersFunction_write_skill
(tmp_path: Path, name: str = "my-skill")
tests/validators/test_rubric_eval.py:37
↓ 39 callersFunctionload
(path)
tests/ci/test_all_tier_workflows.py:12
↓ 39 callersMethodwalk
(self, root: Path, **options: object)
tests/utils/test_secure_fs_platform.py:360
↓ 38 callersFunction_merged_step_trajectory
Merge a complete Harbor multi-step ATIF set or fail closed.
src/skillevaluator/tier3/harbor/collector.py:2859
↓ 38 callersFunction_provider
(provider: str = "openai")
tests/test_harbor_runner_environment.py:27
↓ 38 callersFunctionpayload
(passed=True, status="passed", incomplete=None, findings=1)
tests/ci/test_run_tier1.py:52
↓ 38 callersMethodwait
(self, timeout: int)
tests/test_harbor_local_mode.py:6061
↓ 37 callersFunction_configure_isolated_aws_profile
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, *, config: str, credentials: str = "",
tests/test_harbor_runtime_preflight.py:77
↓ 37 callersFunction_open
(page, directory: Path, name: str = "full")
tests/reporting/test_report_browser.py:441
↓ 37 callersFunction_reward
(activated: bool, reason: str)
tests/test_tier3_case_summary.py:45
↓ 37 callersFunction_traj
(function_name: str, arguments: dict)
tests/tier3/test_apply_patch_security.py:48
↓ 37 callersFunction_write_multistep_trial
( trial_root: Path, trajectories: tuple[tuple[str, dict[str, Any]], ...], *, resume_trajectory
tests/test_harbor_collector_multistep_trajectory.py:65
↓ 36 callersFunction_pass_card
( source_lines: str, *, tier3_status: str = "PASS", tier3_evidence: str = "required for public
tests/test_benchmark_source_provenance.py:396
↓ 36 callersFunctionbuild_token_usage
Return token usage per case, per agent and over all agents, or ``None`` when no attempt was measured. ``agents`` maps each agent to its report pa
src/skillevaluator/tier3/token_usage.py:556
↓ 36 callersFunctionfake_cli
(responses, calls)
tests/ci/test_run_tier1.py:86
↓ 36 callersFunctionstage_wrapper_skill
Write the wrapper skill and its combined dataset at ``destination`` (all or nothing). Member skills are copied without their evaluation directori
src/skillevaluator/tier3/plugin_prepare.py:579
↓ 35 callersFunction_initialized_docker_environment
( tmp_path: Path, *, environment_name: str = "secure-compose-test", persistent_env: dict[str,
tests/test_harbor_secure_docker_environment.py:64
↓ 35 callersFunction_skill
(root: Path, name: str = "demo")
tests/test_publication_identity.py:31
↓ 35 callersFunction_skill
(tmp_path, guidance=None)
tests/tier3/test_dataset_auto_grounding.py:17
↓ 35 callersFunction_use_provider
(monkeypatch: pytest.MonkeyPatch, provider: str, model: str)
tests/inference/test_llm_client.py:1011
↓ 35 callersMethodnow
(cls, tz: tzinfo | None = None)
tests/test_public_benchmark_gate.py:45
↓ 35 callersMethodrecord_usage
( self, usage: dict[str, Any] | None, *, model: str | None = None, attempt: dict[str, Any] | None = No
src/skillevaluator/tier3/harbor/run_budget.py:265
↓ 35 callersFunctionrun_validation
Run selected Tier 1 validators and return structured results. Checks named in *skip* do not run; each is recorded as a user-skipped result, w
src/skillevaluator/tier1/commands.py:309
↓ 34 callersFunction_agent
( with_skill: list[dict[str, Any]], baseline: list[dict[str, Any]], *, with_status: str = "suc
tests/tier3/test_token_usage.py:26
↓ 34 callersFunction_make_mock_client
Create a mock EmbeddingClient that returns predetermined vectors.
tests/embedding/test_registry.py:71
↓ 34 callersFunction_provider
( provider: str, *, model: str = "configured-model", api_key: str | None = "top-secret-key",
tests/test_model_catalog.py:44
↓ 34 callersFunction_record_observation
(records: list[dict])
tests/tier3/test_codex_tool_call_normalization.py:1097
↓ 34 callersFunctionapply_policy
Remap finding severities per the active policy and recompute pass/fail. Validators emit findings with a default severity; this central pass appli
src/skillevaluator/validators/policy.py:368
↓ 34 callersFunctionload_agent_data
Load per-agent summaries, rewards, lift, and execution coverage.
src/skillevaluator/tier3/harbor/report_data.py:861
↓ 34 callersFunctionmarkdown_link_targets
Return parser-normalized Markdown destinations in document order.
src/skillevaluator/validators/markdown.py:225
↓ 34 callersFunctions
(t)
src/skillevaluator/reporting/assets/chart.umd.min.js:7
↓ 33 callersFunction_check_prerequisites
Check Harbor and the selected environment (built-in, backend, GKE, or local mode).
src/skillevaluator/tier3/harbor/runner.py:2620
↓ 33 callersFunction_command
()
tests/tier3/test_workflow.py:47
↓ 33 callersFunction_finding_messages
(skill_dir: Path)
tests/validators/test_quality_score.py:132
↓ 33 callersFunction_is_completed
(run: Path)
tests/test_results_location_legacy_compatibility.py:184
↓ 33 callersMethod_tier3_report_data
Return safe Tier 3 display data without certifying partial evidence.
src/skillevaluator/reporting/html.py:3103
↓ 33 callersFunction_trial
(case: str, prompt: int, completion: int, cached: int | None = None)
tests/tier3/test_token_usage.py:19
↓ 33 callersFunction_validation_result_from_payload
Wrap a canonical Tier 3 payload in the shared validation-result model.
src/skillevaluator/evaluation/tier3_report.py:637
← previousnext →101–200 of 24,741, ranked by callers