MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_routing_rejects_dir_listing
Listing/finding a SKILL.md (no read verb) is not a read — must not pass.
tests/tier3/test_checks_path_extraction.py:630
Functiontest_routing_rejects_skill_md_string_search_even_when_output_mentions_expected_skill
()
tests/tier3/test_checks_path_extraction.py:749
Functiontest_routing_rejects_skill_word_search
()
tests/tier3/test_checks_path_extraction.py:695
Functiontest_routing_rejects_wrong_skill_sed
Broadening read-verb detection must not weaken wrong-skill detection.
tests/tier3/test_checks_path_extraction.py:616
Functiontest_routing_tri_state_preserves_explicit_precedence_and_legacy_unlabeled
(entry, expected)
tests/tier3/test_negative_control_invocation_evidence.py:47
Functiontest_row_and_metric_limits
(payload)
tests/tier3/test_external_metrics.py:339
Functiontest_row_cap_note_appears_only_when_drilldown_rows_were_capped
()
tests/reporting/test_tier3_failure_reasons.py:981
Methodtest_row_line_must_match_the_references_list
(self, mock_tools, tmp_path: Path, case: str)
tests/validators/test_security.py:10683
Methodtest_row_must_name_an_unmodeled_manifest
(self)
tests/validators/test_skillspector_hook_surface.py:1110
Methodtest_row_record_counts_must_match_the_declarations
( self, counts: dict, declarations: int, accepted: bool )
tests/validators/test_skillspector_hook_surface.py:1141
Functiontest_rows_cells_and_notes_are_bounded_with_local_disclosures
()
tests/reporting/test_native_evidence.py:161
Methodtest_rows_of_another_shape_stay_incomplete
(self, mutate: Callable[[dict], None])
tests/validators/test_skillspector_hook_surface.py:980
Functiontest_rubric_progress_status_renders_skill_name_literally
( name: str, tmp_path: Path, monkeypatch: pytest.MonkeyPatch )
tests/reporting/test_rich_markup_escaping.py:634
Functiontest_rule_and_hook_refs_are_rejected_with_a_clear_error
check-19 e03: these refs could never match a tool call.
tests/tier3/test_f90_signals_conflict.py:166
Functiontest_rule_frontmatter_uses_the_stripped_staging_text
(tmp_path)
tests/test_plugin_discovery.py:422
Functiontest_rule_ids_named_in_messages_exist
()
tests/harness/test_overlay.py:721
Methodtest_rule_only_skillspector_baseline_ignores_scanner_version
( self, mock_tools, sample_skill_dir: Path, tmp_path: Path )
tests/validators/test_security.py:2179
Functiontest_rule_ref_to_directory_or_hardlink_is_not_a_valid_rule
(tmp_path: Path)
tests/validators/test_plugin_dependencies.py:221
Methodtest_rule_without_matching_call_is_reported_and_counted
(self)
tests/tier3/test_plugin_signals.py:727
Functiontest_rules_are_byte_identical_to_native_mode_and_on_request_rules_stay_out_of_both
(loose, source)
tests/tier3/test_private_loose_source.py:165
Functiontest_rules_whose_tool_was_never_called_count_against_the_rate
()
tests/tier3/test_f90_mcp_arguments.py:76
Functiontest_run_agent_eval_partial_run_returns_ran_with_errors
(monkeypatch, tmp_path)
tests/test_commands.py:2828
Methodtest_run_bandit_end_to_end
(self, mock_run)
tests/validators/test_code_risk_structured.py:78
Methodtest_run_can_replace_ambient_environment
Sensitive subprocesses can opt into an exact invocation environment.
tests/utils/test_tool_runner.py:381
Methodtest_run_config_supplies_the_identity
(self)
tests/test_benchmark_source_provenance.py:239
Methodtest_run_decodes_non_utf8_output_tolerantly
A child emitting bytes outside the selected codec must not lose its captured streams. A strict decode raises UnicodeDecodeError inside subpro
tests/utils/test_tool_runner.py:324
Methodtest_run_decodes_text_with_replacement
subprocess.run keeps text mode but replaces undecodable bytes.
tests/utils/test_tool_runner.py:354
Functiontest_run_directory_failed_baseline_trial_reaches_the_coverage_notice
The loader's condition errors carry the Harbor exception type the notice names.
tests/reporting/test_tier3_incomplete_arms.py:301
Functiontest_run_end_close_during_a_refused_heartbeat_stop_shows_the_cursor
(monkeypatch: pytest.MonkeyPatch)
tests/test_tier3_progress.py:1367
Functiontest_run_error_is_found_for_a_trial_whose_name_does_not_start_with_its_case
( tmp_path: Path, case_id: str, trial_name: str, records: dict )
tests/test_tier3_case_summary.py:1110
Functiontest_run_errors_already_listed_per_case_are_not_repeated
(tmp_path: Path)
tests/test_tier3_next_steps.py:64
Functiontest_run_errors_rank_after_skill_issues_and_are_not_blamed_on_the_skill
(tmp_path: Path)
tests/test_tier3_case_summary.py:856
Functiontest_run_errors_with_different_causes_stay_separate_across_agents
(tmp_path: Path)
tests/test_tier3_case_summary.py:978
Methodtest_run_executes_when_available
run executes command when tool is available.
tests/utils/test_tool_runner.py:278
Methodtest_run_gitleaks_end_to_end
(self, mock_run)
tests/validators/test_secrets_structured.py:59
Functiontest_run_harbor_accepts_complete_successful_job
(monkeypatch: pytest.MonkeyPatch, tmp_path: Path)
tests/test_harbor_runner_status.py:2704
Functiontest_run_harbor_accepts_pinned_harbor_0132_job_stats
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_harbor_runner_status.py:2710
Functiontest_run_harbor_caps_configured_adc_timeout_and_rejects_near_expiry
Cap SKILLEVALUATOR_VERTEX_ADC_JOB_TIMEOUT_SEC to remaining token ceiling and reject <= 30s tokens.
tests/test_harbor_runner_environment.py:2558
Functiontest_run_harbor_checks_cancellation_before_refreshing_credentials
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_harbor_runner_status.py:608
Functiontest_run_harbor_defers_to_harbor_phase_timeouts_without_outer_deadline
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, task_toml: str, )
tests/test_harbor_runner_status.py:2265
Functiontest_run_harbor_enforces_adc_job_timeout_and_stages_credential_source
Enforce bounded ADC timeout on non-GKE jobs and stage SKILL_EVAL_LLM_CREDENTIAL_SOURCE for the verifier.
tests/test_harbor_runner_environment.py:2507
Functiontest_run_harbor_eval_rejects_native_windows_before_provider_or_config
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_local_mode.py:5459
Functiontest_run_harbor_eval_rejects_provenance_key_inside_skill_before_creation
( monkeypatch: pytest.MonkeyPatch, tmp_path, )
tests/test_harbor_runner_environment.py:1522
Functiontest_run_harbor_eval_stages_per_agent_credential_trees
( monkeypatch: pytest.MonkeyPatch, tmp_path, )
tests/test_harbor_runner_environment.py:1158
Functiontest_run_harbor_fails_closed_with_redacted_tail_on_output_overflow
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_harbor_runner_status.py:2349
Functiontest_run_harbor_preserves_explicit_vertex_credential
Preserve explicit operator credentials on Vertex URLs without overwriting from host ADC.
tests/test_harbor_runner_environment.py:2440
Functiontest_run_harbor_real_subprocess_redacts_percent_decoded_proxy_components
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_harbor_runner_status.py:2447
Functiontest_run_harbor_redacts_detached_proxy_components_and_persisted_launch_error
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, proxy_uri: str, )
tests/test_harbor_runner_status.py:2396
Functiontest_run_harbor_redacts_detached_proxy_components_on_failure_paths
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, failure_path: str, )
tests/test_harbor_runner_status.py:2486
Functiontest_run_harbor_redacts_secrets_created_by_final_diagnostic_assembly
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, secret: str, bounded_result: runner._Bounde
tests/test_harbor_runner_status.py:2576
Functiontest_run_harbor_reissues_adc_token_for_bounded_jobs
Verify _run_harbor refreshes ADC token in child env without leaking secrets to argv and sets a bounded timeout.
tests/test_harbor_runner_environment.py:2388
Functiontest_run_harbor_rejects_duplicate_rewarded_trial_name
(monkeypatch: pytest.MonkeyPatch, tmp_path: Path)
tests/test_harbor_runner_status.py:2838
Functiontest_run_harbor_rejects_incomplete_pinned_harbor_0132_job_stats
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, counter: str, expected: str, )
tests/test_harbor_runner_status.py:2741
Functiontest_run_harbor_rejects_incomplete_real_harbor_statistics
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, stats: dict[str, object], expected: str, )
tests/test_harbor_runner_status.py:2806
Functiontest_run_harbor_rejects_missing_job_result
(monkeypatch: pytest.MonkeyPatch, tmp_path: Path)
tests/test_harbor_runner_status.py:2666
Functiontest_run_harbor_rejects_non_successful_trial_states
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, overrides: dict[str, object], expected: str
tests/test_harbor_runner_status.py:2690
Functiontest_run_harbor_rejects_reward_coverage_shortfall
(monkeypatch: pytest.MonkeyPatch, tmp_path: Path)
tests/test_harbor_runner_status.py:2820
Functiontest_run_harbor_rejects_zero_trials
(monkeypatch: pytest.MonkeyPatch, tmp_path: Path)
tests/test_harbor_runner_status.py:2673
Functiontest_run_harbor_reports_timeout_after_bounded_runner_contains_tree
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_harbor_runner_status.py:2531
Functiontest_run_harbor_uses_collision_safe_marker_for_synthesized_secret
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_harbor_runner_status.py:2610
Methodtest_run_keeps_the_selected_text_codec
Output in the parent's selected codec (a non-UTF-8 locale) keeps its non-ASCII paths.
tests/utils/test_tool_runner.py:341
Methodtest_run_level_runtime_row_accompanies_provider_failure
The SKILL.md runtime row may accompany a helper-file or meta-analysis outage.
tests/validators/test_security.py:7107
Methodtest_run_merges_extra_env_over_ambient
env additions are merged onto os.environ, not substituted for it.
tests/utils/test_tool_runner.py:367
Methodtest_run_of_10_is_high
(self, tmp_path: Path)
tests/validators/test_unicode_smuggle.py:131
Methodtest_run_of_40_is_critical
(self, tmp_path: Path)
tests/validators/test_unicode_smuggle.py:141
Functiontest_run_on_one_skill_judges_only_that_skills_waivers
One waiver file serves a whole folder; a run on one skill must not call the others stale.
tests/validators/test_pii_waivers.py:344
Functiontest_run_record_bounds_launch_errors_and_the_runner_drops_only_explained_ones
()
tests/tier3/test_baseline_timeouts.py:1592
Functiontest_run_result_bounds_multibyte_launch_errors_for_report_loader
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_tier3_progress.py:3169
Methodtest_run_returns_error_when_unavailable
run returns ToolResult with error when tool not installed.
tests/utils/test_tool_runner.py:269
Methodtest_run_root_inside_home_with_spaces_is_usable
(self)
tests/test_harbor_local_sandbox.py:1539
Methodtest_run_root_is_only_writable_bind
(self, tmp_path: Path)
tests/test_harbor_local_sandbox.py:399
Functiontest_run_scoped_baseline_alias_validation_skips_rescan_and_is_path_bound
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_runtime_skill_isolation.py:4937
Methodtest_run_semgrep_end_to_end
(self, mock_run)
tests/validators/test_code_risk_structured.py:106
Methodtest_run_semgrep_excludes_canonical_generated_benchmark
(self, mock_run, tmp_path)
tests/validators/test_code_risk_structured.py:126
Methodtest_run_semgrep_follows_filesystem_case_aliases
(self, mock_run, tmp_path)
tests/validators/test_code_risk_structured.py:148
Methodtest_run_semgrep_reports_parse_errors_with_relative_paths
( self, tmp_path: Path, mock_tool_available )
tests/validators/test_security_validators.py:810
Methodtest_run_time_and_time_budget_do_not_change_the_finding_id
Two runs that differ only in the run time and the time budget give the same suggestion and ``finding_id``. SkillSpector sets ``limit_seconds`
tests/validators/test_skillspector_reference_diagnostics.py:389
Functiontest_run_time_substitution_is_undecidable_not_a_failure
A path the shell fills in at run time cannot be compared statically.
tests/tier3/test_script_invocation_evidence.py:239
Functiontest_run_validation_plugin_does_not_require_catalog_path
Plugin validation must succeed without any catalog (no dedup dependency).
tests/cli/test_validate_plugin.py:46
Functiontest_run_validation_plugin_skips_quality_and_lint
quality/lint are skill-only and must be skipped for plugins.
tests/cli/test_validate_plugin.py:33
Functiontest_run_validation_plugin_uses_plugin_schema_validator
(tmp_path: Path)
tests/cli/test_validate_plugin.py:26
Methodtest_run_validation_scopes_only_detected_agent_artifacts
( self, monkeypatch: pytest.MonkeyPatch, sample_skill_dir: Path, content_type:
tests/validators/test_skillspector_scope.py:1811
Functiontest_run_verdict_ignores_an_agent_that_failed_only_its_baseline
()
tests/test_tier3_result_display.py:396
Functiontest_run_verdict_ignores_an_agent_whose_with_skill_arm_failed
(shape: str)
tests/test_tier3_result_display.py:279
Functiontest_run_verdict_is_incomplete_when_no_agent_ran_its_with_skill_arm
(shape: str)
tests/test_tier3_result_display.py:301
Functiontest_run_verdict_is_pass_when_the_agent_that_ran_passes
(shape: str)
tests/test_tier3_result_display.py:289
Functiontest_run_verdict_uses_passing_agent_counts_and_has_no_failure_reason
()
tests/test_tier3_attempt_verdict.py:560
Methodtest_run_without_env_leaves_subprocess_default
No env argument means subprocess inherits the parent environment.
tests/utils/test_tool_runner.py:394
Functiontest_run_wrapper_defers_timestamp_lookup_to_directory_loader
(monkeypatch, tmp_path: Path)
tests/evaluation/test_tier3_benchmark_contract.py:467
Functiontest_runner_additional_secret_values_redact_callback_result_and_error
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_secure_docker_environment.py:6969
Functiontest_runner_and_retained_publisher_never_export_detected_value
(tmp_path, capsys, monkeypatch)
tests/ci/test_secret_redaction.py:87
Functiontest_runner_atomically_publishes_one_complete_final_result
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_tier3_progress.py:5683
Functiontest_runner_continues_when_reporter_callbacks_raise
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_tier3_progress.py:5720
Functiontest_runner_does_not_mark_invalid_configuration_ready
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_tier3_progress.py:5439
Functiontest_runner_does_not_retain_recovered_deadline_as_launch_failure
(tmp_path, monkeypatch)
tests/test_harbor_timeout_recovery.py:301
Functiontest_runner_emits_truthful_stages_plan_and_per_agent_state
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_tier3_progress.py:5217
Functiontest_runner_fails_publication_closed_for_external_repo_runtime_context
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, context_mode: str, source_is_target_bound:
tests/test_tier3_progress.py:4526
Functiontest_runner_forwards_host_retry_settings_to_staged_task_and_verifier_runtime
Verify host SKILL_EVAL_LLM_* retry settings flow through runner staging, subprocess env, and eval.py.
tests/test_harbor_task_toml_retry_env.py:146
Functiontest_runner_fresh_reused_and_incompatible
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, grading_mode: str )
tests/test_baseline_store.py:403
Functiontest_runner_invokes_reporter_callbacks_only_on_coordinator_thread
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_tier3_progress.py:5471
← previousnext →21,601–21,700 of 24,741, ranked by callers