MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_verifier_main_recovers_malformed_accuracy_and_goal_judges
Recover malformed accuracy and goal judge responses on retry and record overall score.
tests/tier3/test_judge_failure_artifacts.py:1020
Functiontest_verifier_preserves_rate_limit_error_body_when_retry_after_exceeds_cap
Verify the terminal 429 report retains its provider diagnostic body when Retry-After exceeds max_delay.
tests/test_retry.py:180
Functiontest_verifier_provenance_preserves_relative_hashes_through_source_alias
( monkeypatch, tmp_path, package_root, use_alias )
tests/tier3/test_grader_validation.py:677
Functiontest_verifier_provenance_rejects_files_outside_source_root
(monkeypatch, tmp_path, symlink_escape)
tests/tier3/test_grader_validation.py:728
Functiontest_verifier_provider_env_vars_includes_retry_settings
Verify _VERIFIER_PROVIDER_ENV_VARS allowlist contains all retry configuration keys.
tests/test_harbor_task_toml_retry_env.py:31
Functiontest_verifier_python_keeps_virtualenv_dependencies
Resolving the interpreter symlink loses the venv's installed dependencies.
tests/test_harbor_local_environment.py:347
Functiontest_verifier_refreshes_adc_token_on_401_for_vertex_openapi
Verify verifier template refreshes ADC token in-process upon HTTP 401 when calling Vertex OpenAPI with ADC provenance.
tests/test_tier3_public_runtime.py:1836
Functiontest_verifier_refreshes_adc_token_on_401_via_gcloud_cli
Verify verifier template refreshes ADC token in-process via gcloud CLI fallback upon HTTP 401.
tests/test_tier3_public_runtime.py:1893
Functiontest_verifier_retries_leave_time_to_write_failure_artifacts
Ensure slow transient calls finish before Harbor verifier timeout kills artifact writes.
tests/tier3/test_judge_failure_artifacts.py:303
Functiontest_verifier_retries_non_string_judge_text_before_collector_and_report
Retry non-string judge text payloads before collector and report generation.
tests/tier3/test_judge_failure_artifacts.py:1084
Functiontest_verifier_retries_transient_loopback_statuses_only
(monkeypatch, statuses, calls, ok)
tests/test_transport_rules.py:165
Functiontest_verifier_run_saves_ragas_exchanges_with_the_other_judges
( verifier: ModuleType, monkeypatch: pytest.MonkeyPatch )
tests/tier3/test_ragas_judge_evidence.py:752
Functiontest_verifier_security_uses_staged_workdir
(tmp_path, monkeypatch, configured, native, command, score)
tests/tier3/test_security_workdir_provenance.py:98
Functiontest_version_bump_is_allowed
(tmp_path: Path)
tests/validators/test_version.py:116
Functiontest_version_check_is_in_the_default_tier1_lineup
()
tests/test_commands.py:25
Functiontest_version_compatibility_uses_shared_identity
(evaluation, changed)
tests/test_version_comparison.py:250
Functiontest_version_downgrade_is_rejected
(tmp_path: Path)
tests/validators/test_version.py:125
Methodtest_version_forms_of_blocked_licenses_are_blocked
`+`, `-only`, `-or-later` and deprecated `-with-*-exception` forms must not pass as unknown.
tests/validators/test_license.py:570
Functiontest_version_only_conflict_is_not_reported_as_an_identity_conflict
(tmp_path: Path)
tests/test_openclaw_plugin_discovery.py:143
Functiontest_version_outside_profile_adds_advisory
(tmp_path: Path)
tests/harness/test_overlay.py:1118
Functiontest_version_parses_tool_output
(tmp_path, monkeypatch)
tests/utils/test_tool_runner.py:787
Functiontest_version_probe_executes_canonical_parent_when_path_directory_is_symlinked
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_local_mode.py:4849
Functiontest_version_probe_ignores_the_callers_directory_and_environment
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch )
tests/test_nemo_gym_backend.py:1391
Functiontest_version_reports_pinned_release_for_managed_tools
(tmp_path, monkeypatch)
tests/utils/test_tool_runner.py:701
Functiontest_version_two_digest_has_a_stable_known_answer
(tmp_path: Path)
tests/test_publication_identity.py:66
Functiontest_versioned_interpreter_names_resolve_their_script
(check, command, script)
tests/tier3/test_script_invocation_evidence.py:909
Functiontest_vertex_model_probe_auth_failure
Reject probe with authorization failure and disposition=FATAL on 403.
tests/test_harbor_runtime_preflight.py:3704
Functiontest_vertex_model_probe_missing_project_id
Reject probe with invalid_configuration and disposition=FATAL when project ID is missing.
tests/test_harbor_runtime_preflight.py:3760
Functiontest_vertex_model_probe_missing_token
Inability to obtain access token returns authentication failure, fatal by default, degraded in GKE mode.
tests/test_harbor_runtime_preflight.py:3774
Functiontest_vertex_model_probe_model_not_found
404 on Vertex rawPredict returns model_not_found and disposition=FATAL.
tests/test_harbor_runtime_preflight.py:3732
Functiontest_vertex_model_probe_network_timeout
Network timeout during Vertex probe returns unavailable and disposition=DEGRADED.
tests/test_harbor_runtime_preflight.py:3789
Functiontest_vertex_model_probe_rejects_invalid_identifiers
Reject un-sanitized or traversal characters in GCP region or project ID.
tests/test_harbor_runtime_preflight.py:3845
Functiontest_vertex_model_probe_rejects_invalid_model_id
Reject un-sanitized model ID strings with path separators.
tests/test_harbor_runtime_preflight.py:3862
Functiontest_vertex_model_probe_rejects_redirect
Redirect response during probe is rejected as invalid_configuration.
tests/test_harbor_runtime_preflight.py:3815
Functiontest_vertex_model_probe_success
Verify successful Vertex AI probe returns ok=True and disposition=VERIFIED.
tests/test_harbor_runtime_preflight.py:3679
Functiontest_vertex_model_probe_success_2xx
Verify non-200 2xx responses (e.g. 204) are treated as successful probe.
tests/test_harbor_runtime_preflight.py:3875
Functiontest_vertex_model_probe_unexpected_status
Verify non-2xx status without HTTPError returns other_http failure kind.
tests/test_harbor_runtime_preflight.py:3899
Methodtest_vertex_openapi_does_not_refresh_token_when_explicit_api_key_provided
Do not overwrite user-provided API key with ADC token when 401 occurs.
tests/inference/test_llm_client.py:764
Functiontest_vertex_openapi_probe_authorization_failure_fatal_in_gke
Verify HTTP 403 on Vertex OpenAPI probe remains FATAL in GKE mode.
tests/test_harbor_runtime_preflight.py:4581
Functiontest_vertex_openapi_probe_http_errors
Verify HTTP error classification on Vertex AI OpenAPI probe.
tests/test_harbor_runtime_preflight.py:4473
Functiontest_vertex_openapi_probe_missing_token
Verify probe fails when no token can be acquired.
tests/test_harbor_runtime_preflight.py:4540
Functiontest_vertex_openapi_probe_rejects_redirect
Verify redirect on Vertex AI OpenAPI probe fails fast as invalid configuration.
tests/test_harbor_runtime_preflight.py:4509
Functiontest_vertex_openapi_probe_success_falls_back_to_google_adc
Verify probe acquires token via _get_google_access_token if provider api_key is None.
tests/test_harbor_runtime_preflight.py:4426
Functiontest_vertex_openapi_probe_success_with_api_key
Verify successful 1-token probe returns ok=True and disposition=VERIFIED.
tests/test_harbor_runtime_preflight.py:4380
Functiontest_vertex_openapi_probe_timeout
Verify timeout during Vertex OpenAPI probe returns unavailable and disposition=DEGRADED.
tests/test_harbor_runtime_preflight.py:4559
Methodtest_vertex_openapi_refreshes_token_on_auth_error
Refresh Google access token and retry on 401 AuthenticationError for Vertex OpenAPI endpoints.
tests/inference/test_llm_client.py:733
Functiontest_view_adds_the_case_verdict_to_feedback_saved_before_it
( tmp_path: Path, llm_calls: list[str], fixes: Any, shown_fix: str )
tests/tier3/test_view_results_feedback.py:171
Functiontest_view_ignores_feedback_saved_for_another_run
(tmp_path: Path, monkeypatch: pytest.MonkeyPatch)
tests/tier3/test_view_results_feedback.py:219
Functiontest_view_opens_an_existing_report_without_rebuilding
(tmp_path: Path, monkeypatch: pytest.MonkeyPatch)
tests/tier3/test_view_results_feedback.py:243
Functiontest_view_preserves_each_family_within_byte_limit_for_escape_heavy_text
()
tests/reporting/test_telemetry_view.py:243
Functiontest_view_rebuilds_feedback_when_none_is_saved_for_the_run
( tmp_path: Path, llm_calls: list[str], result: dict[str, Any] )
tests/tier3/test_view_results_feedback.py:206
Functiontest_view_regenerates_missing_report_with_canonical_renderer
(monkeypatch, tmp_path: Path)
tests/reporting/test_unified_tier3_report.py:3157
Functiontest_view_regenerates_only_the_html_report
(monkeypatch, tmp_path: Path)
tests/reporting/test_tier3_completed_agent_docs.py:270
Functiontest_view_renders_multi_line_skip_reason_on_continuation_rows
(capsys)
tests/test_commands.py:3642
Functiontest_view_reuses_saved_feedback_when_report_is_missing
(tmp_path: Path, llm_calls: list[str])
tests/tier3/test_view_results_feedback.py:116
Functiontest_view_writes_rule_based_fixes_when_a_verdict_run_lost_them
The saved feedback still leads the report; only the missing fixes file gets the rule-based text.
tests/tier3/test_view_results_feedback.py:145
Functiontest_visible_artifact_scan_stops_enumerating_at_entry_limit
The limit must bound directory enumeration, not only later stat calls.
tests/test_harbor_runtime_preflight.py:4270
Functiontest_visible_html_text_excludes_embedded_data
(tag)
tests/reporting/test_declarative_coverage.py:52
Functiontest_vulnerable_target_preserves_findings
(tmp_path, monkeypatch)
tests/validators/test_target_dependencies.py:146
Functiontest_wait_for_container_exec_ready_aborts_immediately_on_gke_warden_missing_pod
Translate GKE Warden missing-pod WebSocket AttributeError into an immediate RuntimeError without 60x retry sleep.
tests/test_harbor_gke_environment.py:1453
Functiontest_wait_rechecks_usage_and_only_then_marks_not_run
(monkeypatch)
tests/tier3/test_budget_reservation.py:69
Functiontest_waived_value_does_not_hide_another_value_on_the_same_line
(skill: Path, tmp_path: Path)
tests/validators/test_pii_waivers.py:145
Functiontest_waiver_applies_on_its_expiry_day
A waiver applies through ``expires``; the date is pinned so a run crossing UTC midnight cannot flake.
tests/validators/test_pii_waivers.py:167
Functiontest_waiver_digest_follows_the_value_findings_report
Waivers hash whatever a finding reports, so narrowing the reported value cannot break them.
tests/validators/test_pii_waivers.py:516
Functiontest_waiver_dispositions_are_added_to_json_reports
(skill: Path, tmp_path: Path)
tests/validators/test_pii_waivers.py:532
Functiontest_waiver_dispositions_share_the_sarif_location_of_pii_findings
(skill: Path, tmp_path: Path)
tests/validators/test_pii_waivers.py:387
Functiontest_waiver_file_inside_the_scanned_tree_is_scanned_like_any_other_file
A waiver file kept in the skill ships with it, so its own text is PII-scanned.
tests/validators/test_pii_waivers.py:400
Functiontest_waiver_file_with_chained_merges_is_rejected_without_expanding_them
A merge bomb in a checked-out waiver file must fail fast instead of stalling the run.
tests/validators/test_pii_waivers.py:187
Functiontest_waiver_findings_carry_no_digest
Once a waived value is masked anywhere, a digest in the report would let readers test guesses offline.
tests/validators/test_pii_waivers.py:465
Functiontest_waiver_for_a_deleted_file_in_the_scan_is_stale
(skill: Path, tmp_path: Path)
tests/validators/test_pii_waivers.py:381
Functiontest_waiver_for_a_file_outside_the_scan_is_neither_applied_nor_judged
(skill: Path, tmp_path: Path)
tests/validators/test_pii_waivers.py:136
Functiontest_waiver_for_a_value_the_scan_does_not_report_is_stale
( skill: Path, tmp_path: Path, category: str, content: str, value: str, )
tests/validators/test_pii_waivers.py:420
Functiontest_waiver_stops_applying_when_the_evidence_changes
( skill: Path, tmp_path: Path, content: str | None, overrides: dict, )
tests/validators/test_pii_waivers.py:120
Functiontest_waiver_through_a_symlinked_skill_directory_applies
A waiver applies wherever it is judged, whichever path to the file it names.
tests/validators/test_pii_waivers.py:282
Functiontest_waiver_whose_file_cannot_be_located_is_not_judged
A path the OS rejects (here an embedded NUL) is skipped, never a crash of the scan.
tests/validators/test_pii_waivers.py:305
Functiontest_waiver_window_may_span_a_year
(skill: Path, tmp_path: Path)
tests/validators/test_pii_waivers.py:177
Functiontest_waivers_of_skills_a_folder_run_stopped_before_are_not_judged
(tmp_path: Path)
tests/validators/test_pii_waivers.py:368
Methodtest_walks_the_tree_once_regardless_of_extension_count
(self, skill_with_artifacts, monkeypatch)
tests/validators/test_artifact_dir_exclusion.py:73
Functiontest_wall_time_in_output_is_not_counted
()
tests/tier3/test_trajectory_timing.py:103
Functiontest_wandb_preflight_uses_the_cwsandbox_environment_type
(monkeypatch: pytest.MonkeyPatch)
tests/test_harbor_environment_contract.py:518
Functiontest_wandb_prerequisites_accept_api_key_or_netrc
(monkeypatch: pytest.MonkeyPatch, tmp_path: Path)
tests/test_harbor_environment_contract.py:505
Functiontest_wandb_runs_as_harbor_cwsandbox_with_wandb_auth
()
tests/test_harbor_environment_contract.py:25
Methodtest_warning_does_not_fail
(self)
tests/models/test_result.py:162
Functiontest_warning_findings_are_not_failures
()
tests/reporting/test_junit_reporter.py:376
Methodtest_warning_names_three_rows
(self, tmp_path: Path)
tests/validators/test_security.py:10160
Functiontest_warning_only_report_preserves_pass_and_records_strict_failure
(tmp_path)
tests/validators/test_private_format_boundaries.py:116
Functiontest_warnings_name_the_fields_the_dataset_uses
(tmp_path: Path)
tests/tier3/test_dataset_baseline_fairness.py:821
Functiontest_warnings_name_the_physical_line_after_hash_continuations
(tmp_path, pip_audit)
tests/validators/test_dependency_advisory_findings.py:681
Functiontest_warnings_quote_a_bounded_amount_of_dataset_text
(field, text)
tests/tier3/test_dataset_baseline_fairness.py:2049
Functiontest_wasted_session_input_is_labelled_with_its_text
()
tests/tier3/test_tool_productivity.py:145
Functiontest_weak_tier3_policy_decoy_cannot_waive_canonical_required_evidence
()
tests/reporting/test_advisory_skip_verdict.py:944
Methodtest_weighted_formula
(self)
tests/validators/test_quality_score.py:197
Functiontest_what_a_hidden_command_password_runs_into_stays_masked
(flag: str, text: str, expected: str)
tests/tier3/test_security_finding_evidence.py:3019
Functiontest_what_to_fix_classifies_and_ranks_by_attempts_lost
(tmp_path: Path)
tests/test_tier3_case_summary.py:821
Functiontest_wheel_allowlist_counts_separately_reconstructed_matches_on_one_line
(tmp_path: Path)
tests/test_oss_boundary.py:1112
Functiontest_wheel_allowlist_does_not_strip_an_arbitrary_member_prefix
(tmp_path: Path)
tests/test_oss_boundary.py:1187
Functiontest_wheel_allowlist_rejects_two_same_rule_matches_on_the_anchor_line
(tmp_path: Path)
tests/test_oss_boundary.py:1100
Functiontest_wheel_and_sdist_members_are_scanned_without_extraction
(tmp_path: Path)
tests/test_oss_boundary.py:940
Functiontest_wheel_and_sdist_scan_unknown_text_settings_files
(tmp_path: Path)
tests/test_oss_boundary.py:965
Functiontest_wheel_does_not_require_allowance_for_an_absent_source_path
(tmp_path: Path)
tests/test_oss_boundary.py:1046
← previousnext →24,201–24,300 of 24,741, ranked by callers