MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_gitlab_same_project_mr_still_selects_enriched_tier1
()
tests/ci/test_gateway_benchmark_workflows.py:111
Functiontest_gitlab_save_and_pr_refuse_to_replace_existing_root_config
(repository, all_release, monkeypatch, mode)
tests/test_ci_onboarding_all_tiers.py:1182
Functiontest_gitlab_selfhosted_save_is_two_files_and_no_global_auth
(repository, all_release)
tests/test_ci_onboarding_all_tiers.py:229
Functiontest_gitlab_sends_note_without_the_final_lf_removed_by_server
(gitlab, monkeypatch, action)
tests/ci/test_feedback.py:697
Functiontest_gitlab_setup_selector_uses_pinned_safe_git_reader
()
tests/ci/test_all_tier_workflows.py:419
Functiontest_gitlab_shared_native_timeout_is_a_literal_duration
()
tests/ci/test_all_tier_workflows.py:411
Functiontest_gitlab_signed_job_claims_are_not_replaced_by_ci_environment
(monkeypatch)
tests/ci/test_feedback.py:1314
Functiontest_gitlab_status_accepts_generated_root_and_filters_default_branch
(repository, all_release, monkeypatch)
tests/test_ci_onboarding_all_tiers.py:921
Functiontest_gitlab_status_recognizes_tier2_job_overrides
(repository, all_release, monkeypatch)
tests/test_ci_onboarding_all_tiers.py:1234
Functiontest_gitlab_status_reports_admission_version_prerequisite
(repository, monkeypatch, version, phrase)
tests/test_ci_onboarding_all_tiers.py:802
Functiontest_gitlab_status_requires_trusted_all_tier_root_config
(repository, monkeypatch)
tests/test_ci_onboarding_all_tiers.py:914
Functiontest_gitlab_status_warns_about_root_jobs_without_feedback_guard
(repository, all_release, monkeypatch)
tests/test_ci_onboarding_all_tiers.py:1209
Functiontest_gitlab_target_fetch_preserves_existing_diff_base_ancestry
(tmp_path)
tests/ci/test_prepare_security_baselines.py:59
Functiontest_gitlab_template_ignores_the_outer_ci_environment
(tmp_path, monkeypatch)
tests/ci/test_gitlab_template.py:151
Functiontest_gitlab_tier1_and_native_tiers_select_the_same_skills
(tmp_path)
tests/ci/test_all_tier_workflows.py:577
Functiontest_gitlab_tier1_runs_only_digest_pinned_legacy_image
()
tests/ci/test_all_tier_workflows.py:147
Functiontest_gitlab_tier2_has_no_docker_service_or_tls_settings
()
tests/ci/test_all_tier_workflows.py:272
Functiontest_gitlab_token_api_rejects_host_mismatch_before_network
(monkeypatch)
tests/test_ci_onboarding_all_tiers.py:373
Functiontest_gitlab_transitive_config_sources_cannot_be_mutable
(include)
tests/ci/test_feedback.py:1366
Functiontest_gitlab_uses_persisted_graphql_mr_without_unsupported_pipeline_rest_endpoint
(gitlab)
tests/ci/test_feedback.py:866
Functiontest_gitlab_variable_scalars_are_explicit_strings_for_gitlab_yaml_parser
(repository, all_release)
tests/test_ci_onboarding_all_tiers.py:744
Functiontest_gitleaks_curl_auth_allowlists_are_exactly_scoped
()
tests/test_ci_workflows.py:414
Methodtest_gitleaks_git_history_keeps_resolved_source
(self, mock_run, malicious_skill_path)
tests/validators/test_scanner_path_resolution.py:135
Methodtest_gitleaks_malformed_findings_report_is_incomplete
(self, _mock_run, skill_dir)
tests/validators/test_scan_incomplete.py:114
Methodtest_gitleaks_no_git_direct_file_uses_safe_relative_source
( self, mock_run, tmp_path, ancestor_name )
tests/validators/test_scanner_path_resolution.py:118
Methodtest_gitleaks_no_git_does_not_expose_allowlisted_ancestors
( self, mock_run, tmp_path, ancestor_name )
tests/validators/test_scanner_path_resolution.py:102
Methodtest_gitleaks_no_git_uses_dot_source_from_resolved_skill_dir
(self, mock_run, malicious_skill_path)
tests/validators/test_scanner_path_resolution.py:88
Functiontest_gitleaks_single_value_allowlists_are_exactly_scoped
()
tests/test_ci_workflows.py:373
Methodtest_gitleaks_timeout_marks_incomplete
(self, _mock_run, skill_dir)
tests/validators/test_scan_incomplete.py:58
Functiontest_gitleaks_uses_event_specific_full_history_scopes
()
tests/test_ci_workflows.py:432
Functiontest_given_models_are_rendered_without_rereading_or_changing_them
(tmp_path, monkeypatch)
tests/ci/test_report_views.py:565
Functiontest_gke_environment_create_pod_autopilot_storage_idempotency_and_custom_probe_image
Scope 10Gi->10176Mi storage adjustment to non-accelerator Autopilot pods, preserve >10Gi requests, and avoid duplicates on retry.
tests/test_harbor_gke_environment.py:488
Functiontest_gke_environment_create_pod_blocks_metadata_host_and_bound_ksa_when_unprivileged
Enforce metadata NetworkPolicy, trusted init + companion probe containers, token automount=False, and reject bound KSAs unless opted in.
tests/test_harbor_gke_environment.py:178
Functiontest_gke_environment_create_pod_fails_closed_on_service_account_api_error
Fail closed when ServiceAccount inspection raises a Kubernetes API error without Workload Identity opt-in.
tests/test_harbor_gke_environment.py:121
Functiontest_gke_environment_create_pod_rejects_weakened_existing_network_policy
Reject pod creation when an existing or concurrently created (409) NetworkPolicy does not block both GKE metadata IPs.
tests/test_harbor_gke_environment.py:319
Functiontest_gke_environment_exec_adc_token_refresh_and_isolation
Refresh Vertex ADC tokens only when OPENAI_API_KEY is already active in the target execution environment.
tests/test_harbor_gke_environment.py:1341
Functiontest_gke_environment_exec_handles_harbor_024_stream_closed_and_pod_not_found
Raise GKEExecStreamClosedError without replaying the command, redact it, and raise GKEPodNotFoundError on Warden missing pod.
tests/test_harbor_gke_environment.py:1515
Functiontest_gke_environment_exec_never_replays_dispatched_command
Drive Harbor's real GKE exec so a lost completion status after the side effect raises after one dispatch.
tests/test_harbor_gke_environment.py:1128
Functiontest_gke_environment_exec_retries_transient_kubelet_errors
Retry transient kubelet 10250 errors raised before dispatch, but do not retry regular command failures.
tests/test_harbor_gke_environment.py:1039
Functiontest_gke_environment_preflight_normalizes_multipath_kubeconfig
Accept split KUBECONFIG in SkillEvaluatorGKEEnvironment.preflight() and raise SystemExit when missing.
tests/test_harbor_gke_environment.py:1494
Functiontest_gke_environment_seam_routes_untargeted_exec_to_main_and_probes_trusted_sidecar
Verify Harbor untargeted exec calls route to 'main' while metadata isolation verification runs in 'skillevaluator-metadata-probe'.
tests/test_harbor_gke_environment.py:914
Functiontest_gke_environment_wait_for_container_exec_ready_drains_stream_and_retries
Drain the Kubernetes WSClient stream, retry readiness, and enforce the in-pod metadata isolation probe in the trusted probe container.
tests/test_harbor_gke_environment.py:676
Functiontest_gke_environment_wait_for_pod_ready_fails_closed_on_init_probe_failure
Inspect init_container_statuses during _wait_for_pod_ready so a failed pre-start probe deletes the pod immediately.
tests/test_harbor_gke_environment.py:594
Functiontest_gke_mode_accepts_vertex_ai_without_anthropic_api_key
Accept Claude Code on Vertex AI in GKE mode with zero Anthropic API keys when Workload Identity is enabled.
tests/test_harbor_gke_mode.py:335
Functiontest_gke_mode_rejects_claude_code_without_vertex_or_anthropic_key
Reject Claude Code in GKE mode if neither Anthropic key nor Vertex flag is set.
tests/test_harbor_gke_mode.py:351
Functiontest_gke_mode_rejects_vertex_ai_without_project_id
Reject Claude Code with Vertex AI if no project ID is configured.
tests/test_harbor_gke_mode.py:466
Functiontest_gke_mode_suggests_vertex_suffix_when_anthropic_key_missing
Include Vertex AI suggestion suffix in GKE mode when Anthropic key is missing.
tests/test_harbor_gke_mode.py:393
Functiontest_gke_required_kwargs_contains_expected_keys
Verify all 5 required GKE kwargs are defined.
tests/test_harbor_gke_mode.py:56
Functiontest_gke_runtime_preflight_forwards_environment_kwargs
Forward environment kwargs in GKE runtime preflight so --ek flags are emitted.
tests/test_harbor_gke_mode.py:423
Functiontest_gke_workflow_resolves_env_vars_and_enforces_precedence_before_preflight
Resolve SKILLEVALUATOR_GKE_* env vars with CLI > env precedence before workflow preflight.
tests/tier3/test_workflow.py:696
Functiontest_gke_workload_identity_fails_closed_by_default_and_allows_opt_in
Reject GKE Vertex Workload Identity unless SKILLEVALUATOR_GKE_ALLOW_WORKLOAD_IDENTITY=1 or --ek allow_workload_identity=true.
tests/test_harbor_gke_mode.py:1046
Methodtest_glob_matcher_covers_what_a_shell_matches
A bracket expression may match any one character, and a pattern longer than every name matches none.
tests/validators/test_security.py:11043
Functiontest_global_report_budget_prioritizes_best_agent_details
()
tests/reporting/test_unified_tier3_report.py:2273
Functiontest_global_search_reveals_dataset_attempt_body_matches_and_preserves_user_disclosures
(page, reports)
tests/reporting/test_report_browser.py:1313
Methodtest_globs_accepted
Test that globs field is accepted for Rules (unlike Skills).
tests/validators/test_rules_schema.py:295
Methodtest_globs_and_bare_directories_in_code_reach_the_files
(self, tmp_path: Path, skill_text: str)
tests/validators/test_skillspector_scope.py:1308
Methodtest_globs_in_refs
(self)
tests/tier3/test_plugin_signals.py:903
Methodtest_globs_is_optional
Test that globs field is optional for Rules.
tests/validators/test_rules_schema.py:281
Functiontest_glue_reads_cross_component_cases_on_the_five_dimension_basis
Glue that only shows in routing (discoverability) is 0 on the common basis but not in the glue verdict.
tests/tier3/test_private_ladder_stats.py:247
Functiontest_glued_id_or_hash_is_not_redacted
()
tests/tier3/test_secret_redaction_patterns.py:61
Functiontest_glued_id_or_hash_tokens_are_not_secrets
(token)
tests/tier3/test_checks_secret_patterns.py:296
Functiontest_goal_accepts_unique_object_before_unmatched_trailing_prose_without_retry
( judge_module, monkeypatch, )
tests/tier3/test_judge_result_semantics.py:850
Functiontest_goal_accuracy_custom_judges_end_state_evidence
(monkeypatch)
tests/tier3/test_judge_uses_evidence.py:23
Functiontest_goal_accuracy_keeps_evidence_past_old_3000_cutoff
(monkeypatch)
tests/tier3/test_judge_uses_evidence.py:58
Functiontest_goal_count_includes_passes_in_a_case_with_other_failed_attempts
(tmp_path)
tests/reporting/test_case_findings_report.py:97
Functiontest_goal_flags_keep_agent_identity_when_trial_names_repeat
(tmp_path)
tests/reporting/test_case_findings_report.py:75
Functiontest_goal_initial_provider_error_does_not_retry
(judge_module, monkeypatch)
tests/tier3/test_judge_result_semantics.py:468
Functiontest_goal_invalid_twice_errors_after_exactly_one_retry
(judge_module, monkeypatch)
tests/tier3/test_judge_result_semantics.py:445
Functiontest_goal_met_or_unscored_passes_are_not_flagged
(tmp_path: Path)
tests/test_tier3_case_summary.py:1405
Functiontest_goal_normalizes_and_bounds_text_fields
(judge_module, monkeypatch)
tests/tier3/test_judge_result_semantics.py:834
Functiontest_goal_not_met_fails_without_changing_overall_score
()
tests/test_tier3_attempt_verdict.py:57
Functiontest_goal_not_met_heading_counts_attempts_that_missed_the_goal
()
tests/test_tier3_case_summary.py:2126
Functiontest_goal_not_met_uses_the_canned_fix_only_when_the_agent_stopped_for_input
(tmp_path: Path)
tests/test_tier3_case_summary.py:2075
Functiontest_goal_prompt_omits_empty_tool_section_and_keeps_real_one
(judge_module, monkeypatch)
tests/tier3/test_judge_contracts.py:290
Functiontest_goal_rejects_nonstandard_constant_in_metadata
(judge_module, monkeypatch)
tests/tier3/test_judge_result_semantics.py:874
Functiontest_goal_rejects_unscorable_response_shapes
(judge_module, monkeypatch, content: str)
tests/tier3/test_judge_result_semantics.py:800
Functiontest_goal_retries_invalid_response_once_and_recovers
( judge_module, monkeypatch, invalid: str, )
tests/tier3/test_judge_result_semantics.py:371
Functiontest_goal_retries_non_string_text_fields_once_and_recovers
( judge_module, monkeypatch, field: str, )
tests/tier3/test_judge_result_semantics.py:403
Functiontest_goal_retry_provider_error_is_bounded_redacted_and_uses_retry_provenance
( judge_module, monkeypatch, )
tests/tier3/test_judge_result_semantics.py:484
Functiontest_goal_score_follows_achieved_not_a_contradicting_score
(judge_module, monkeypatch)
tests/tier3/test_judge_contracts.py:317
Functiontest_goal_score_is_binary_from_achieved_and_ignores_reported_score
( judge_module, monkeypatch, payload: dict, expected: float, )
tests/tier3/test_judge_result_semantics.py:820
Functiontest_good_answers_succeed_as_plain_text_and_in_an_mcp_text_block
()
tests/tier3/test_f90_mcp_failure_text.py:72
Functiontest_good_answers_that_talk_about_errors_succeed_in_claude_shape
()
tests/tier3/test_f90_mcp_failure_text.py:60
Functiontest_good_answers_that_talk_about_errors_succeed_in_codex_shape
()
tests/tier3/test_f90_mcp_failure_text.py:66
Methodtest_gpl_frontmatter_is_not_hidden_by_mit_license_file
Blocked frontmatter must still fail even if the LICENSE file is allowed.
tests/validators/test_license.py:402
Methodtest_gpl_license_file_without_frontmatter_is_blocked
LICENSE-only GPL remains blocked when frontmatter has no license field.
tests/validators/test_license.py:393
Methodtest_gps_coordinates_with_geospatial_context_still_flagged
A geospatial cue on the line, two lines above, or in the file name keeps the finding.
tests/validators/test_security.py:7605
Methodtest_gps_deeply_nested_geojson_inner_position_flagged
Nested coordinates arrays are merged before marking lines, and the inner position stays cued.
tests/validators/test_security.py:7672
Methodtest_gps_display_variable_not_flagged
X11 DISPLAY variables like :0.0 should not be flagged.
tests/validators/test_security.py:7440
Methodtest_gps_labels_are_applied_per_pair
(self, tmp_path: Path, line: str)
tests/validators/test_security.py:7513
Methodtest_gps_low_precision_pair_not_flagged
Low-precision decimal pairs in API/reference docs are not GPS coordinates. Regression test for the reported false positives in ``refe
tests/validators/test_security.py:7450
Methodtest_gps_pretty_printed_geojson_geometry_positions_flagged
Every position inside a GeoJSON geometry's coordinates array is cued, however deep it is nested.
tests/validators/test_security.py:7661
Methodtest_gps_real_coordinates_still_flagged
Real GPS coordinates should still be flagged.
tests/validators/test_security.py:7428
Methodtest_gps_set_geo_command_not_flagged
CLI geolocation commands should not be flagged as GPS data.
tests/validators/test_security.py:7418
Methodtest_gps_shape_requires_geospatial_context
Precise vectors and unlabeled decimal pairs are not location PII without a geospatial cue.
tests/validators/test_security.py:7482
Methodtest_gps_technical_context_outweighs_soft_location_cue
Technical coordinate labels win over location, office, or address wording.
tests/validators/test_security.py:7540
Methodtest_gps_technical_word_after_pair_does_not_hide_location_label
A technical word that follows the pair does not label it, so the location cue still counts.
tests/validators/test_security.py:7498
Methodtest_gps_zero_value_variable_assignment_not_flagged
Variable assignments with 0.0, 0.0 should not be flagged as GPS.
tests/validators/test_security.py:7408
Functiontest_gpt5_family_rejects_custom_temperature
Verify gpt-5 models do not support custom temperature.
tests/test_llm_judge.py:57
Functiontest_gpt6_judge_payload_sends_judge_reasoning_effort
(model: str)
tests/test_llm_judge.py:95
← previousnext →16,801–16,900 of 24,741, ranked by callers