MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_gateway_rejection_explains_immutable_image_limitation_without_endpoint
(launch, capsys)
tests/ci/test_run_tier1_image.py:185
Functiontest_gateway_rejects_blank_model_overrides
(gateway, resolver, variable)
tests/test_gateway_defaults.py:82
Functiontest_gateway_separate_embedding_route_keeps_chat_settings
(gateway)
tests/test_gateway_defaults.py:47
Functiontest_gateway_shared_credentials_are_scoped_per_agent
(gateway)
tests/test_gateway_defaults.py:127
Functiontest_gateway_uses_harness_specific_defaults
(gateway, agent, expected)
tests/test_gateway_defaults.py:95
Functiontest_gating_and_claude_only_fields
(tmp_path: Path)
tests/harness/test_overlay.py:693
Functiontest_gating_cannot_disguise_tier3_as_tier1
()
tests/reporting/test_retained_tiers.py:111
Methodtest_gating_tiers_record_each_validator_policy
(self)
tests/reporting/test_reporters.py:663
Functiontest_generate_harbor_tasks_dual_arm_suffix
Verify that generate_harbor_tasks propagates arm suffix into task.toml name.
tests/test_tier3_public_runtime.py:2775
Functiontest_generate_suggestions_prompt_includes_refs_and_uses_larger_budget
(monkeypatch)
tests/tier3/test_suggestion_grounding.py:609
Functiontest_generate_suggestions_structured_returns_objects
(monkeypatch)
tests/tier3/test_suggestion_grounding.py:629
Methodtest_generated_artifacts_not_flagged_as_unexpected
Generated validation/signing artifacts are expected in published skill roots.
tests/validators/test_folder_validation.py:376
Functiontest_generated_baseline_rejects_aliased_authored_skill
(tmp_path: Path)
tests/test_harbor_runtime_skill_isolation.py:2499
Functiontest_generated_baseline_rejects_renamed_target_manifest_payload
(tmp_path: Path)
tests/test_harbor_runtime_skill_isolation.py:2514
Methodtest_generated_benchmark_report_duplicate_does_not_produce_finding
( self, mock_embed, _mock_llm, mock_analyze, tmp_path: Path )
tests/deduplication/intra_skill/test_intra_skill_validator.py:1043
Functiontest_generated_benchmark_reports_baseline
(tmp_path: Path, baseline: dict)
tests/test_baseline_store.py:291
Functiontest_generated_call_public_llm_does_not_redact_successful_content
( verifier_module, monkeypatch: pytest.MonkeyPatch, )
tests/test_verifier_error_redaction.py:320
Functiontest_generated_call_public_llm_exhausts_retries_on_persistent_429
Verify call_public_llm exhausts retries and returns redacted error on persistent 429.
tests/test_verifier_error_redaction.py:480
Functiontest_generated_call_public_llm_fails_fast_if_retry_after_exceeds_max_delay
Verify call_public_llm fails fast without sleeping if Retry-After exceeds max_delay.
tests/test_verifier_error_redaction.py:582
Functiontest_generated_call_public_llm_fails_fast_on_http_401
Verify call_public_llm fails fast immediately on HTTP 401 Unauthorized without retry or sleep.
tests/test_verifier_error_redaction.py:553
Functiontest_generated_call_public_llm_preserves_fallback_detection_and_redacts_exhausted_error
( verifier_module, monkeypatch: pytest.MonkeyPatch, )
tests/test_verifier_error_redaction.py:285
Functiontest_generated_call_public_llm_redacts_generic_exception
( verifier_module, monkeypatch: pytest.MonkeyPatch, )
tests/test_verifier_error_redaction.py:207
Functiontest_generated_call_public_llm_redacts_http_error_body
( verifier_module, monkeypatch: pytest.MonkeyPatch, )
tests/test_verifier_error_redaction.py:185
Functiontest_generated_call_public_llm_redacts_provider_returned_errors
( verifier_module, monkeypatch: pytest.MonkeyPatch, provider: str, credential_name: str, p
tests/test_verifier_error_redaction.py:262
Functiontest_generated_call_public_llm_respects_retry_after_header
Verify call_public_llm respects Retry-After header on HTTP 429.
tests/test_verifier_error_redaction.py:512
Functiontest_generated_call_public_llm_retries_transient_http_error_and_succeeds
Verify call_public_llm retries on transient HTTP status codes and succeeds on subsequent attempt.
tests/test_verifier_error_redaction.py:447
Functiontest_generated_cases_expect_the_entry_point_not_the_helper
(generate: Any)
tests/tier3/test_dataset_primary_script.py:61
Functiontest_generated_custom_grader_mixed_metric_surfaces_match_collector_without_credential_drift
( tmp_path: Path, mode: str, )
tests/tier3/test_judge_failure_artifacts.py:1793
Functiontest_generated_custom_grader_omits_unsafe_dict_valued_metric_from_every_artifact
( tmp_path: Path, mode: str, )
tests/tier3/test_judge_failure_artifacts.py:1703
Functiontest_generated_custom_grader_rejects_metric_overflow_before_retaining_raw_names
( tmp_path: Path, mode: str, )
tests/tier3/test_judge_failure_artifacts.py:1642
Functiontest_generated_custom_only_script_accepts_overall_only_custom_reward
Accept overall-only reward payloads from custom grading scripts.
tests/tier3/test_judge_failure_artifacts.py:1599
Functiontest_generated_custom_only_script_rejects_invalid_overall_score
( tmp_path: Path, source: str, invalid_score: str, )
tests/tier3/test_judge_failure_artifacts.py:1885
Methodtest_generated_directory_case_alias_follows_filesystem_semantics
(self, tmp_path: Path)
tests/validators/test_artifact_dir_exclusion.py:551
Functiontest_generated_eval_output_is_still_classified_as_generated
(tmp_path: Path, eval_dir: str, output_dir: str)
tests/validators/test_security_authored_evals.py:124
Functiontest_generated_exclusions_apply_only_to_the_expected_node_kind
(tmp_path: Path)
tests/test_publication_identity.py:234
Methodtest_generated_file_case_alias_follows_filesystem_semantics
( self, tmp_path: Path, absolute_location: bool, )
tests/validators/test_artifact_dir_exclusion.py:531
Methodtest_generated_file_case_alias_follows_filesystem_semantics
(self, tmp_path: Path)
tests/validators/test_base.py:447
Methodtest_generated_file_case_alias_follows_filesystem_semantics
(self, skill_root: Path)
tests/deduplication/utils/test_skill_collector.py:921
Functiontest_generated_github_caller_grants_every_reusable_job_permission
(repository, all_release)
tests/test_ci_onboarding_all_tiers.py:628
Functiontest_generated_gitlab_root_is_accepted_by_trusted_feedback_and_has_root_rules
(repository, all_release)
tests/test_ci_onboarding_all_tiers.py:641
Functiontest_generated_images_satisfy_docker_agent_setup_dependencies
Harbor's agent setup apt-installs, in every trial, each prerequisite the image lacks.
tests/tier3/test_installed_agent_bootstrap.py:427
Functiontest_generated_main_image_replaces_authored_healthcheck_with_constant_readiness
( tmp_path: Path, image_mode: str )
tests/test_harbor_compose_readiness.py:26
Functiontest_generated_markdown_tables_parse_with_github_table_rules
(tmp_path)
tests/ci/test_report_views.py:353
Functiontest_generated_marker_is_bound_to_its_original_destination
(tmp_path: Path)
tests/test_harbor_output_provenance.py:473
Functiontest_generated_mcp_header_normalization_keeps_escaped_and_malformed_templates_rejected
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, header: str )
tests/test_harbor_runtime_skill_isolation.py:719
Functiontest_generated_mcp_headers_parse_in_pinned_harbor_without_changing_rich_json
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, header: str )
tests/test_harbor_runtime_skill_isolation.py:625
Methodtest_generated_name_outside_source_root_is_not_filtered
(self, tmp_path: Path)
tests/validators/test_artifact_dir_exclusion.py:567
Functiontest_generated_output_cleanup_refuses_symlink_escape
(tmp_path: Path)
tests/test_issue55_collector_truth.py:3693
Functiontest_generated_pass_card_encodes_commas_in_agent_identity
( tmp_path: Path, field: str, encoded_comma: bool, )
tests/test_public_benchmark_gate.py:990
Functiontest_generated_provider_error_preserves_selected_judge_provenance
( verifier_module, monkeypatch: pytest.MonkeyPatch, )
tests/test_verifier_error_redaction.py:231
Functiontest_generated_rejects_output_root_inside_evaluator_source
(tmp_path: Path, source_name: str)
tests/test_harbor_runtime_skill_isolation.py:1456
Functiontest_generated_results_do_not_invalidate_source_tree_identity
(tmp_path: Path, authored_evals_dir: bool)
tests/tier3/test_plugin_prepare_review.py:233
Functiontest_generated_results_roots_are_accepted
(tmp_path: Path, results: str | None)
tests/tier3/test_private_runtime_regressions.py:195
Methodtest_generated_skill_card_duplicate_does_not_produce_finding
( self, mock_embed, _mock_llm, mock_analyze, tmp_path: Path )
tests/deduplication/intra_skill/test_intra_skill_validator.py:1013
Functiontest_generated_standard_grading_scripts_stop_after_evaluator_failure
Halt generated grading scripts immediately when standard evaluator fails.
tests/tier3/test_judge_failure_artifacts.py:1576
Functiontest_generated_task_copies_explicit_ref_without_shared_files_directory
(tmp_path: Path, base_image: str)
tests/test_harbor_input_staging.py:369
Functiontest_generated_task_dockerfile_installs_declared_apt_packages
( tmp_path: Path, apt_packages: str | None, expected: list[str] )
tests/test_harbor_apt_install_lines.py:75
Functiontest_generated_task_keeps_evaluator_provider_variables_out_of_agent_environment
(tmp_path)
tests/test_tier3_public_runtime.py:943
Functiontest_generated_task_rebases_custom_dockerfile_content
(tmp_path: Path)
tests/test_harbor_adapter_safety.py:293
Functiontest_generated_task_stages_public_provider_variables_for_the_verifier
(tmp_path)
tests/test_tier3_public_runtime.py:909
Functiontest_generated_task_verifier_timeout_covers_all_structured_judge_attempts
(tmp_path)
tests/test_tier3_public_runtime.py:928
Functiontest_generated_tasks_accept_distinct_portable_ids
(tmp_path: Path)
tests/test_harbor_adapter_safety.py:284
Functiontest_generated_tasks_apply_per_entry_input_isolation
(tmp_path: Path)
tests/test_harbor_input_staging.py:274
Functiontest_generated_tasks_disable_claude_code_background_tasks
(tmp_path: Path, with_skill: bool)
tests/test_harbor_claude_background_tasks.py:49
Functiontest_generated_tasks_disable_claude_code_refusal_fallback
(tmp_path: Path, with_skill: bool)
tests/test_harbor_claude_refusal_fallback.py:138
Functiontest_generated_tasks_project_runtime_skills_without_root_evals
(tmp_path: Path)
tests/test_harbor_runtime_skill_isolation.py:304
Functiontest_generated_tasks_reject_portable_normalization_collisions_before_removal
(tmp_path: Path)
tests/test_harbor_adapter_safety.py:253
Functiontest_generated_tasks_reject_resolved_path_escape_before_removal
(tmp_path: Path)
tests/test_harbor_adapter_safety.py:266
Functiontest_generated_tasks_reject_runtime_process_loader_environment
(tmp_path: Path, name: str)
tests/test_harbor_runtime_skill_isolation.py:3454
Functiontest_generated_tasks_reject_runtime_skill_discovery_environment
(tmp_path: Path, name: str)
tests/test_harbor_runtime_skill_isolation.py:2670
Functiontest_generated_tasks_reject_unsafe_agent_workdir
(tmp_path: Path, agent_workdir: str)
tests/test_harbor_runtime_skill_isolation.py:2626
Functiontest_generated_tasks_reject_unsafe_ids_before_removing_existing_tasks
( tmp_path: Path, unsafe_id: str, )
tests/test_harbor_adapter_safety.py:237
Functiontest_generated_tasks_reject_workdir_inside_runtime_projection
(tmp_path: Path, agent_workdir: str)
tests/test_harbor_runtime_skill_isolation.py:2650
Functiontest_generated_tasks_stage_only_names_and_placeholders
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_secure_docker_environment.py:1601
Functiontest_generated_verifier_builds_exactly_one_anthropic_native_messages_path
( monkeypatch: pytest.MonkeyPatch, base_url: str, expected_url: str, )
tests/test_tier3_public_runtime.py:1552
Functiontest_generated_verifier_rejects_non_http_provider_base_urls
(monkeypatch)
tests/test_tier3_public_runtime.py:1501
Functiontest_generated_verifier_rejects_unsafe_anthropic_api_roots_without_echoing_credentials
( monkeypatch: pytest.MonkeyPatch, variable: str, base_url: str, )
tests/test_tier3_public_runtime.py:1605
Functiontest_generated_verifier_uses_the_official_anthropic_messages_url_when_base_is_unset
( monkeypatch: pytest.MonkeyPatch, )
tests/test_tier3_public_runtime.py:1625
Methodtest_generated_wrapper_skill_is_not_a_selection
(self)
tests/tier3/test_plugin_signals.py:661
Functiontest_generation_announces_provider_and_handoff_on_stderr
(generation_skill)
tests/tier3/test_dataset_generation_progress.py:42
Functiontest_generation_uses_existing_deterministic_fallback_on_model_failure
(skill, configured, monkeypatch)
tests/tier3/test_workflow.py:348
Functiontest_generation_uses_one_private_evals_snapshot
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_harbor_input_staging.py:306
Functiontest_generic_atif_exec_call_with_input_is_unchanged
(source, extractor)
tests/tier3/test_codex_tool_call_normalization.py:360
Functiontest_generic_clean_tier3_skip_respects_publication_policy
( tmp_path: Path, tier3_required: bool, overall_status: str, tier3_status: str, )
tests/golden/test_benchmark_card.py:962
Functiontest_generic_optional_metadata_cannot_waive_partial_required_tier
(tier: str)
tests/reporting/test_advisory_skip_verdict.py:611
Functiontest_generic_optional_metadata_cannot_waive_required_skipped_tier2
()
tests/golden/test_benchmark_card.py:595
Functiontest_generic_optional_metadata_cannot_waive_skipped_tier1
()
tests/golden/test_benchmark_card.py:608
Functiontest_generic_public_and_defensive_text_is_not_flagged
(public_text: str)
tests/test_oss_boundary.py:110
Functiontest_generic_report_sections_match_executed_tier_combinations
(tmp_path: Path)
tests/reporting/test_unified_tier3_report.py:3196
Functiontest_generic_secrets_are_only_exposed_to_selected_routes
(repository, routing_release, args, expected)
tests/test_ci_onboarding_model_routing.py:173
Methodtest_generic_service_account_identity_is_detected
A generic service-account email and matching home path are detected.
tests/validators/test_security.py:1220
Functiontest_generic_tier3_skip_metadata_cannot_hide_payload_truth
( payload_verdict: str, summary_verdict: str, execution_status: str, expected_status: str, )
tests/reporting/test_advisory_skip_verdict.py:577
Functiontest_generic_truncated_parser_can_keep_a_complete_empty_result
(monkeypatch: pytest.MonkeyPatch)
tests/inference/test_llm_client.py:1809
Functiontest_genuine_invocations_keep_full_credit
(check, command)
tests/tier3/test_script_invocation_evidence.py:138
Methodtest_get_install_hint_for_known_tools
Known tools return specific install hints.
tests/utils/test_tool_runner.py:405
Methodtest_get_install_hint_for_unknown_tools
Unknown tools return generic hint.
tests/utils/test_tool_runner.py:455
Methodtest_get_name_from_supported_path_shapes
Directories, manifest files, and nested dirs all use the skill directory name.
tests/utils/test_helpers.py:91
Methodtest_git_init_nested_below_host_home_uses_metadata_only_ancestor_traversal
( self, strict_reads: bool, )
tests/test_harbor_local_sandbox.py:1553
Functiontest_git_package_on_a_moving_ref_is_remote_code
A hook package from git is code the plugin does not ship, also when its ref is a moving branch. The hook check reads the pin's ``remote`` field,
tests/validators/test_plugin_component_risk.py:482
Functiontest_git_spec_on_a_moving_branch_is_floating
(config, floating: bool)
tests/validators/test_mcp_static.py:271
← previousnext →16,601–16,700 of 24,741, ranked by callers