MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_catalog_records_unreadable_linked_groups
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, workers: int )
tests/test_catalog_discovery.py:358
Methodtest_catalog_rejects_target_without_root_skill
( self, _mock_client_cls, mock_registry_cls, mock_extract, tmp_path: Path )
tests/validators/test_similarity.py:403
Functiontest_catalog_rejects_unsafe_base_urls
(monkeypatch, base_url: str, expected_kind: str)
tests/test_model_catalog.py:644
Functiontest_catalog_reports_shared_metadata_id_and_blocks_under_registry
(tmp_path: Path)
tests/golden/test_policy_behavior.py:143
Functiontest_catalog_root_spellings_are_matched_by_file_identity
(tmp_path: Path)
tests/tier3/test_decoys.py:298
Functiontest_catalog_run_strips_terminal_controls_from_skill_names_and_reasons
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch )
tests/reporting/test_rich_markup_escaping.py:739
Functiontest_catalog_selection_is_bounded_and_validates_limit
()
tests/test_model_catalog.py:936
Functiontest_catalog_selection_uses_configured_model_and_filters_non_chat_candidates
()
tests/test_model_catalog.py:918
Functiontest_catalog_skill_entry_skips_stale_json_without_report_name
(tmp_path: Path)
tests/test_commands.py:2430
Functiontest_catalog_status_and_header_trickles_cannot_extend_the_wall_clock_deadline
( immediate: bytes, trickled: bytes, remainder: bytes, )
tests/test_model_catalog.py:310
Methodtest_catalog_still_rejects_unsafe_text_on_load_and_save
( self, tmp_path: Path, field: str, character: str )
tests/embedding/test_catalog.py:163
Functiontest_catalog_stops_at_skill_boundary
(tmp_path: Path, workers: int, filename: str)
tests/test_catalog_discovery.py:161
Functiontest_catalog_summary_binds_exact_json_report_not_sarif_sidecar
(monkeypatch)
tests/test_commands.py:2280
Functiontest_catalog_summary_same_second_rerun_overwrites_json
(monkeypatch)
tests/test_commands.py:2330
Methodtest_catalog_target_with_unusable_frontmatter_explains_the_frontmatter_rule
( self, _mock_client_cls, mock_registry_cls, tmp_path: Path, frontmatter: str )
tests/validators/test_similarity.py:426
Functiontest_catalog_transport_rejects_http_when_allowlist_changes_after_validation
( monkeypatch, allowlist: str | None, lookup: str )
tests/test_model_catalog.py:730
Functiontest_catalog_transport_rejects_unapproved_http_before_building_opener
(monkeypatch, allowlist: str)
tests/test_model_catalog.py:802
Methodtest_catalog_tree_evidence_reads_a_bounded_sample
(self, tmp_path: Path, write_skill, monkeypatch)
tests/embedding/test_catalog.py:784
Methodtest_catalog_tree_missing_some_first_sorted_skills_still_skips
(self, tmp_path: Path, write_skill)
tests/embedding/test_catalog.py:813
Methodtest_catalog_tree_with_a_minority_of_edited_skills_still_skips
(self, tmp_path: Path, write_skill)
tests/embedding/test_catalog.py:746
Functiontest_catalog_validate_reports_each_waiver_only_in_its_own_skill
One waiver file serves the catalog; no child may report its siblings' waivers as stale. A symlinked waiver file keeps the link's directory as the
tests/test_commands.py:1215
Functiontest_catalog_with_repeated_leaf_names_keeps_identities_distinct
(tmp_path: Path)
tests/reporting/test_junit_reporter.py:717
Methodtest_catalog_without_another_skill_entry_never_skips
(self, tmp_path: Path, write_skill)
tests/embedding/test_catalog.py:758
Functiontest_catalog_without_shared_ids_records_no_conflicts
(tmp_path: Path)
tests/golden/test_policy_behavior.py:207
Functiontest_category_observed_coverage_distinguishes_correct_expected_abstentions
(label, observed)
tests/tier3/test_grader_policy.py:269
Methodtest_caveats_are_not_a_2_13_3_reference_caveat
The 2.13.3 reference-caveat rule reads only ``partial`` reports, so caveats always need a known target. In a direct scan without a reference
tests/validators/test_security.py:11597
Methodtest_caveats_keep_the_findings
Accepted caveats do not hide a finding: the scan fails on it and is complete.
tests/validators/test_security.py:11712
Methodtest_caveats_must_all_be_missing_references
(self, tmp_path: Path)
tests/validators/test_security.py:11630
Methodtest_caveats_need_a_scan_root
(self)
tests/validators/test_security.py:11584
Methodtest_caveats_report_completes_the_llm_stage
(self, mock_tools, tmp_path: Path)
tests/validators/test_security.py:11244
Methodtest_caveats_report_completes_the_scan
(self, mock_tools, tmp_path: Path, version: str)
tests/validators/test_security.py:11229
Methodtest_caveats_status_before_2_13_4_is_not_recognized
(self, mock_tools, tmp_path: Path)
tests/validators/test_security.py:11265
Methodtest_caveats_that_contradict_the_status_drop_the_report
SkillSpector never gives this status to another gap, so such a report is not trusted at all.
tests/validators/test_security.py:11842
Functiontest_cd_option_before_a_project_directory_still_reports_the_relative_script
(tmp_path: Path)
tests/validators/test_plugin_component_risk.py:1034
Functiontest_cd_relative_scripts_are_read
(tmp_path: Path, command: str)
tests/validators/test_plugin_static_risk_review_fixes.py:379
Functiontest_cdata_mode_falls_back_to_legacy_stdlib_signature
(monkeypatch: pytest.MonkeyPatch)
tests/validators/test_markdown.py:97
Functiontest_cdata_mode_forwards_escapable_to_new_stdlib_signature
(monkeypatch: pytest.MonkeyPatch)
tests/validators/test_markdown.py:113
Functiontest_cell_content_cannot_create_markup_mentions_or_extra_rows
()
tests/ci/test_comment_summary.py:239
Functiontest_census_arm_summary_counts_missing_and_unreadable_trials
()
tests/tier3/test_private_runtime_evidence.py:77
Functiontest_census_normalization_cannot_hide_changed_emitted_frontmatter
(tmp_path, monkeypatch)
tests/tier3/test_private_comparison_staging.py:101
Functiontest_census_reader_preserves_absent_recorded_and_unsafe
(tmp_path)
tests/tier3/test_private_runtime_evidence.py:60
Functiontest_certificate_evidence_survives_more_than_32_wrappers_and_a_cycle
()
tests/test_retry_exception_chain.py:60
Functiontest_certificate_failures_are_terminal_in_engine_and_verifier
()
tests/test_transport_rules.py:53
Functiontest_certified_paths_skipped_by_tier1_keep_publication_incomplete
(tmp_path: Path)
tests/test_publication_evidence.py:453
Functiontest_changed_mode_runs_only_changed_skill_from_real_git_history
(tmp_path)
tests/ci/test_run_tier1.py:1008
Functiontest_changed_paths_detects_an_unmodified_copy_source_outside_docs
( git_repo: tuple[Path, str], )
tests/test_ci_change_classifier.py:312
Functiontest_changed_pr_before_patch_is_not_overwritten
(trusted, monkeypatch, field, value)
tests/ci/test_benchmark_publish.py:443
Functiontest_changed_pr_body_before_patch_is_not_overwritten
(trusted, monkeypatch)
tests/ci/test_benchmark_publish.py:558
Functiontest_changed_private_site_stops_before_history_or_environment
(monkeypatch, tmp_path, info, pages, changes)
tests/test_ci_onboarding_pages.py:329
Functiontest_changed_selection_without_changes_is_explicitly_skipped
(consumer, tmp_path)
tests/ci/test_run_tiers.py:221
Functiontest_changed_selection_without_work_is_neutral_and_preserves_deleted_directories
(tmp_path)
tests/ci/test_native_reports.py:156
Functiontest_changed_skill_path_with_del_is_rejected_before_scanning
(repository)
tests/ci/test_select_changed_skills.py:74
Functiontest_changed_tier3_provider_does_not_inherit_build_model_or_api_root
()
tests/ci/test_model_config.py:38
Functiontest_changelog_documents_the_budget_stopped_failed_run
(tmp_path: Path)
tests/reporting/test_tier3_completed_agent_docs.py:572
Functiontest_changelog_entry_matches_the_docs
()
tests/reporting/test_tier3_completed_agent_docs.py:404
Functiontest_changelog_limits_the_empty_expected_output_rejection_to_the_object_shape
()
tests/test_harbor_scaffold_transactions.py:235
Functiontest_changelog_records_the_my_skill_md_docs_correction_after_0_9_0
()
tests/tier3/test_dataset_baseline_fairness.py:1237
Functiontest_changelog_says_a_stream_can_still_send_refused_output_later
()
tests/test_tier3_progress.py:2336
Functiontest_changes_between_stability_passes_fail_closed
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, after_capture: int )
tests/tier3/test_decoy_identity.py:159
Functiontest_changes_within_skill_select_only_that_skill
(repository, operation)
tests/ci/test_select_changed_skills.py:95
Functiontest_changing_quality_default_updates_library_and_plugin_gates
(below_threshold_skill: Path)
tests/test_quality_policy_gate.py:41
Methodtest_char_count
(self)
tests/deduplication/utils/test_chunker.py:97
Functiontest_character_facts_keep_numeric_dataset_answers_scoreable
(tmp_path, monkeypatch, filename, contents)
tests/tier3/test_judge_failure_artifacts.py:114
Functiontest_character_facts_use_the_same_formatting_as_prompt_placeholders
(monkeypatch)
tests/test_character_facts.py:85
Functiontest_character_identity_and_ascii_context
(character)
tests/test_character_facts.py:28
Functiontest_chart_canvases_explain_data_and_keep_visible_failure_fallback
(agent_count: int)
tests/reporting/test_chart_dependency.py:105
Functiontest_chart_fallback_is_visible_until_the_shared_lifecycle_draws_chart
()
tests/reporting/test_telemetry_view.py:179
Functiontest_chart_initialization_failure_keeps_fallback_and_recovers
()
tests/reporting/test_chart_dependency.py:156
Functiontest_chart_package_resources_are_cached_across_report_instances
()
tests/reporting/test_chart_dependency.py:144
Functiontest_chart_projection_excludes_unconsumed_evidence_and_keeps_measurements
()
tests/reporting/test_report_efficiency.py:20
Functiontest_chart_reference_opens_ancestor_evidence_and_reuses_shared_navigation
()
tests/reporting/test_chart_dependency.py:304
Functiontest_chart_runtime_unavailable_exposes_fallbacks_and_linked_tables_without_errors
(page, reports)
tests/reporting/test_report_browser.py:1101
Functiontest_chat_default_models_are_the_single_source_of_truth
()
tests/test_provider_config.py:70
Functiontest_chat_http_diagnostics_after_successful_embedding
Run collection, SDK requests, clustering, analysis and reports over loopback HTTP.
tests/test_tier2_command_failures.py:211
Methodtest_chat_names_the_model_that_answered
(self, monkeypatch: pytest.MonkeyPatch)
tests/inference/test_llm_client.py:1635
Functiontest_chat_only_tiers_do_not_require_or_forward_unused_embedding_routes
(tier, embedding_settings)
tests/ci/test_model_config.py:188
Functiontest_chat_only_tiers_do_not_validate_an_unused_embedding_route
The Tier 1 image job receives no embedding settings and must not fail on them.
tests/ci/test_model_config.py:232
Functiontest_chat_only_tiers_keep_shared_chat_key_but_exclude_embedding_only_key
(tier)
tests/ci/test_model_config.py:215
Functiontest_chat_refusal_is_advisory_in_real_cli_and_reports
A content-filtered chat reply is an advisory finding, not a blocking incomplete scan.
tests/test_tier2_command_failures.py:342
Methodtest_chdir_and_env_inside_sandbox
(self, tmp_path: Path)
tests/test_harbor_local_sandbox.py:635
Functiontest_check_activation_accepts_alternate_skill_with_partial_credit
()
tests/tier3/test_checks_path_extraction.py:115
Functiontest_check_activation_fails_when_wrong_skill_read
()
tests/tier3/test_checks_path_extraction.py:103
Functiontest_check_activation_recognizes_claude_code_read_file_path
Claude Code uses Read({file_path: ...}) — must be detected as activation.
tests/tier3/test_checks_path_extraction.py:74
Functiontest_check_activation_recognizes_codex_exec_command_cat_skill_md_with_frontmatter_name
()
tests/tier3/test_checks_path_extraction.py:132
Functiontest_check_activation_still_works_with_legacy_path_key
Backward compat: other agents using 'path' still work.
tests/tier3/test_checks_path_extraction.py:89
Methodtest_check_by_itself_does_not_claim_error_handling
(self, tmp_path: Path)
tests/validators/test_quality_score.py:1003
Functiontest_check_failures
(expected, scored, slots, failures, code)
tests/tier3/test_baseline_timeouts.py:1512
Functiontest_check_headlines_require_usable_arms
(failed_arm: str | None)
tests/test_tier3_result_display.py:162
Functiontest_check_names_are_plain_with_the_raw_id_beside_them
()
tests/reporting/test_report_redesign.py:343
Functiontest_check_negative_case_detects_file_path_incorrect_skill_read
Negative case: agent should NOT have read api-caller SKILL.md.
tests/tier3/test_checks_path_extraction.py:328
Functiontest_check_negative_case_passes_when_skill_not_read
()
tests/tier3/test_checks_path_extraction.py:342
Functiontest_check_pod_terminated_fails_fast_when_pod_deleted_or_preempted
Fail immediately when the evaluation pod is 404 Not Found while ignoring transient non-404 API errors.
tests/test_harbor_gke_environment.py:1426
Functiontest_check_prerequisites_gke_does_not_require_kubectl
Verify missing kubectl binary does not fail GKE prerequisites since Harbor uses Python client.
tests/test_harbor_gke_mode.py:181
Functiontest_check_prerequisites_gke_expands_user_in_kubeconfig
Expand user home directory (tilde) in KUBECONFIG entries.
tests/test_harbor_gke_mode.py:614
Functiontest_check_prerequisites_gke_live_cluster_rejects_bound_service_account_without_opt_in
Reject GKE live cluster preflight for non-Vertex agents when namespace default SA is bound or NetworkPolicy is weak.
tests/test_harbor_gke_mode.py:1274
Functiontest_check_prerequisites_gke_live_probe
When verify_live_cluster=True, runs a live probe and reports failure if unreachable.
tests/test_harbor_gke_mode.py:285
Functiontest_check_prerequisites_gke_ready
Prerequisites check passes when tools, kubeconfig, and kwargs are all present.
tests/test_harbor_gke_mode.py:274
Functiontest_check_prerequisites_gke_reports_missing_gcloud
Fail prerequisites check if gcloud is missing from PATH.
tests/test_harbor_gke_mode.py:174
Functiontest_check_prerequisites_gke_reports_missing_kubeconfig
Prerequisites check fails if kubeconfig file does not exist.
tests/test_harbor_gke_mode.py:253
Functiontest_check_prerequisites_gke_reports_missing_kubernetes_package
Fail prerequisites check when kubernetes package is not installed.
tests/test_harbor_gke_mode.py:305
← previousnext →13,701–13,800 of 24,741, ranked by callers