MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_bundled_check_suite_and_usage
(monkeypatch)
tests/tier3/test_grader_validation.py:459
Functiontest_bundled_examples_use_preferred_objects_with_fifteen_cases
()
tests/tier3/test_dataset_migration.py:307
Methodtest_bundled_external_tool_requests_sibling_first
(self, tmp_path)
tests/utils/test_tool_runner.py:163
Methodtest_bundled_file_path_is_contained_and_listed_once
(self, tmp_path: Path)
tests/validators/test_security.py:10079
Functiontest_bundled_grader_suite_size_matches_the_shipped_suite
()
tests/test_metrics_reference_docs.py:545
Functiontest_bundled_installer_without_ca_bundle_installs_ca_certificates
( tmp_path: Path, agent_class: type, manager: str, expected_install: str )
tests/tier3/test_installed_agent_bootstrap.py:158
Functiontest_bundled_profile_wraps_malformed_yaml
(tmp_path: Path, monkeypatch: pytest.MonkeyPatch)
tests/validators/test_policy.py:188
Functiontest_bundled_release_and_ci_namespace_are_available
()
tests/test_ci_onboarding.py:519
Methodtest_bundled_scanner_hints_point_at_reinstalling_skillevaluator
Bundled scanners hint at reinstalling SkillEvaluator, not a bare pip install. Bandit is a core dependency, so the fix needs no extra; pip-aud
tests/utils/test_tool_runner.py:422
Methodtest_bundled_skill_directory_variable_reaches_its_own_files
(self, tmp_path: Path)
tests/validators/test_skillspector_scope.py:1537
Methodtest_bundled_skill_mentioning_other_skills_does_not_reach_its_files
(self, tmp_path: Path)
tests/validators/test_skillspector_scope.py:1516
Functiontest_bundled_skill_pii_uses_its_own_author_without_sharing_identity
(tmp_path, monkeypatch)
tests/validators/test_private_plugin_composition.py:128
Methodtest_bundled_skill_reference_through_plugin_root_retains_finding
(self, tmp_path: Path)
tests/validators/test_skillspector_scope.py:1548
Methodtest_bundled_skill_script_bare_directory_reaches_its_own_files
(self, tmp_path: Path)
tests/validators/test_skillspector_scope.py:1355
Methodtest_bundled_tool_explicit_absolute_override_is_auditable
(self, tmp_path, monkeypatch)
tests/utils/test_tool_runner.py:178
Methodtest_bundled_tool_prefers_interpreter_sibling_over_path
Bundled scanners must come from the SkillEvaluator environment.
tests/utils/test_tool_runner.py:134
Methodtest_bundled_tool_rejects_relative_override
(self, monkeypatch)
tests/utils/test_tool_runner.py:194
Functiontest_bwrap_masks_explicit_denied_key_inside_broad_read_root
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_local_sandbox.py:1696
Methodtest_bwrap_smoke_argv_binds_lib_dirs
(self)
tests/test_harbor_local_sandbox.py:186
Methodtest_bwrap_smoke_argv_matches_real_namespace_shape
(self)
tests/test_harbor_local_sandbox.py:196
Functiontest_byte_order_mark_and_missing_closer
()
tests/test_plugin_markdown.py:93
Functiontest_bytecode_is_compiled_at_build_time_and_never_written_at_runtime
()
tests/ci/test_image_build.py:75
Methodtest_cache_includes_metadata
(self, tmp_path: Path)
tests/embedding/test_registry.py:559
Functiontest_cached_above_prompt_is_invalid
()
tests/tier3/test_token_usage.py:142
Functiontest_cached_input_pricing_overrides_reported_cost
()
tests/tier3/test_run_budget.py:94
Functiontest_cached_tokens_are_part_of_the_prompt_and_never_added_again
()
tests/tier3/test_token_usage.py:83
Functiontest_calculate_full_jitter_delay_bounds
Verify calculate_full_jitter_delay generates non-negative values bounded by backoff.
tests/test_retry.py:106
Functiontest_calibrated_agents_can_run_concurrently
(monkeypatch, tmp_path)
tests/tier3/test_run_budget.py:241
Functiontest_calibration_serializes_whole_unit_and_builds_largest_estimate
(monkeypatch, dimension)
tests/tier3/test_budget_reservation.py:123
Functiontest_call_api_omits_body_only_when_data_is_absent
(monkeypatch: pytest.MonkeyPatch)
tests/test_reference_skill_scripts.py:176
Functiontest_call_api_preserves_explicit_falsey_json_bodies
( body: object, encoded: bytes, monkeypatch: pytest.MonkeyPatch, )
tests/test_reference_skill_scripts.py:154
Functiontest_call_public_llm_reports_a_refusal_as_an_error
(monkeypatch: pytest.MonkeyPatch)
tests/test_llm_judge.py:123
Functiontest_call_public_llm_returns_partial_text_from_a_truncated_reply
(monkeypatch: pytest.MonkeyPatch)
tests/test_llm_judge.py:110
Functiontest_call_public_llm_uses_production_gpt5_payload_without_temperature
Verify call_public_llm formats gpt-5 payload with max_completion_tokens and no temperature.
tests/test_llm_judge.py:139
Functiontest_call_required_judge_enforces_skill_eval_llm_judge_budget_sec
Verify _call_required_judge applies the SKILL_EVAL_LLM_JUDGE_BUDGET_SEC deadline to judge execution.
tests/test_retry.py:704
Functiontest_call_sed_underscored_flag_and_config_forms_are_masked
(text: str, expected: str)
tests/tier3/test_security_finding_evidence.py:3571
Functiontest_call_sed_underscored_flag_and_config_rules_keep_ordinary_text
(text: str)
tests/tier3/test_security_finding_evidence.py:3589
Functiontest_callable_defaults_are_named_without_evaluation
()
tests/golden/test_cli_surface.py:29
Functiontest_callback_primary_exception_retains_cleanup_failure_evidence
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_secure_docker_environment.py:1547
Functiontest_callback_suppressing_one_cleanup_cancellation_is_not_reentered_or_leaked
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_local_mode.py:3017
Functiontest_called_but_nothing_succeeded_is_not_green_and_not_counted
()
tests/tier3/test_f90_mcp_proof_status.py:78
Functiontest_caller_approval_matches_git_line_ending_filters
(repository)
tests/test_ci_onboarding.py:604
Functiontest_caller_publication_failure_preserves_saved_snapshot_status_and_cleanup
(tmp_path, monkeypatch)
tests/tier3/test_private_runtime_report_acceptance.py:307
Functiontest_callers_without_declarations_keep_the_tagged_message
A caller that audits pins it collected itself converts this tagged form into its own findings.
tests/validators/test_dependency_advisory_findings.py:668
Functiontest_callers_write_that_would_block_is_finished_when_progress_closes
( monkeypatch: pytest.MonkeyPatch, refused_writes: set[int], )
tests/test_tier3_progress.py:1302
Functiontest_calls_the_server_rejected_do_not_pass_their_rules
()
tests/tier3/test_f90_mcp_arguments.py:62
Functiontest_canary_context_does_not_follow_a_linked_output_parent
(canary, tmp_path)
tests/tier3/test_private_plugin_canary.py:81
Functiontest_canary_in_declared_mcp_and_web_arguments_is_a_sink
(security)
tests/tier3/test_private_plugin_security.py:169
Functiontest_canary_preparation_bounds_server_and_workspace_collections
(canary)
tests/tier3/test_private_plugin_canary.py:74
Functiontest_canary_rate_refuses_invalid_counts_instead_of_producing_out_of_range_values
()
tests/tier3/test_private_runtime_evidence.py:259
Functiontest_canary_reaching_a_sink_is_critical_and_redacted
(security, command, kind)
tests/tier3/test_private_plugin_security.py:147
Functiontest_canary_readback_confirms_the_actual_decoy_and_refuses_parent_links
(security, tmp_path)
tests/tier3/test_private_plugin_security.py:201
Functiontest_canary_reading_and_workspace_use_are_not_leaks
(security, command)
tests/tier3/test_private_plugin_security.py:164
Functiontest_canary_root_and_server_caps_are_not_silent
(security)
tests/tier3/test_private_plugin_security.py:328
Functiontest_canary_scalar_validation_is_exact_and_control_free
(security)
tests/tier3/test_private_plugin_security.py:367
Functiontest_canary_summary_compares_rates_and_keeps_unknown_attribution
()
tests/tier3/test_private_runtime_evidence.py:102
Functiontest_cancellation_during_timeout_callback_cleanup_is_not_swallowed
( tmp_path: Path, repeat_cancellation: bool, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_local_mode.py:3125
Functiontest_cancellation_during_timeout_diagnostic_callback_reaps_callback_tasks
( tmp_path: Path, repeat_cancellation: bool, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_local_mode.py:3069
Functiontest_cancellation_during_timeout_reap_preserves_cancellation_and_containment
( tmp_path: Path, repeat_cancellation: bool, )
tests/test_harbor_local_mode.py:3386
Functiontest_cancellation_is_not_retried_and_still_cleans_up
(tmp_path: Path)
tests/tier3/test_plugin_execution.py:84
Functiontest_cancellation_remains_primary_when_timeout_reap_fails
( tmp_path: Path, repeat_cancellation: bool, )
tests/test_harbor_local_mode.py:3457
Functiontest_cancellation_wakes_waiter_before_in_flight_job_finishes
(monkeypatch, tmp_path)
tests/tier3/test_budget_reservation.py:354
Functiontest_cancellation_with_observer_restores_context_and_cleans_wrapper
(tmp_path)
tests/tier3/test_plugin_evidence.py:531
Functiontest_cancelled_native_session_closes_constrained_client
(monkeypatch, fake_mcp)
tests/tier3/test_mcp_proof_sdk_compat.py:356
Functiontest_candidate_caller_propagates_reusable_read_permissions
()
tests/ci/test_all_tier_workflows.py:330
Functiontest_candidate_descriptor_is_declared_as_package_data
()
tests/test_ci_onboarding_all_tiers.py:709
Functiontest_candidate_onboarding_uses_the_released_tier1_image
()
tests/ci/test_all_tier_workflows.py:16
Functiontest_candidate_templates_do_not_change_legacy_bundle_generation
()
tests/ci/test_all_tier_workflows.py:219
Functiontest_canonical_budget_drops_failure_reasons_before_recorded_evidence
(monkeypatch: pytest.MonkeyPatch)
tests/reporting/test_tier3_failure_reasons.py:1016
Functiontest_canonical_budget_records_dropped_diagnostics
(monkeypatch)
tests/reporting/test_tier3_expectations.py:359
Functiontest_canonical_budget_restores_reasons_after_dropping_diagnostics
(monkeypatch, oversized_reasons)
tests/reporting/test_tier3_expectations.py:370
Functiontest_canonical_case_id_arm_stripping
Verify _canonical_case_id normalizes task identifiers across naming and attempt variants.
tests/test_tier3_public_runtime.py:2195
Functiontest_canonical_dimension_mean_rejects_partial_or_invalid_dimension_sets
(values: list[float])
tests/test_harbor_metrics_truth.py:672
Functiontest_canonical_export_agent_seconds_matches_retained_run
(tmp_path: Path)
tests/reporting/test_dashboard_data.py:766
Functiontest_canonical_html_keeps_normalized_evidence_with_one_outer_pointer_distinct
()
tests/reporting/test_unified_tier3_report.py:1657
Functiontest_canonical_html_renders_evaluator_evidence_and_custom_metric_details
()
tests/reporting/test_unified_tier3_report.py:1599
Functiontest_canonical_html_tolerates_legacy_string_evidence_refs
()
tests/reporting/test_unified_tier3_report.py:1701
Methodtest_canonical_identity_is_preserved
(self)
tests/test_benchmark_source_provenance.py:83
Functiontest_canonical_missing_attempt_keeps_usage_totals_unknown
()
tests/reporting/test_dashboard_data.py:149
Functiontest_canonical_missing_trial_counts_each_declared_trial_once
()
tests/reporting/test_dashboard_data.py:159
Functiontest_canonical_multi_agent_pass_survives_partial_peer_evidence
(tmp_path: Path)
tests/test_public_benchmark_gate.py:1114
Functiontest_canonical_payload_bounds_high_cardinality_custom_report_details
()
tests/reporting/test_unified_tier3_report.py:2214
Functiontest_canonical_payload_deduplicates_repeated_metric_evidence
()
tests/reporting/test_unified_tier3_report.py:1774
Functiontest_canonical_payload_enforces_total_serialized_budget
()
tests/reporting/test_unified_tier3_report.py:2674
Functiontest_canonical_payload_keeps_both_agents_and_arms
()
tests/reporting/test_declarative_coverage.py:174
Functiontest_canonical_payload_sanitizes_unsafe_integers_for_browser_json_semantics
()
tests/reporting/test_unified_tier3_report.py:2503
Functiontest_canonical_record_drops_unknown_fields_and_is_a_copy
()
tests/test_skillspector_provenance.py:43
Methodtest_canonical_ref_bad_source_rejected
(self)
tests/models/test_plugin.py:82
Methodtest_canonical_ref_repo_without_slash_rejected
(self)
tests/models/test_plugin.py:86
Functiontest_canonical_report_fails_closed_on_different_condition_policies
()
tests/reporting/test_unified_tier3_report.py:1203
Functiontest_canonical_report_fails_closed_on_mixed_recorded_score_policies
()
tests/reporting/test_unified_tier3_report.py:1169
Functiontest_canonical_report_prefers_agentskills_dataset_fields
()
tests/reporting/test_unified_tier3_report.py:1522
Functiontest_canonical_report_serializers_disclose_non_finite_numbers
()
tests/reporting/test_unified_tier3_report.py:2586
Methodtest_canonical_support_dirs_accepted
Canonical public skill support directories must not be flagged. Skills may carry agent metadata, local verification, executable helpers,
tests/validators/test_schema.py:825
Functiontest_capability_errors_reject_unsupported_combinations
()
tests/test_nemo_gym_backend.py:53
Functiontest_capitalized_http_words_on_one_line_are_checked_once_per_client
(monkeypatch)
tests/tier3/test_network_client_mentions.py:539
Functiontest_capped_report_uses_recorded_lift_when_available
( tmp_path: Path, recorded_lift: bool, metric: str, dimension: str, dimension_lift: float )
tests/test_judge_lift_integrity.py:281
Methodtest_captured_report_matches_its_tree
Editing a tree requires recapturing its report with the same SkillSpector release and --no-llm.
tests/validators/test_skillspector_hook_surface.py:423
Functiontest_card_lists_failed_behavior_results_once
()
tests/reporting/test_tier3_failure_reasons.py:489
Methodtest_card_records_the_evaluated_source
(self)
tests/test_benchmark_source_provenance.py:261
← previousnext →13,501–13,600 of 24,741, ranked by callers