MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_trial_that_is_neither_scored_nor_timed_out_is_refused
(tmp_path: Path)
tests/tier3/test_baseline_timeouts.py:630
Functiontest_trial_whose_reward_identity_the_collector_rejected_keeps_its_own_case
(tmp_path: Path, records: dict)
tests/test_tier3_case_summary.py:1217
Functiontest_trial_without_judge_references_writes_empty_attempt_artifact
(tmp_path: Path)
tests/tier3/test_judge_contracts.py:821
Functiontest_trial_without_trajectory_keeps_case_level_skill_na
(tmp_path, monkeypatch, labels, na_reasons)
tests/tier3/test_skill_metric_applicability.py:207
Functiontest_trigger_cut_at_the_listing_window_counts_as_missing
(tmp_path: Path)
tests/harness/test_overlay.py:1234
Functiontest_trimmed_card_evidence_keeps_the_lift_note
(monkeypatch: pytest.MonkeyPatch)
tests/test_judge_lift_integrity.py:528
Functiontest_trivy_remote_fallback_reports_vulnerability
(audit, tmp_path)
tests/validators/test_private_dependency_audit.py:261
Methodtest_troubleshooting_header_accepts_configured_alternative
(self, tmp_path: Path, alternative: str)
tests/validators/test_quality_score.py:348
Functiontest_truncated_attempt_details_do_not_invent_an_attempts_passed_count
(tmp_path)
tests/reporting/test_case_findings_report.py:107
Functiontest_truncated_attempts_cannot_produce_a_complete_effect_or_clean_bill
(tmp_path: Path)
tests/test_tier3_case_summary.py:1854
Functiontest_truncated_case_sample_does_not_claim_baseline_saturation
(tmp_path: Path)
tests/test_tier3_case_summary.py:1864
Functiontest_truncated_markdown_keeps_table_boundaries_intact
()
tests/ci/test_report_views.py:416
Functiontest_truncated_quoted_credential_is_omitted_from_evidence
(security, quote)
tests/tier3/test_private_plugin_security.py:491
Methodtest_truncated_reply_uses_the_fallback_without_salvage
(self, monkeypatch: pytest.MonkeyPatch)
tests/inference/test_llm_client.py:1328
Methodtest_truncated_report_keeps_every_issue
( self, tmp_path: Path, findings_after_filtering: int | None, passed: bool )
tests/validators/test_skillspector_scope.py:643
Methodtest_truncated_report_stays_incomplete
A list SkillSpector cut short may leave out the line that names a file the scan left out.
tests/validators/test_security.py:10003
Functiontest_truncated_rollout_line_is_skipped_and_reported
(tmp_path: Path)
tests/test_nemo_gym_backend.py:1115
Functiontest_truncated_scanner_readback_cannot_become_a_zero_leak_rate
(tmp_path, monkeypatch)
tests/tier3/test_private_runtime_evidence.py:180
Functiontest_truncated_verifier_reply_without_complete_verdicts_names_no_model
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, reply: str )
tests/inference/test_llm_client.py:1793
Methodtest_truncates_large_files
(self, tmp_path)
tests/validators/test_rubric_eval.py:281
Methodtest_truncation_error_names_the_model_that_wrote_the_partial_reply
( self, monkeypatch: pytest.MonkeyPatch, provider: str, model: str, make_response, expected: str )
tests/inference/test_llm_client.py:1720
Methodtest_truncation_is_a_typed_error_that_keeps_partial_text
( self, monkeypatch: pytest.MonkeyPatch, provider: str, model: str, response )
tests/inference/test_llm_client.py:1270
Functiontest_trusted_aggregate_receipt_is_captured_before_untrusted_artifact_read
()
tests/ci/test_all_tier_workflows.py:175
Functiontest_trusted_comment_has_one_combined_report_link_and_compact_verified_skill_summary
(request, platform)
tests/ci/test_feedback.py:280
Functiontest_trusted_custom_workdir_supplies_a_stable_output_scope
(security_check)
tests/tier3/test_agent_output_cleanup_scope.py:97
Functiontest_trusted_feedback_keeps_paid_failures_advisory_and_tier1_blocking
(request, platform, tier1_status)
tests/ci/test_feedback.py:187
Functiontest_trusted_native_workflow_entrypoints_and_pip_ignore_consumer_import_paths
()
tests/ci/test_all_tier_workflows.py:310
Functiontest_trusted_observed_activation_is_a_public_metrics_helper
()
tests/test_tier3_case_summary.py:1733
Functiontest_trusted_pins_accept_documented_list_separators
(monkeypatch, separator)
tests/ci/test_publish_reports.py:605
Functiontest_trusted_selector_mapping_ignores_task_name_and_unmapped_paths
()
tests/test_harbor_collector_runtime_failures.py:2001
Functiontest_trusted_selector_mapping_rejects_sidecar_reward_without_readable_trial_result
( tmp_path: Path, trial_result_state: str, )
tests/test_harbor_collector_runtime_failures.py:2151
Functiontest_trusted_task_selector_mapping_does_not_parse_attempt_like_identities_as_ordinals
(tmp_path: Path)
tests/test_harbor_collector_runtime_failures.py:2029
Functiontest_trusted_task_selector_mapping_invalidates_unresolved_authored_identity
( result: dict[str, object], )
tests/test_harbor_collector_runtime_failures.py:2245
Functiontest_trusted_task_selector_mapping_uses_structural_attempt_for_custom_only_stop_on_pass
(tmp_path: Path)
tests/test_harbor_collector_runtime_failures.py:2066
Methodtest_try_except_pair_counts_regardless_of_markers
(self, tmp_path: Path)
tests/validators/test_quality_score.py:1622
Methodtest_tsx_component_marked_executable_by_current_skillspector_stays_blocking
SkillSpector 2.12+ marks ``.tsx`` executable, so UI strings in it are not suppressed.
tests/validators/test_skillspector_scope.py:1283
Functiontest_tuning_observations_cannot_fill_assessment_coverage
()
tests/tier3/test_grader_policy.py:214
Functiontest_two_dataset_ids_with_the_same_cut_name_are_never_guessed
()
tests/tier3/test_token_usage.py:709
Methodtest_two_digit_block_numbers_cost_one_more_character
(self, make_chunk, length: int, shared: bool)
tests/deduplication/intra_skill/test_intra_llm_analyzer.py:212
Functiontest_two_hook_sources_are_two_pieces
(tmp_path: Path)
tests/tier3/test_private_loose_source.py:780
Functiontest_two_pass_at_k_keys_for_one_case_merge_the_same_way_in_either_order
(cut_first: bool)
tests/tier3/test_token_usage.py:771
Methodtest_two_payloads_naming_different_repositories_fail_closed
(self, reversed_order: bool)
tests/test_benchmark_source_provenance.py:1380
Methodtest_two_payloads_that_agree_still_render_the_identity
Folding must not turn a repeated identity into a conflict.
tests/test_benchmark_source_provenance.py:1388
Methodtest_two_repositories_sharing_an_evaluator_keep_distinct_source_revisions
The regression this feature exists to prevent. Both skills are evaluated by the same evaluator container. Before the evaluated-source
tests/test_benchmark_source_provenance.py:316
Methodtest_two_results_disagreeing_fail_closed
(self)
tests/test_benchmark_source_provenance.py:883
Functiontest_two_stream_transport_uses_real_session
A modern two-stream boundary must work even in locked SDK1 CI.
tests/tier3/test_mcp_proof_sdk_compat.py:36
Methodtest_two_vectors
(self)
tests/embedding/test_client.py:429
Functiontest_type_specific_table_lists_every_type_specific_deduction
The type-specific table is pinned one-for-one, not just by the sizes it uses. Its rows quote one amount per check and never collapse them with "e
tests/test_metrics_reference_docs.py:507
Functiontest_unactivated_skill_does_not_generate_quality_advice
()
tests/test_harbor_skill_activation.py:34
Functiontest_unapplied_host_configuration_makes_the_run_incomplete
(tmp_path: Path)
tests/tier3/test_openclaw_prepare.py:133
Functiontest_unattributed_native_wrapper_observation_is_scanned_once
(template, rendered, expected_status)
tests/tier3/test_codex_tool_call_normalization.py:522
Functiontest_unavailable_environment_fails_before_generation
(skill, configured, monkeypatch)
tests/tier3/test_workflow.py:461
Functiontest_unavailable_exit_stays_not_checked
(metadata, native_events, tmp_path)
tests/tier3/test_opencode_exit_status.py:67
Methodtest_unavailable_judging_invalidates_folder_aggregate
(self, tmp_path)
tests/validators/test_rubric_eval.py:1037
Functiontest_unavailable_lift_is_never_zero
(tmp_path, failure)
tests/test_tier3_history.py:120
Methodtest_unavailable_llm_report_metadata_is_incomplete_and_redacted_everywhere
( self, mock_tools, sample_skill_dir: Path, )
tests/validators/test_security.py:1701
Functiontest_unavailable_model_points_to_the_model_setting
(status)
tests/inference/test_diagnostics.py:34
Functiontest_unavailable_overall_comparison_does_not_label_a_missing_measurement_fail
()
tests/reporting/test_native_feature_restoration.py:208
Functiontest_unavailable_pairing_does_not_render_a_zero_evidence_row
()
tests/reporting/test_unified_tier3_report.py:3967
Functiontest_unavailable_reason_uses_reward_load_error_not_trajectory_provenance
()
tests/reporting/test_tier3_failure_reasons.py:632
Functiontest_unavailable_rows_use_recorded_public_signal_mapping_only
(metric, dimensions)
tests/reporting/test_native_overview_comparison.py:102
Methodtest_unavailable_when_no_uv_exists_anywhere
(self, tmp_path, monkeypatch)
tests/utils/test_tool_runner.py:776
Methodtest_unavailable_without_uv_or_when_unmanaged
(self, managed)
tests/utils/test_scanner_install.py:132
Functiontest_unblocked_empty_runtime_env_is_preserved
()
tests/test_harbor_local_mode.py:4250
Methodtest_unbounded_json_integer_fails_closed
(self, mock_tools, sample_skill_dir: Path)
tests/validators/test_security.py:1469
Functiontest_unbounded_previous_version_is_rejected_without_crashing
(tmp_path: Path)
tests/validators/test_version.py:88
Functiontest_unbounded_wall_time_is_rejected
(wall_time: str)
tests/tier3/test_trajectory_timing.py:71
Methodtest_uncertain_evidence_keeps_generic_error
Mixed, malformed, truncated or older reports cannot pass for a provider outage.
tests/validators/test_security.py:7210
Functiontest_uncertain_shell_contexts_do_not_gain_notice_exemptions
(module, command)
tests/tier3/test_license_notice_network_bypasses.py:127
Functiontest_unchanged_cards_do_not_create_branch
(trusted, monkeypatch)
tests/ci/test_benchmark_publish.py:280
Functiontest_unchanged_skill_pr_is_successful_neutral_noop_with_reports
(tmp_path, capsys)
tests/ci/test_run_tier1.py:1033
Functiontest_unchecked_behavior_step_projects_insufficient_evidence
(unchecked_behavior, reverse_expectations)
tests/tier3/test_grader_validation.py:353
Functiontest_unclassified_cases_make_negative_coverage_unknown
()
tests/tier3/test_dataset_sufficiency.py:95
Functiontest_unclosed_fence_keeps_all_remaining_lines
(opener: str)
tests/deduplication/utils/test_chunker_fences.py:51
Functiontest_unclosed_html_scanning_is_linear
Measure regex input work for the four guarded forms, not all Markdown.
tests/validators/test_hygiene_link_boundaries.py:162
Methodtest_unclosed_xml_tag_in_description_rejected
Test validation fails when description contains an unclosed tag-like value.
tests/validators/test_schema.py:541
Methodtest_unclosed_xml_tag_in_description_remains_quality_error
Unclosed tag-like descriptions remain covered by XML-tag detection.
tests/validators/test_quality_score.py:498
Functiontest_unconfigured_usage_does_not_imply_a_budget_verification
()
tests/tier3/test_run_budget.py:729
Functiontest_unconfirmed_gym_stop_launches_nothing_more_and_keeps_finished_arms
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, hanging_launch: int )
tests/test_nemo_gym_backend.py:2570
Methodtest_unconfirmed_llm_analysis_keeps_attempted_provenance
( self, monkeypatch, sample_skill_dir: Path, )
tests/validators/test_security.py:5897
Methodtest_unconstructible_value_is_invalid_frontmatter
(self, tmp_path: Path, line: str)
tests/embedding/test_extractor.py:872
Methodtest_unconstructible_value_keeps_frontmatter_in_the_analyzed_text
(self, skill_root: Path, line: str)
tests/deduplication/utils/test_skill_collector.py:198
Methodtest_undecodable_object_blocks_a_later_object
(self, monkeypatch: pytest.MonkeyPatch)
tests/inference/test_llm_client.py:959
Methodtest_undecodable_object_fails_closed
(self, monkeypatch: pytest.MonkeyPatch, prefix: str, payload: str)
tests/inference/test_llm_client.py:910
Functiontest_undelivered_components_degrade
(tmp_path: Path)
tests/harness/test_plugin_compat.py:84
Functiontest_underscored_flags_and_config_keys_keep_the_assignment_reading
(copy: str, command: str, expected: str)
tests/tier3/test_security_finding_evidence.py:3912
Methodtest_underscored_repository_is_published_verbatim
Markdown escaping would rewrite ``_`` and publish a repository that does not exist.
tests/test_benchmark_source_provenance.py:286
Functiontest_unevaluable_grouping_rule_does_not_stop_the_user_copy_lookup
()
tests/harness/test_overlay.py:590
Functiontest_unexpected_capability_root_is_not_normalized_as_equal
(tmp_path)
tests/tier3/test_private_comparison_staging.py:124
Functiontest_unexpected_case_fails_execution_coverage
(tmp_path: Path)
tests/test_harbor_collector_runtime_failures.py:1749
Methodtest_unexpected_exit_does_not_echo_scanner_stdout
(self, mock_run)
tests/validators/test_secrets_structured.py:95
Functiontest_unexpected_file_in_generated_parent_still_breaks_resource_parity
(tmp_path: Path)
tests/tier3/test_private_loose_dependency_scaffold_review.py:62
Methodtest_unexpected_files_warning
Test warning for unexpected files in skill root.
tests/validators/test_folder_validation.py:472
Methodtest_unexpected_llm_exit_redacts_provider_diagnostic_from_all_reports
( self, mock_tools, sample_skill_dir: Path, )
tests/validators/test_security.py:1669
Functiontest_unexpected_render_failure_cannot_publish_a_passing_receipt
(tmp_path, monkeypatch)
tests/ci/test_publish_reports.py:581
Functiontest_unexpected_run_provenance_is_rejected
(tmp_path, changes)
tests/ci/test_publish_reports.py:1212
Methodtest_unexpected_scoping_error_keeps_every_finding
( self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch )
tests/validators/test_skillspector_scope.py:724
Methodtest_unexpected_skillspector_exit_fails_closed
Exit codes other than the scanner's clean/findings policy codes are tool failures.
tests/validators/test_security.py:1484
Methodtest_unexpected_skillspector_suppression_fails_closed
(self, mock_tools, sample_skill_dir: Path)
tests/validators/test_security.py:1897
Functiontest_unexpected_worker_error_is_a_row_error_and_later_skills_still_run
(tmp_path, capsys, workers)
tests/ci/test_run_tier1.py:1857
← previousnext →23,501–23,600 of 24,741, ranked by callers