MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_extract_json_rejects_finite_syntax_that_overflows_to_infinity
(text)
tests/tier3/test_judge_parse_robustness.py:259
Functiontest_extract_json_rejects_multiple_complete_documents
(text)
tests/tier3/test_judge_parse_robustness.py:220
Functiontest_extract_json_rejects_nonstandard_json_constants
(constant)
tests/tier3/test_judge_parse_robustness.py:245
Functiontest_extract_json_returns_none_for_excessive_nesting
()
tests/tier3/test_judge_parse_robustness.py:225
Functiontest_extract_json_returns_none_for_unrecoverable_text
(text)
tests/tier3/test_judge_parse_robustness.py:154
Functiontest_extract_json_tolerates_observed_shapes
(text, expected)
tests/tier3/test_judge_parse_robustness.py:149
Functiontest_extract_path_empty_when_missing
()
tests/tier3/test_checks_path_extraction.py:54
Functiontest_extract_path_handles_file_path_key
()
tests/tier3/test_checks_path_extraction.py:31
Functiontest_extract_path_handles_missing_args
()
tests/tier3/test_checks_path_extraction.py:64
Functiontest_extract_path_handles_non_dict_args
()
tests/tier3/test_checks_path_extraction.py:59
Functiontest_extract_path_handles_path_key
()
tests/tier3/test_checks_path_extraction.py:36
Functiontest_extract_path_handles_raw_key
()
tests/tier3/test_checks_path_extraction.py:41
Functiontest_extract_path_prefers_file_path_first
()
tests/tier3/test_checks_path_extraction.py:46
Methodtest_extracts_all_fields
(self, make_cluster)
tests/deduplication/intra_skill/test_intra_llm_analyzer.py:483
Functiontest_facts_detect_formats_name_and_components
(tmp_path: Path)
tests/harness/test_plugin_compat.py:51
Functiontest_facts_injected_at_top_of_all_bundles_and_verified_field
()
tests/tier3/test_verified_facts.py:37
Functiontest_fail_includes_file_and_line
()
tests/reporting/test_junit_reporter.py:73
Functiontest_failed_agent_eval_summary_row_shows_verdict_reason_and_lift
()
tests/reporting/test_terminal_polish.py:360
Functiontest_failed_agent_has_no_quality_headline_in_multi_agent_summary
(failed_agent)
tests/test_tier3_result_display.py:206
Functiontest_failed_agent_has_no_synthetic_score_in_comparison_or_html
(tmp_path: Path)
tests/test_harbor_html_report_failures.py:183
Functiontest_failed_agent_without_diagnostic_says_no_score_and_explains_failure
()
tests/test_tier3_result_display.py:1272
Functiontest_failed_and_incomplete_counts_agree_across_all_surfaces
(tmp_path)
tests/ci/test_native_reports.py:254
Functiontest_failed_arm_distinct_attempts_with_identical_usage_each_count
Retries share a case ID, and these two also share their token counts. Each is still one attempt. Attempts are told apart by the trial directory t
tests/reporting/test_failed_arm_trials.py:146
Functiontest_failed_arm_keeps_cost_when_all_executed_observations_were_retained
()
tests/tier3/test_private_comparison_contracts.py:144
Functiontest_failed_arm_keeps_scored_trial_activation
(tmp_path: Path, arm: str, key: str)
tests/reporting/test_failed_arm_trials.py:108
Functiontest_failed_arm_retains_coverage_without_restoring_withheld_quality
(tmp_path)
tests/reporting/test_declarative_coverage.py:351
Functiontest_failed_arm_rewards_without_trial_id_are_counted_once
Older rewards carry no ``trial_id``: the diagnostic trial and the usage-only copy are one attempt.
tests/reporting/test_failed_arm_trials.py:126
Functiontest_failed_arm_score_bearing_outputs_are_unchanged
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, arm: str, key: str )
tests/reporting/test_failed_arm_trials.py:248
Functiontest_failed_base_image_build_is_degraded
(tmp_path: Path, monkeypatch: pytest.MonkeyPatch)
tests/test_nemo_gym_backend.py:2219
Functiontest_failed_baseline_fails_the_run_unless_tolerated_and_the_docs_say_so
The baseline arm stays failed either way; only the option keeps the agent and the run.
tests/tier3/test_baseline_timeouts.py:1129
Functiontest_failed_bash_with_leading_error_stays_failed_in_plugin_signals
(agent_class, native_events, tmp_path)
tests/tier3/test_opencode_exit_status.py:132
Functiontest_failed_brace_expanded_cat_keeps_read_credit
(grader: tuple[ModuleType, ModuleType], tmp_path: Path)
tests/tier3/test_failed_skill_reads.py:395
Functiontest_failed_build_preserves_committed_index_and_coverage
( tmp_path: Path, initial_malformed: int, failure: str )
tests/embedding/test_collection_coverage.py:188
Functiontest_failed_cards_preserve_distinct_problem_text_with_the_same_action_and_evidence
()
tests/reporting/test_native_report_parity.py:201
Functiontest_failed_cat_of_home_relative_skill_loses_credit
(grader)
tests/tier3/test_failed_skill_reads.py:443
Functiontest_failed_cat_with_dash_prefixed_second_file_keeps_read_credit
( grader: tuple[ModuleType, ModuleType], tmp_path: Path, delimiter: str )
tests/tier3/test_failed_skill_reads.py:369
Functiontest_failed_cat_with_expanding_operand_keeps_read_credit
(grader, command: str)
tests/tier3/test_failed_skill_reads.py:429
Functiontest_failed_claude_calls_end_to_end_do_not_prove_the_server
()
tests/tier3/test_f90_mcp_proof_status.py:105
Methodtest_failed_cluster_adds_no_model_beside_a_fallback_verdict
( self, mock_embed, mock_llm, mock_analyze, tmp_path: Path )
tests/deduplication/intra_skill/test_intra_skill_validator.py:1540
Functiontest_failed_compound_shell_read_keeps_credit
(grader, command)
tests/tier3/test_failed_skill_reads.py:125
Functiontest_failed_condition_hides_stale_metric_values
()
tests/test_tier3_result_display.py:1244
Functiontest_failed_condition_rebuild_does_not_follow_a_symlinked_trials_directory
(tmp_path: Path)
tests/test_tier3_case_summary.py:1985
Functiontest_failed_condition_rebuilds_case_rows_from_trial_rewards
(tmp_path: Path)
tests/test_tier3_case_summary.py:1062
Functiontest_failed_condition_rebuilds_native_task_rows_under_the_case_the_collector_recorded
(tmp_path: Path)
tests/test_tier3_case_summary.py:1145
Functiontest_failed_condition_rows_are_rebuilt_once_per_what_to_fix_call
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch )
tests/test_tier3_case_summary.py:1177
Functiontest_failed_conditions_keep_rebuilt_scores_diagnostic
(tmp_path: Path)
tests/test_tier3_case_summary.py:2035
Functiontest_failed_conditions_keep_requested_grading_policy
( tmp_path: Path, grading_mode: str | None, skip_baseline: bool )
tests/test_harbor_unscored_grading_policy.py:123
Functiontest_failed_configuration_reasons_identify_each_arm
(tmp_path)
tests/tier3/test_interactions.py:330
Functiontest_failed_custom_agent_does_not_gain_authoritative_scores_from_cached_cards
()
tests/reporting/test_native_overview_comparison.py:386
Functiontest_failed_custom_only_summary_cannot_publish_stale_overall_score
(tmp_path: Path)
tests/test_harbor_html_report_failures.py:540
Functiontest_failed_default_run_without_metrics_is_not_labeled_custom_only
(tmp_path: Path)
tests/test_harbor_html_report_failures.py:502
Functiontest_failed_docker_doctor_prevents_native_provider_execution
(tmp_path, monkeypatch)
tests/ci/test_all_tier_preflight.py:200
Functiontest_failed_execution_actions_do_not_repeat_next_steps_or_expand_long_evidence
()
tests/reporting/test_native_report_parity.py:121
Methodtest_failed_execution_with_policy_exit_code_gets_provider_message
A report with execution_successful=false is explained the same way after exit 0 or 1.
tests/validators/test_security.py:7086
Functiontest_failed_finding_and_json_include_evidence_unavailable_reason
()
tests/tier3/test_plain_language_findings.py:71
Methodtest_failed_first_run_install_is_reported_not_launched
(self, monkeypatch)
tests/utils/test_scanner_install.py:194
Functiontest_failed_gym_launch_is_reported_not_succeeded
(tmp_path: Path, monkeypatch: pytest.MonkeyPatch)
tests/test_nemo_gym_backend.py:1178
Functiontest_failed_harbor_trial_preserves_redacted_judge_diagnostics
( tmp_path: Path, verifier_relative: Path, )
tests/test_issue55_collector_truth.py:627
Functiontest_failed_head_or_redirect_resolution_is_incomplete
(er)
tests/validators/test_private_endpoint_resolution.py:160
Functiontest_failed_hook_gate_with_incomplete_scan_is_listed_as_failed_not_as_setup
A Critical plus failed gate outranks an incomplete scan, as in result_status and the other reporters.
tests/reporting/test_tier3_validate_cases.py:161
Functiontest_failed_judge_does_not_depend_on_expected_coverage_options
(tmp_path: Path)
tests/test_issue55_collector_truth.py:696
Functiontest_failed_judge_keeps_its_rejected_attempts_without_a_score
(tmp_path: Path)
tests/tier3/test_judge_contracts.py:834
Functiontest_failed_judge_sidecar_is_merged_but_never_scored_and_reason_is_safe
(tmp_path: Path)
tests/test_issue55_collector_truth.py:583
Functiontest_failed_judge_without_sidecar_still_fails_closed_with_generic_reason
(tmp_path: Path)
tests/test_issue55_collector_truth.py:676
Functiontest_failed_late_creation_cleanup_retains_process_for_stop_retry
( tmp_path: Path, caplog: pytest.LogCaptureFixture, secret: str, )
tests/test_harbor_local_mode.py:3924
Functiontest_failed_later_agent_does_not_relabel_a_successful_legacy_agent
(tmp_path: Path)
tests/test_issue55_collector_truth.py:450
Methodtest_failed_llm_enrichment_preserves_deterministic_findings_and_is_incomplete
( self, mock_tools, llm_failure: ToolResult, sample_skill_dir: Path, )
tests/validators/test_security.py:1593
Methodtest_failed_llm_stage_keeps_attempted_provenance
( self, monkeypatch, sample_skill_dir: Path, exit_code: int, stderr: s
tests/validators/test_security.py:5925
Methodtest_failed_llm_stage_without_json_records_no_provenance
( self, monkeypatch, sample_skill_dir: Path, )
tests/validators/test_security.py:5956
Methodtest_failed_local_verdict_adds_blocking_aggregate_finding
(self, tmp_path)
tests/validators/test_rubric_eval.py:854
Functiontest_failed_manifest_cannot_publish_a_passed_native_report
(tmp_path)
tests/ci/test_benchmark_reports.py:199
Functiontest_failed_native_skill_call_is_not_activation
(grader)
tests/tier3/test_failed_skill_reads.py:157
Functiontest_failed_negative_keeps_task_diagnostics
(tmp_path: Path, activation_status: str, failure_kind: str)
tests/test_tier3_case_summary.py:1473
Functiontest_failed_next_steps_keep_distinct_agent_case_and_condition_in_rendered_cards
()
tests/reporting/test_native_report_parity.py:247
Functiontest_failed_or_incomplete_inventory_is_incomplete
(tmp_path, monkeypatch, output, exit_code)
tests/validators/test_target_dependencies.py:121
Functiontest_failed_or_interrupted_generation_stops_without_success
(monkeypatch, error)
tests/tier3/test_dataset_generation_progress.py:98
Functiontest_failed_peer_keeps_every_agent_failed
(tmp_path: Path)
tests/tier3/test_baseline_timeouts.py:705
Functiontest_failed_ragas_provider_replies_are_saved_with_their_prompt
( verifier: ModuleType, monkeypatch: pytest.MonkeyPatch, first_reply: Any, problem: str )
tests/tier3/test_ragas_judge_evidence.py:316
Functiontest_failed_read_does_not_satisfy_workflow_before_execution
( grader: tuple[ModuleType, ModuleType], action: str, )
tests/tier3/test_failed_skill_reads.py:94
Functiontest_failed_reports_show_the_counts
(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, skills: tuple)
tests/reporting/test_skillspector_reference_summary.py:101
Functiontest_failed_result_outranks_incomplete_in_html_tile_and_header
()
tests/reporting/test_final_render_context.py:128
Functiontest_failed_retries_are_not_first_attempt_clean
Repeated failures of a similar command are all recorded as unrecovered.
tests/tier3/test_error_recovery.py:444
Methodtest_failed_run_has_no_synthetic_score_or_top_performer
(self)
tests/reporting/test_reporters.py:2014
Methodtest_failed_run_produces_false_agent_eval_result
(self, tmp_path: Path)
tests/reporting/test_reporters.py:2085
Functiontest_failed_run_reservation_cleanup_refuses_a_substituted_empty_path
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_tier3_progress.py:4262
Functiontest_failed_run_whose_budget_left_attempts_not_run_reads_incomplete
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, blocking: bool, failure: str )
tests/reporting/test_tier3_completed_agent_docs.py:501
Functiontest_failed_runs_never_produce_zero_or_numeric_interaction
(tmp_path, failure)
tests/tier3/test_interactions.py:308
Functiontest_failed_scan_does_not_create_or_replace_saved_index
(tmp_path: Path, full_body, existing_output)
tests/embedding/test_pairwise_preflight.py:81
Functiontest_failed_security_score_explains_card_and_drilldown_cell
()
tests/reporting/test_tier3_failure_reasons.py:139
Functiontest_failed_sidecar_overrides_conflicting_reward_status_and_errors
(tmp_path: Path)
tests/test_issue55_collector_truth.py:2575
Functiontest_failed_skill_lookup_is_not_activation
(grader, content, name, error_metadata)
tests/tier3/test_failed_skill_reads.py:197
Functiontest_failed_stop_gets_a_final_kill_and_the_timeout_is_still_reported
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch )
tests/test_nemo_gym_backend.py:2521
Functiontest_failed_summary_cannot_display_a_passing_skill_report
()
tests/ci/test_publish_reports.py:1195
Methodtest_failed_summary_status_survives_reader
(self, tmp_path: Path)
tests/reporting/test_reporters.py:2061
Functiontest_failed_task_outcomes_keep_observed_deltas_and_disclosures
(tmp_path, reused_baseline)
tests/reporting/test_resource_comparison.py:120
Functiontest_failed_tier2_cannot_masquerade_as_a_clean_optional_skip
()
tests/golden/test_benchmark_card.py:728
Methodtest_failed_tier2_names_render_in_tier2_and_block
( self, validator_name: str, tmp_path: Path, )
tests/reporting/test_reporters.py:2159
Methodtest_failed_tier2_scan_is_reported_as_an_error_not_a_duplicate
(self, tmp_path: Path, monkeypatch)
tests/reporting/test_reporters.py:334
Methodtest_failed_tier2_without_structured_findings_still_blocks
(self)
tests/reporting/test_reporters.py:2192
Functiontest_failed_tier3_execution_status_overrides_pass_verdict_across_reporters
()
tests/reporting/test_advisory_skip_verdict.py:993
← previousnext →16,001–16,100 of 24,741, ranked by callers