MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_removing_version_label_is_rejected_with_previous_version
A previous-version bound cannot be bypassed by removing the current label.
tests/validators/test_version.py:261
Methodtest_render_agent_eval_missing_scores_are_unavailable
(self, scores, expected_cells)
tests/reporting/test_reporters.py:287
Methodtest_render_agent_eval_score_availability
( self, with_skill, expected_with_skill, baseline, expected_baseline, lift, expected_lift )
tests/reporting/test_reporters.py:259
Methodtest_render_agent_eval_with_unavailable_comparison
( self, baseline: float | None, lift: float | None, expected_baseline: str,
tests/reporting/test_reporters.py:233
Methodtest_render_all
Test rendering multiple results.
tests/reporting/test_reporters.py:369
Methodtest_render_all
Test JSON output for multiple results.
tests/reporting/test_reporters.py:528
Methodtest_render_all
Test HTML output for multiple results.
tests/reporting/test_reporters.py:2452
Methodtest_render_all
Test Markdown output for multiple results.
tests/reporting/test_reporters.py:2643
Methodtest_render_all_does_not_name_a_current_directory_that_holds_no_skill
( self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch )
tests/reporting/test_reporters.py:601
Methodtest_render_all_lists_the_html_top_issues_without_the_html_anchor
( self, mixed_results: list[ValidationResult] )
tests/reporting/test_reporters.py:626
Methodtest_render_all_marks_skipped_live_eval_as_advisory
(self)
tests/reporting/test_reporters.py:205
Methodtest_render_all_names_the_explicit_skill_of_a_clean_single_skill_run
(self)
tests/reporting/test_reporters.py:550
Methodtest_render_all_names_the_skill_at_a_local_target_path
(self, tmp_path: Path)
tests/reporting/test_reporters.py:561
Methodtest_render_all_names_the_skill_for_every_spelling_of_a_local_target
``.`` and ``..`` have no folder name of their own; the folder they open does.
tests/reporting/test_reporters.py:585
Methodtest_render_all_persists_resolved_and_per_result_benchmark_policy
(self)
tests/reporting/test_reporters.py:707
Methodtest_render_all_persists_typed_rubric_execution_metadata
(self)
tests/reporting/test_reporters.py:689
Methodtest_render_all_records_the_evaluator_version_and_target
(self, success_result: ValidationResult)
tests/reporting/test_reporters.py:614
Methodtest_render_all_shows_passing_findings_without_changing_pass_state
(self, severity: Severity)
tests/reporting/test_reporters.py:406
Methodtest_render_all_shows_passing_findings_without_changing_pass_state
(self, severity: Severity)
tests/reporting/test_reporters.py:2654
Methodtest_render_canonical_tier3_without_baseline
A normal skill-only run uses N/A rather than crashing on absent baseline scores.
tests/reporting/test_reporters.py:2683
Functiontest_render_drops_malformed_reasons_and_excerpts
()
tests/reporting/test_tier3_failure_reasons.py:736
Methodtest_render_escapes_markup_in_legacy_errors
(self)
tests/reporting/test_reporters.py:322
Methodtest_render_escapes_markup_in_success_details
(self)
tests/reporting/test_reporters.py:353
Functiontest_render_evaluation_result_invokes_findings_report
(monkeypatch)
tests/test_commands.py:3338
Functiontest_render_explicit_identity_never_inspects_artifact_paths
(payload)
tests/reporting/test_retained_html.py:72
Methodtest_render_failure
Test rendering a failed validation.
tests/reporting/test_reporters.py:195
Methodtest_render_failure
Test JSON output for failed validation.
tests/reporting/test_reporters.py:517
Methodtest_render_failure
Test HTML output for failed validation.
tests/reporting/test_reporters.py:2441
Methodtest_render_failure
Test Markdown output for failed validation.
tests/reporting/test_reporters.py:2634
Methodtest_render_failure_escapes_markup_in_finding_fields
Dynamic finding text should not be parsed as Rich markup.
tests/reporting/test_reporters.py:297
Methodtest_render_failure_produces_valid_sarif
(self, failure_result: ValidationResult)
tests/reporting/test_reporters.py:1537
Methodtest_render_finding_shows_both_related_paths
(self)
tests/reporting/test_reporters.py:415
Methodtest_render_finding_shows_both_related_paths
(self)
tests/reporting/test_reporters.py:2663
Methodtest_render_success
Test rendering a successful validation.
tests/reporting/test_reporters.py:185
Methodtest_render_success
Test JSON output for successful validation.
tests/reporting/test_reporters.py:507
Methodtest_render_success
Test HTML output for successful validation.
tests/reporting/test_reporters.py:2431
Methodtest_render_success
Test Markdown output for successful validation.
tests/reporting/test_reporters.py:2625
Methodtest_render_tier3_harbor_links_and_evidence
Tier 3 CLI output should include Harbor job, analysis, and step links.
tests/reporting/test_reporters.py:423
Methodtest_render_tier3_harbor_links_and_evidence
Tier 3 HTML should link Harbor logs, analysis, and exact trajectory steps.
tests/reporting/test_reporters.py:2461
Methodtest_render_tier3_harbor_links_and_evidence
Tier 3 Markdown should include compact Harbor and evidence links.
tests/reporting/test_reporters.py:2671
Methodtest_render_tier3_normalized_evidence_uses_distinct_evidence_ids
(self)
tests/reporting/test_reporters.py:2737
Functiontest_render_time_error_append_does_not_hide_a_recognizable_legacy_blob_copy
()
tests/reporting/test_insights_view.py:253
Functiontest_render_with_saved_feedback_skips_the_judge
(tmp_path: Path, llm_calls: list[str])
tests/tier3/test_view_results_feedback.py:261
Methodtest_rendered_card_satisfies_the_strict_gate
(self, tmp_path: Path, source: dict[str, str])
tests/test_benchmark_source_provenance.py:507
Functiontest_rendered_explanation_never_contradicts_the_computed_verdict
The printed decision is the status ``_overall_status`` computes.
tests/reporting/test_verdict_evidence.py:346
Functiontest_rendered_score_transitions_satisfy_the_gate_and_keep_verdict_bands
()
tests/golden/test_benchmark_card.py:1419
Functiontest_renderer_version_is_read_with_the_brand_instead_of_the_controls
()
tests/reporting/test_report_version_placement.py:43
Functiontest_rendering_retained_failed_evidence_does_not_rebuild_or_call_a_provider
(monkeypatch)
tests/reporting/test_insights_view.py:176
Functiontest_reordered_rows_compose_by_exact_skill_path
(tmp_path)
tests/ci/test_benchmark_reports.py:136
Functiontest_reparse_point_is_treated_as_unsafe
(monkeypatch, tmp_path: Path)
tests/embedding/test_extractor.py:338
Functiontest_repeated_attempts_do_not_change_distinct_case_count
()
tests/tier3/test_dataset_sufficiency.py:66
Functiontest_repeated_attempts_pair_by_case_with_distinct_trial_ids
()
tests/reporting/test_dashboard_data.py:435
Functiontest_repeated_cancellation_during_process_creation_still_reaps_launcher
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_local_mode.py:4031
Functiontest_repeated_canonical_model_overrides_are_rejected
()
tests/test_commands.py:210
Methodtest_repeated_dead_targets_are_reported_once
Repeated links do not generate duplicate findings without line metadata.
tests/validators/test_hygiene.py:262
Functiontest_repeated_finding_across_locations_and_validators_counts_one_highest_severity_issue
()
tests/ci/test_summary_metrics.py:99
Methodtest_repeated_frontmatter_key_is_still_stripped
(self, skill_root: Path)
tests/deduplication/utils/test_skill_collector.py:236
Methodtest_repeated_key_keeps_the_last_value_like_agent_clients
(self, tmp_path: Path)
tests/embedding/test_extractor.py:850
Functiontest_repeated_negated_skill_use_finishes_with_a_generous_resource_bound
()
tests/tier3/test_dataset_baseline_fairness.py:1726
Functiontest_repeated_pii_finding_id_survives_line_moves
(tmp_path)
tests/reporting/test_reporters.py:927
Functiontest_repeated_projection_preserves_coverage_case_and_trace_associations
()
tests/reporting/test_insights_view.py:369
Functiontest_repeated_ref_within_one_trial_remains_available
()
tests/tier3/test_suggestion_grounding.py:53
Methodtest_repeated_section_bodies_complete_through_real_chunking_and_clustering
( self, mock_embed, _mock_llm, mock_analyze, tmp_path: Path, headings: list[str], body: str, newline:
tests/deduplication/intra_skill/test_intra_skill_validator.py:596
Methodtest_repeated_text_is_shared_only_when_that_saves_characters
( self, make_chunk, length: int, shared: bool )
tests/deduplication/intra_skill/test_intra_llm_analyzer.py:185
Functiontest_repeated_unsafe_anchor_is_reported_once_per_document
(tmp_path: Path)
tests/validators/test_hygiene_link_boundaries.py:62
Methodtest_repeated_value_collapses_to_one_finding
(self, skill_with_repeated_emails)
tests/validators/test_pii_grouping.py:63
Functiontest_replaced_results_parent_does_not_bypass_no_follow_validation
(tmp_path)
tests/tier3/test_plugin_runtime_identity.py:205
Functiontest_replacement_is_atomic_and_preserves_mode
(repository, monkeypatch)
tests/test_ci_onboarding_writes.py:23
Functiontest_replacing_markdown_declarations_still_suppress_packaged_defaults
(tmp_path, profile)
tests/test_plugin_discovery.py:468
Functiontest_replay_refuses_cells_outside_the_display_schema
(tmp_path, value)
tests/reporting/test_private_runtime_replay.py:202
Functiontest_replay_refuses_incompatible_snapshot_schema
(tmp_path, change)
tests/reporting/test_private_runtime_replay.py:56
Functiontest_replay_refuses_snapshots_that_misstate_completeness_or_omissions
(tmp_path, change, reason)
tests/reporting/test_private_runtime_replay.py:156
Functiontest_replay_refuses_unrecognized_section_fields
(tmp_path)
tests/reporting/test_private_runtime_replay.py:210
Functiontest_replay_refuses_unsafe_selected_paths
(tmp_path, kind)
tests/reporting/test_private_runtime_replay.py:31
Functiontest_replay_rejects_invalid_or_unbounded_json
(tmp_path, content)
tests/reporting/test_private_runtime_replay.py:179
Functiontest_replay_rejects_snapshot_that_exceeds_byte_bound_after_sanitization
(tmp_path)
tests/reporting/test_private_runtime_replay.py:185
Methodtest_reply_fields_at_the_length_limit_are_kept_whole
(self, make_cluster)
tests/deduplication/intra_skill/test_intra_llm_analyzer.py:590
Methodtest_reply_nested_too_deep_to_parse_is_retried
( self, tmp_path, monkeypatch: pytest.MonkeyPatch, message, problem )
tests/validators/test_rubric_eval.py:1432
Methodtest_reply_that_is_not_a_json_object_is_rejected
(self, make_cluster, reply: object)
tests/deduplication/intra_skill/test_intra_llm_analyzer.py:613
Methodtest_reply_with_an_integer_past_the_digit_limit_is_retried
(self, tmp_path)
tests/validators/test_rubric_eval.py:1401
Methodtest_report_accounting_must_match_a_fully_inspected_bundle
(self, tmp_path: Path)
tests/validators/test_security.py:9891
Functiontest_report_activation_validation_uses_the_shared_trusted_helper
(monkeypatch)
tests/test_tier3_case_summary.py:1748
Functiontest_report_attempt_chip_uses_configured_attempt_threshold
()
tests/reporting/test_unified_tier3_report.py:310
Functiontest_report_bounds_reserve_identity_and_later_risk_evidence
()
tests/reporting/test_private_static_reports.py:359
Functiontest_report_coerces_and_bounds_non_string_reasons_from_existing_artifacts
Coerce and bound non-string reason fields from legacy judge artifacts.
tests/tier3/test_judge_failure_artifacts.py:1183
Functiontest_report_counts_status_and_completeness_must_agree
(change)
tests/ci/test_publish_reports.py:1188
Functiontest_report_default_theme_is_dark_for_tier3_and_light_for_static
()
tests/reporting/test_native_report_parity.py:57
Functiontest_report_directory_cannot_modify_consumer_checkout
(consumer)
tests/ci/test_run_tiers.py:231
Functiontest_report_discovery_handles_timestamped_results
(tmp_path: Path)
tests/test_evaluation_service.py:1063
Functiontest_report_discovery_returns_none_when_absent
(tmp_path: Path)
tests/test_evaluation_service.py:1081
Methodtest_report_filter_uses_scanned_filesystem_case_semantics
(self, mock_run, tmp_path: Path)
tests/validators/test_artifact_dir_exclusion.py:169
Functiontest_report_handles_incomplete_judge_evidence
(tmp_path: Path, missing_evidence: str)
tests/test_judge_lift_integrity.py:232
Functiontest_report_ids_are_unique
()
tests/reporting/test_report_redesign.py:216
Functiontest_report_interpretation_with_a_direct_negative_subject_does_not_warn
(text)
tests/tier3/test_dataset_baseline_fairness.py:1752
Functiontest_report_leads_with_skill_effect_and_goal_not_met_fix
(tmp_path)
tests/reporting/test_case_findings_report.py:48
Functiontest_report_loader_marks_missing_trajectory_steps_unavailable
(tmp_path: Path)
tests/reporting/test_report_data_bounds.py:189
Functiontest_report_loader_marks_missing_trajectory_token_counters_unavailable
(tmp_path: Path)
tests/reporting/test_report_data_bounds.py:169
Functiontest_report_loader_omits_unrepresentable_trajectory_token_counters
(tmp_path: Path)
tests/reporting/test_report_data_bounds.py:142
Functiontest_report_loading_validates_without_copying_metadata
(tmp_path, monkeypatch)
tests/ci/test_run_tier1.py:587
Functiontest_report_metric_discovery_honors_explicit_custom_only_declaration
()
tests/reporting/test_unified_tier3_report.py:534
← previousnext →21,101–21,200 of 24,741, ranked by callers