MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_validate_same_entry_alias_expects_recorded_publication_name
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, recorded_name: str, alias_name: str, )
tests/test_commands.py:791
Methodtest_validate_scripts_missing_documentation
Test validation warns about script without documentation.
tests/validators/test_workflows_schema.py:587
Methodtest_validate_scripts_missing_shebang
Test validation warns about shell script without shebang.
tests/validators/test_workflows_schema.py:548
Methodtest_validate_scripts_with_valid_shell_script
Test validation of valid shell script with shebang.
tests/validators/test_workflows_schema.py:505
Methodtest_validate_security_only
Test validate_security_only method runs only skillspector scan.
tests/validators/test_security.py:6333
Methodtest_validate_security_only_with_llm
Test validate_security_only with LLM runs CLI without --no-llm.
tests/validators/test_security.py:6364
Methodtest_validate_security_only_without_llm
Test validate_security_only without LLM runs CLI with --no-llm.
tests/validators/test_security.py:6589
Methodtest_validate_semgrep_crash_is_not_a_clean_pass
A Semgrep fatal error must fail validation, not report a clean scan.
tests/validators/test_security_validators.py:607
Methodtest_validate_semgrep_findings
Test Semgrep analysis with findings.
tests/validators/test_security_validators.py:885
Methodtest_validate_semgrep_timeout_is_not_a_clean_pass
A Semgrep execution failure must fail validation.
tests/validators/test_security_validators.py:579
Functiontest_validate_shows_panel_and_verdict_note
(tmp_path: Path)
tests/harness/test_overlay.py:293
Functiontest_validate_shows_source_errors_once
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, blocking: bool, report_formats: str | None,
tests/reporting/test_tier3_source_validation_reporting.py:479
Functiontest_validate_skill_evals_accepts_placeholder_expected_script_as_explicit_no_script
(tmp_path)
tests/tier3/test_evals_config.py:684
Functiontest_validate_skill_evals_does_not_warn_expected_script_for_guide_only_skill
(tmp_path)
tests/tier3/test_evals_config.py:656
Functiontest_validate_skill_evals_warns_expected_script_when_scripts_exist
(tmp_path)
tests/tier3/test_evals_config.py:669
Methodtest_validate_skill_with_secrets
Test detection of secrets in a skill.
tests/validators/test_security_validators.py:97
Methodtest_validate_team_rules_folder
Test validation of team-rules folder structure.
tests/validators/test_rules_schema.py:316
Methodtest_validate_team_rules_project_subfolder
Test validation of team-rules with project subfolder.
tests/validators/test_rules_schema.py:340
Methodtest_validate_team_workflow_structure
Test validation of team-workflows structure.
tests/validators/test_workflows_schema.py:462
Functiontest_validate_terminal_prints_agent_rows_and_notes
(run_result: tuple[Path, Any])
tests/reporting/test_tier3_token_usage_reports.py:821
Functiontest_validate_terminal_shows_coverage
()
tests/reporting/test_declarative_coverage.py:212
Functiontest_validate_text_json_and_strict_info
(skill)
tests/tier3/test_dataset_coverage.py:87
Functiontest_validate_tier1_banner_is_stable_in_narrow_terminal
(monkeypatch, tmp_path: Path)
tests/test_commands.py:288
Functiontest_validate_tier2_default_is_blocking_and_can_be_advisory
(monkeypatch)
tests/test_commands.py:2872
Functiontest_validate_tier3_allows_report_output_elsewhere_in_included_skill_repo
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_commands.py:1579
Functiontest_validate_tier3_rejects_authored_report_output_inside_included_skill
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_commands.py:1532
Functiontest_validate_tier3_source_reports_config_errors_instead_of_crashing
(tmp_path, content, message)
tests/tier3/test_evals_config.py:351
Functiontest_validate_tier3_writes_the_same_report_pair
(monkeypatch, tmp_path: Path)
tests/reporting/test_tier3_completed_agent_docs.py:298
Functiontest_validate_tier_aliases_and_selector
(monkeypatch)
tests/test_commands.py:1354
Functiontest_validate_tiers_selector_does_not_create_publication_waiver
(monkeypatch, tmp_path: Path)
tests/test_commands.py:2935
Functiontest_validate_tiers_selector_is_authoritative_over_full
(monkeypatch)
tests/test_commands.py:3501
Functiontest_validate_tiers_without_tier1_is_rejected_before_any_tier_runs
(monkeypatch, tmp_path, tiers)
tests/cli/test_validate_defaults.py:60
Functiontest_validate_uses_packaged_default_quality_config
(monkeypatch: pytest.MonkeyPatch, tmp_path: Path)
tests/test_commands.py:1007
Methodtest_validate_valid_rule
Test validation passes for valid rule file.
tests/validators/test_rules_schema.py:94
Methodtest_validate_valid_workflow
Test validation passes for valid workflow directory.
tests/validators/test_workflows_schema.py:166
Functiontest_validate_view_strips_terminal_controls
( capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch )
tests/reporting/test_rich_markup_escaping.py:686
Methodtest_validate_with_requirements
Test validation with requirements.txt.
tests/validators/test_security_validators.py:173
Methodtest_validate_with_vulnerabilities
Test detection of vulnerable dependencies.
tests/validators/test_security_validators.py:203
Methodtest_validate_wrong_extension
Test validation fails for non-.mdc file.
tests/validators/test_rules_schema.py:177
Methodtest_validates_multiple_skills
Test validation of folder with multiple skills.
tests/validators/test_license.py:1084
Functiontest_validation_checks_directory_and_frontmatter_names
(tmp_path, expected_skill, field, own_name)
tests/tier3/test_dataset_baseline_fairness.py:781
Methodtest_validation_result_add_finding
Test ValidationResult.add_finding() method.
tests/validators/test_security_validators.py:1019
Methodtest_validation_result_merge_includes_findings
Test that merge() combines findings from both results.
tests/validators/test_security_validators.py:1139
Methodtest_validation_result_merge_with_prefix_for_findings
Test merge_with_prefix() prefixes finding file paths.
tests/validators/test_security_validators.py:1170
Functiontest_validation_without_copies_matches_the_copying_rehydration
(payload)
tests/reporting/test_retained_html.py:222
Functiontest_validator_accepts_generation_and_case_review_metadata
(tmp_path)
tests/tier3/test_dataset_grounding.py:559
Functiontest_validator_distinguishes_complete_from_incomplete_no_matches
( tmp_path: Path, monkeypatch, malformed: int )
tests/embedding/test_collection_coverage.py:57
Methodtest_validator_name
(self)
tests/validators/test_script_lint.py:230
Methodtest_validator_name_and_description
(self)
tests/validators/test_quality_score.py:659
Methodtest_validator_properties
Test validator name and description.
tests/validators/test_license.py:294
Methodtest_validators_export_from_init
Test that new validators are exported from __init__.
tests/validators/test_security_validators.py:951
Methodtest_validators_have_consistent_interface
Test all validators implement ValidatorBase interface.
tests/validators/test_security_validators.py:963
Methodtest_value_already_in_prompt_is_not_attributable
(self)
tests/tier3/test_plugin_signals.py:918
Methodtest_value_flows_from_producer_output_to_later_consumer_input
(self)
tests/tier3/test_plugin_signals.py:909
Functiontest_value_in_the_real_task_prompt_still_fails
()
tests/tier3/test_f90_signals_handoff_prompt.py:98
Functiontest_values_that_are_not_jwt_like_stay_clean
(value: str)
tests/validators/test_mcp_static.py:848
Functiontest_values_with_quotes_and_backslashes_pass_and_a_prefix_of_a_longer_id_does_not
( trajectory: dict[str, Any], value: str, passes: bool )
tests/tier3/test_f90_signals_handoff.py:133
Functiontest_verbose_evaluate_output_lists_the_refused_trial
(tmp_path: Path, skip_baseline: bool)
tests/test_harbor_claude_refusal_fallback.py:387
Functiontest_verbose_options_still_run_the_script
``ruby -v`` prints its version and runs the script; ``ruby --version`` does not.
tests/tier3/test_script_invocation_evidence.py:360
Functiontest_verbose_validate_without_file_reports_lists_every_source_error
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, blocking: bool, )
tests/reporting/test_tier3_source_validation_reporting.py:518
Functiontest_verdict_badge_background_does_not_fill_headline
()
tests/reporting/test_terminal_polish.py:439
Functiontest_verdict_callout_falls_back_for_future_statuses
()
tests/golden/test_benchmark_card.py:1132
Functiontest_verdict_comes_from_the_complete_peer_not_a_tolerated_agent
( tmp_path: Path, peer: str, verdict: str, lead: str )
tests/tier3/test_baseline_timeouts.py:869
Functiontest_verdict_counts_come_from_a_complete_agent_that_set_the_verdict
(tmp_path: Path)
tests/tier3/test_baseline_timeouts.py:1046
Methodtest_verdict_names_no_model_unless_the_client_gives_a_string
(self, make_cluster, model_used: object)
tests/deduplication/intra_skill/test_intra_llm_analyzer.py:646
Functiontest_verdict_prompt_accepts_bounded_long_names_and_newlines
(modules, tmp_path, monkeypatch)
tests/deduplication/plugin/test_private_dedup.py:315
Functiontest_verdict_reads_only_with_skill_na_dimensions
(tmp_path, with_na, verdict, with_skill_na_dimension)
tests/reporting/test_applicability_saved_runs.py:374
Methodtest_verdict_records_the_model_that_answered
(self, make_cluster)
tests/deduplication/intra_skill/test_intra_llm_analyzer.py:627
Functiontest_verdict_thresholds
()
tests/test_tier3_case_summary.py:774
Methodtest_verdict_thresholds
(self, evaluators: dict)
tests/evaluation/test_judges.py:60
Methodtest_verdict_without_a_model_adds_no_model_fields
( self, mock_embed, mock_llm, mock_analyze, tmp_path: Path )
tests/deduplication/intra_skill/test_intra_skill_validator.py:1491
Methodtest_verdicts_kept_from_a_truncated_reply_name_the_model_that_wrote_it
( self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path )
tests/inference/test_llm_client.py:1740
Functiontest_verification_reasoning_is_inert_and_bounded
(format_name: str)
tests/reporting/test_finding_verification.py:167
Functiontest_verified_facts_preserve_final_response_in_behavior_judge_prompt
(monkeypatch)
tests/tier3/test_behavior_evidence.py:1151
Functiontest_verified_finding_shows_reasoning_and_actual_gate
Dropping metadata in a reporter or lowering the confidence threshold breaks this.
tests/reporting/test_finding_verification.py:95
Methodtest_verified_image_paths_keep_scanner_findings
(self, mock_tools, sample_skill_dir: Path)
tests/validators/test_security.py:8933
Methodtest_verified_png_outside_agent_control_directories
( self, mock_tools, sample_skill_dir: Path, path: str )
tests/validators/test_security.py:9037
Methodtest_verified_png_reference_completes_the_llm_stage
(self, mock_tools, sample_skill_dir: Path)
tests/validators/test_security.py:8953
Methodtest_verified_png_reference_completes_the_scan
(self, mock_tools, sample_skill_dir: Path, fixture: str)
tests/validators/test_security.py:8917
Functiontest_verified_recovery_keeps_successful_statistics_and_all_physical_usage
(tmp_path)
tests/tier3/test_private_recovered_timeout_review2.py:53
Functiontest_verifier_401_does_not_retry_non_vertex_endpoints
Verify verifier template does not attempt ADC refresh on HTTP 401 for non-Vertex endpoints.
tests/test_tier3_public_runtime.py:2005
Functiontest_verifier_401_does_not_switch_principals_for_explicit_vertex_credential
Do not replace an explicitly supplied Vertex OpenAPI token with ambient ADC on HTTP 401.
tests/test_tier3_public_runtime.py:1958
Functiontest_verifier_and_reporter_share_downgrade_policy_with_coverage_exemption
A changed rule must drive both actual downgrade behavior and its explanation.
tests/reporting/test_finding_verification.py:256
Functiontest_verifier_case_id_mismatches_are_unscored_by_shared_collector
(tmp_path: Path)
tests/test_nemo_gym_backend.py:197
Functiontest_verifier_constraint_and_skill_requirements_share_resolver_transaction
(tmp_path: Path)
tests/test_harbor_runtime_skill_isolation.py:4221
Functiontest_verifier_dependency_smoke_covers_every_verifier_third_party_import
()
tests/test_harbor_runtime_skill_isolation.py:4299
Functiontest_verifier_dependency_smoke_fails_when_isolated_verifier_cannot_import
(tmp_path: Path)
tests/test_harbor_runtime_skill_isolation.py:4377
Functiontest_verifier_dependency_smoke_runs_like_the_verifier
(tmp_path: Path, mode: str, grading_mode: str)
tests/test_harbor_runtime_skill_isolation.py:4342
Functiontest_verifier_env_block_forwards_retry_settings
Verify _verifier_env_block generates task.toml env lines for staged retry variables.
tests/test_harbor_task_toml_retry_env.py:43
Functiontest_verifier_env_vars_and_runner_include_budget_vars
(monkeypatch)
tests/tier3/test_evidence_compiler.py:136
Functiontest_verifier_exception_does_not_scan_plain_agent_transcript
Correct agent prose must not turn a verifier failure into an agent failure.
tests/test_harbor_collector_runtime_failures.py:719
Functiontest_verifier_git_identity_requires_tracked_files
(tmp_path)
tests/tier3/test_grader_validation.py:1029
Functiontest_verifier_git_ignores_unrelated_dirty_files_and_git_environment
(tmp_path, monkeypatch)
tests/tier3/test_grader_validation.py:1099
Functiontest_verifier_judge_env_stages_templates_not_values
(tmp_path: Path, monkeypatch)
tests/test_nemo_gym_backend.py:566
Methodtest_verifier_keeps_complete_verdicts_from_a_truncated_reply
( self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path, provider: str, model: str, make_response
tests/inference/test_llm_client.py:1306
Functiontest_verifier_main_fails_closed_after_collecting_every_required_judge
Verify verifier main fails closed with exit code 1 after collecting every required judge error.
tests/tier3/test_judge_failure_artifacts.py:232
Functiontest_verifier_main_keeps_absent_references_scoreless_in_numeric_rewards
Keep N/A facts in the sidecar and require them when interpreting numeric rewards.
tests/tier3/test_judge_failure_artifacts.py:1275
Functiontest_verifier_main_keeps_accuracy_fail_closed_after_retry_exhaustion
Keep accuracy metric fail-closed when verifier retries are exhausted.
tests/tier3/test_judge_failure_artifacts.py:1222
Functiontest_verifier_main_keeps_genuine_zero_judge_verdicts_scoreable
Retain genuine zero-score verdicts from successful judge evaluations as scoreable metrics.
tests/tier3/test_judge_failure_artifacts.py:977
Functiontest_verifier_main_normalizes_malformed_or_raised_judge_failures_and_continues
Normalize malformed or raised judge failures and continue evaluating remaining judges.
tests/tier3/test_judge_failure_artifacts.py:1299
← previousnext →24,101–24,200 of 24,741, ranked by callers