Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/NVIDIA/SkillEvaluator
/ functions
Functions
24,741 in github.com/NVIDIA/SkillEvaluator
⨍
Functions
24,741
◇
Types & classes
1,292
↳
Endpoints
703
Function
test_validate_same_entry_alias_expects_recorded_publication_name
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, recorded_name: str, alias_name: str, )
tests/test_commands.py:791
Method
test_validate_scripts_missing_documentation
Test validation warns about script without documentation.
tests/validators/test_workflows_schema.py:587
Method
test_validate_scripts_missing_shebang
Test validation warns about shell script without shebang.
tests/validators/test_workflows_schema.py:548
Method
test_validate_scripts_with_valid_shell_script
Test validation of valid shell script with shebang.
tests/validators/test_workflows_schema.py:505
Method
test_validate_security_only
Test validate_security_only method runs only skillspector scan.
tests/validators/test_security.py:6333
Method
test_validate_security_only_with_llm
Test validate_security_only with LLM runs CLI without --no-llm.
tests/validators/test_security.py:6364
Method
test_validate_security_only_without_llm
Test validate_security_only without LLM runs CLI with --no-llm.
tests/validators/test_security.py:6589
Method
test_validate_semgrep_crash_is_not_a_clean_pass
A Semgrep fatal error must fail validation, not report a clean scan.
tests/validators/test_security_validators.py:607
Method
test_validate_semgrep_findings
Test Semgrep analysis with findings.
tests/validators/test_security_validators.py:885
Method
test_validate_semgrep_timeout_is_not_a_clean_pass
A Semgrep execution failure must fail validation.
tests/validators/test_security_validators.py:579
Function
test_validate_shows_panel_and_verdict_note
(tmp_path: Path)
tests/harness/test_overlay.py:293
Function
test_validate_shows_source_errors_once
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, blocking: bool, report_formats: str | None,
tests/reporting/test_tier3_source_validation_reporting.py:479
Function
test_validate_skill_evals_accepts_placeholder_expected_script_as_explicit_no_script
(tmp_path)
tests/tier3/test_evals_config.py:684
Function
test_validate_skill_evals_does_not_warn_expected_script_for_guide_only_skill
(tmp_path)
tests/tier3/test_evals_config.py:656
Function
test_validate_skill_evals_warns_expected_script_when_scripts_exist
(tmp_path)
tests/tier3/test_evals_config.py:669
Method
test_validate_skill_with_secrets
Test detection of secrets in a skill.
tests/validators/test_security_validators.py:97
Method
test_validate_team_rules_folder
Test validation of team-rules folder structure.
tests/validators/test_rules_schema.py:316
Method
test_validate_team_rules_project_subfolder
Test validation of team-rules with project subfolder.
tests/validators/test_rules_schema.py:340
Method
test_validate_team_workflow_structure
Test validation of team-workflows structure.
tests/validators/test_workflows_schema.py:462
Function
test_validate_terminal_prints_agent_rows_and_notes
(run_result: tuple[Path, Any])
tests/reporting/test_tier3_token_usage_reports.py:821
Function
test_validate_terminal_shows_coverage
()
tests/reporting/test_declarative_coverage.py:212
Function
test_validate_text_json_and_strict_info
(skill)
tests/tier3/test_dataset_coverage.py:87
Function
test_validate_tier1_banner_is_stable_in_narrow_terminal
(monkeypatch, tmp_path: Path)
tests/test_commands.py:288
Function
test_validate_tier2_default_is_blocking_and_can_be_advisory
(monkeypatch)
tests/test_commands.py:2872
Function
test_validate_tier3_allows_report_output_elsewhere_in_included_skill_repo
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_commands.py:1579
Function
test_validate_tier3_rejects_authored_report_output_inside_included_skill
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_commands.py:1532
Function
test_validate_tier3_source_reports_config_errors_instead_of_crashing
(tmp_path, content, message)
tests/tier3/test_evals_config.py:351
Function
test_validate_tier3_writes_the_same_report_pair
(monkeypatch, tmp_path: Path)
tests/reporting/test_tier3_completed_agent_docs.py:298
Function
test_validate_tier_aliases_and_selector
(monkeypatch)
tests/test_commands.py:1354
Function
test_validate_tiers_selector_does_not_create_publication_waiver
(monkeypatch, tmp_path: Path)
tests/test_commands.py:2935
Function
test_validate_tiers_selector_is_authoritative_over_full
(monkeypatch)
tests/test_commands.py:3501
Function
test_validate_tiers_without_tier1_is_rejected_before_any_tier_runs
(monkeypatch, tmp_path, tiers)
tests/cli/test_validate_defaults.py:60
Function
test_validate_uses_packaged_default_quality_config
(monkeypatch: pytest.MonkeyPatch, tmp_path: Path)
tests/test_commands.py:1007
Method
test_validate_valid_rule
Test validation passes for valid rule file.
tests/validators/test_rules_schema.py:94
Method
test_validate_valid_workflow
Test validation passes for valid workflow directory.
tests/validators/test_workflows_schema.py:166
Function
test_validate_view_strips_terminal_controls
( capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch )
tests/reporting/test_rich_markup_escaping.py:686
Method
test_validate_with_requirements
Test validation with requirements.txt.
tests/validators/test_security_validators.py:173
Method
test_validate_with_vulnerabilities
Test detection of vulnerable dependencies.
tests/validators/test_security_validators.py:203
Method
test_validate_wrong_extension
Test validation fails for non-.mdc file.
tests/validators/test_rules_schema.py:177
Method
test_validates_multiple_skills
Test validation of folder with multiple skills.
tests/validators/test_license.py:1084
Function
test_validation_checks_directory_and_frontmatter_names
(tmp_path, expected_skill, field, own_name)
tests/tier3/test_dataset_baseline_fairness.py:781
Method
test_validation_result_add_finding
Test ValidationResult.add_finding() method.
tests/validators/test_security_validators.py:1019
Method
test_validation_result_merge_includes_findings
Test that merge() combines findings from both results.
tests/validators/test_security_validators.py:1139
Method
test_validation_result_merge_with_prefix_for_findings
Test merge_with_prefix() prefixes finding file paths.
tests/validators/test_security_validators.py:1170
Function
test_validation_without_copies_matches_the_copying_rehydration
(payload)
tests/reporting/test_retained_html.py:222
Function
test_validator_accepts_generation_and_case_review_metadata
(tmp_path)
tests/tier3/test_dataset_grounding.py:559
Function
test_validator_distinguishes_complete_from_incomplete_no_matches
( tmp_path: Path, monkeypatch, malformed: int )
tests/embedding/test_collection_coverage.py:57
Method
test_validator_name
(self)
tests/validators/test_script_lint.py:230
Method
test_validator_name_and_description
(self)
tests/validators/test_quality_score.py:659
Method
test_validator_properties
Test validator name and description.
tests/validators/test_license.py:294
Method
test_validators_export_from_init
Test that new validators are exported from __init__.
tests/validators/test_security_validators.py:951
Method
test_validators_have_consistent_interface
Test all validators implement ValidatorBase interface.
tests/validators/test_security_validators.py:963
Method
test_value_already_in_prompt_is_not_attributable
(self)
tests/tier3/test_plugin_signals.py:918
Method
test_value_flows_from_producer_output_to_later_consumer_input
(self)
tests/tier3/test_plugin_signals.py:909
Function
test_value_in_the_real_task_prompt_still_fails
()
tests/tier3/test_f90_signals_handoff_prompt.py:98
Function
test_values_that_are_not_jwt_like_stay_clean
(value: str)
tests/validators/test_mcp_static.py:848
Function
test_values_with_quotes_and_backslashes_pass_and_a_prefix_of_a_longer_id_does_not
( trajectory: dict[str, Any], value: str, passes: bool )
tests/tier3/test_f90_signals_handoff.py:133
Function
test_verbose_evaluate_output_lists_the_refused_trial
(tmp_path: Path, skip_baseline: bool)
tests/test_harbor_claude_refusal_fallback.py:387
Function
test_verbose_options_still_run_the_script
``ruby -v`` prints its version and runs the script; ``ruby --version`` does not.
tests/tier3/test_script_invocation_evidence.py:360
Function
test_verbose_validate_without_file_reports_lists_every_source_error
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, blocking: bool, )
tests/reporting/test_tier3_source_validation_reporting.py:518
Function
test_verdict_badge_background_does_not_fill_headline
()
tests/reporting/test_terminal_polish.py:439
Function
test_verdict_callout_falls_back_for_future_statuses
()
tests/golden/test_benchmark_card.py:1132
Function
test_verdict_comes_from_the_complete_peer_not_a_tolerated_agent
( tmp_path: Path, peer: str, verdict: str, lead: str )
tests/tier3/test_baseline_timeouts.py:869
Function
test_verdict_counts_come_from_a_complete_agent_that_set_the_verdict
(tmp_path: Path)
tests/tier3/test_baseline_timeouts.py:1046
Method
test_verdict_names_no_model_unless_the_client_gives_a_string
(self, make_cluster, model_used: object)
tests/deduplication/intra_skill/test_intra_llm_analyzer.py:646
Function
test_verdict_prompt_accepts_bounded_long_names_and_newlines
(modules, tmp_path, monkeypatch)
tests/deduplication/plugin/test_private_dedup.py:315
Function
test_verdict_reads_only_with_skill_na_dimensions
(tmp_path, with_na, verdict, with_skill_na_dimension)
tests/reporting/test_applicability_saved_runs.py:374
Method
test_verdict_records_the_model_that_answered
(self, make_cluster)
tests/deduplication/intra_skill/test_intra_llm_analyzer.py:627
Function
test_verdict_thresholds
()
tests/test_tier3_case_summary.py:774
Method
test_verdict_thresholds
(self, evaluators: dict)
tests/evaluation/test_judges.py:60
Method
test_verdict_without_a_model_adds_no_model_fields
( self, mock_embed, mock_llm, mock_analyze, tmp_path: Path )
tests/deduplication/intra_skill/test_intra_skill_validator.py:1491
Method
test_verdicts_kept_from_a_truncated_reply_name_the_model_that_wrote_it
( self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path )
tests/inference/test_llm_client.py:1740
Function
test_verification_reasoning_is_inert_and_bounded
(format_name: str)
tests/reporting/test_finding_verification.py:167
Function
test_verified_facts_preserve_final_response_in_behavior_judge_prompt
(monkeypatch)
tests/tier3/test_behavior_evidence.py:1151
Function
test_verified_finding_shows_reasoning_and_actual_gate
Dropping metadata in a reporter or lowering the confidence threshold breaks this.
tests/reporting/test_finding_verification.py:95
Method
test_verified_image_paths_keep_scanner_findings
(self, mock_tools, sample_skill_dir: Path)
tests/validators/test_security.py:8933
Method
test_verified_png_outside_agent_control_directories
( self, mock_tools, sample_skill_dir: Path, path: str )
tests/validators/test_security.py:9037
Method
test_verified_png_reference_completes_the_llm_stage
(self, mock_tools, sample_skill_dir: Path)
tests/validators/test_security.py:8953
Method
test_verified_png_reference_completes_the_scan
(self, mock_tools, sample_skill_dir: Path, fixture: str)
tests/validators/test_security.py:8917
Function
test_verified_recovery_keeps_successful_statistics_and_all_physical_usage
(tmp_path)
tests/tier3/test_private_recovered_timeout_review2.py:53
Function
test_verifier_401_does_not_retry_non_vertex_endpoints
Verify verifier template does not attempt ADC refresh on HTTP 401 for non-Vertex endpoints.
tests/test_tier3_public_runtime.py:2005
Function
test_verifier_401_does_not_switch_principals_for_explicit_vertex_credential
Do not replace an explicitly supplied Vertex OpenAPI token with ambient ADC on HTTP 401.
tests/test_tier3_public_runtime.py:1958
Function
test_verifier_and_reporter_share_downgrade_policy_with_coverage_exemption
A changed rule must drive both actual downgrade behavior and its explanation.
tests/reporting/test_finding_verification.py:256
Function
test_verifier_case_id_mismatches_are_unscored_by_shared_collector
(tmp_path: Path)
tests/test_nemo_gym_backend.py:197
Function
test_verifier_constraint_and_skill_requirements_share_resolver_transaction
(tmp_path: Path)
tests/test_harbor_runtime_skill_isolation.py:4221
Function
test_verifier_dependency_smoke_covers_every_verifier_third_party_import
()
tests/test_harbor_runtime_skill_isolation.py:4299
Function
test_verifier_dependency_smoke_fails_when_isolated_verifier_cannot_import
(tmp_path: Path)
tests/test_harbor_runtime_skill_isolation.py:4377
Function
test_verifier_dependency_smoke_runs_like_the_verifier
(tmp_path: Path, mode: str, grading_mode: str)
tests/test_harbor_runtime_skill_isolation.py:4342
Function
test_verifier_env_block_forwards_retry_settings
Verify _verifier_env_block generates task.toml env lines for staged retry variables.
tests/test_harbor_task_toml_retry_env.py:43
Function
test_verifier_env_vars_and_runner_include_budget_vars
(monkeypatch)
tests/tier3/test_evidence_compiler.py:136
Function
test_verifier_exception_does_not_scan_plain_agent_transcript
Correct agent prose must not turn a verifier failure into an agent failure.
tests/test_harbor_collector_runtime_failures.py:719
Function
test_verifier_git_identity_requires_tracked_files
(tmp_path)
tests/tier3/test_grader_validation.py:1029
Function
test_verifier_git_ignores_unrelated_dirty_files_and_git_environment
(tmp_path, monkeypatch)
tests/tier3/test_grader_validation.py:1099
Function
test_verifier_judge_env_stages_templates_not_values
(tmp_path: Path, monkeypatch)
tests/test_nemo_gym_backend.py:566
Method
test_verifier_keeps_complete_verdicts_from_a_truncated_reply
( self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path, provider: str, model: str, make_response
tests/inference/test_llm_client.py:1306
Function
test_verifier_main_fails_closed_after_collecting_every_required_judge
Verify verifier main fails closed with exit code 1 after collecting every required judge error.
tests/tier3/test_judge_failure_artifacts.py:232
Function
test_verifier_main_keeps_absent_references_scoreless_in_numeric_rewards
Keep N/A facts in the sidecar and require them when interpreting numeric rewards.
tests/tier3/test_judge_failure_artifacts.py:1275
Function
test_verifier_main_keeps_accuracy_fail_closed_after_retry_exhaustion
Keep accuracy metric fail-closed when verifier retries are exhausted.
tests/tier3/test_judge_failure_artifacts.py:1222
Function
test_verifier_main_keeps_genuine_zero_judge_verdicts_scoreable
Retain genuine zero-score verdicts from successful judge evaluations as scoreable metrics.
tests/tier3/test_judge_failure_artifacts.py:977
Function
test_verifier_main_normalizes_malformed_or_raised_judge_failures_and_continues
Normalize malformed or raised judge failures and continue evaluating remaining judges.
tests/tier3/test_judge_failure_artifacts.py:1299
← previous
next →
24,101–24,200 of 24,741, ranked by callers