MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Methodtest_pull_request_merge_ref_uses_secondary_checkout_head
Source links use the target checkout even when workflow metadata describes another repository.
tests/utils/test_helpers.py:332
Functiontest_pull_requests_read_but_never_write_the_layer_cache
()
tests/ci/test_image_build.py:135
Functiontest_punctuation_heredoc_body_cannot_change_restored_runtime_user
(tmp_path: Path)
tests/test_harbor_runtime_skill_isolation.py:3756
Functiontest_push_error_suggests_workflow_permission_without_exposing_stderr
(monkeypatch, tmp_path)
tests/test_ci_onboarding_git.py:115
Functiontest_pyproject_advisory_names_the_declared_pin_and_group
(tmp_path, pip_audit)
tests/validators/test_dependency_advisory_findings.py:216
Functiontest_pyproject_optional_groups_and_mcp_pins_are_audited
(audit, tmp_path)
tests/validators/test_private_dependency_audit.py:97
Functiontest_pyproject_without_applicable_pins_records_execution_evidence
(tmp_path, monkeypatch)
tests/validators/test_target_dependencies.py:188
Functiontest_python_advisories_are_structured_counted_and_attributed
(audit, tmp_path, vulnerability, severity)
tests/validators/test_private_dependency_audit.py:508
Methodtest_python_comment_email_is_not_flagged
Python hash comments remain skipped so they are not newly flagged as PII.
tests/validators/test_security.py:715
Methodtest_python_comment_in_markdown_fence_is_not_flagged
Python comments inside a fenced Markdown block stay skipped.
tests/validators/test_security.py:950
Functiontest_python_module_docstring_uses_actual_ast_lines
()
tests/deduplication/utils/test_chunk_bounds.py:39
Functiontest_python_modules_a_hook_script_imports_are_followed
(tmp_path: Path, statement: str, module: str)
tests/validators/test_plugin_static_risk_review_fixes.py:416
Functiontest_python_runner_arbitrary_equality_pin_is_audited
(audit, tmp_path)
tests/validators/test_private_dependency_audit.py:332
Functiontest_python_script_skill_still_needs_run_script
(tmp_path)
tests/validators/test_quality_score.py:2176
Functiontest_python_valid_empty_advisory_report_stays_complete
(audit, tmp_path)
tests/validators/test_private_dependency_audit.py:530
Methodtest_python_write_outside_run_root_blocked
(self, run_root: Path, tmp_path: Path)
tests/test_harbor_local_sandbox.py:1440
Functiontest_pythonhttpsverify_empty_or_false_not_flagged
()
tests/validators/test_mcp_static.py:806
Functiontest_pythonhttpsverify_zero_flagged
()
tests/validators/test_mcp_static.py:815
Functiontest_qualification_runs_anchored_output_check
(judge_server, suite_dir, tmp_path, response, passed)
tests/tier3/test_grader_validation.py:143
Functiontest_quality_check_applies_custom_quality_config
( command: list[str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_commands.py:949
Functiontest_quality_check_cli_writes_junit_xml
(tmp_path: Path)
tests/reporting/test_junit_reporter.py:606
Functiontest_quality_check_rejects_missing_quality_config
(command: list[str], tmp_path: Path)
tests/test_commands.py:967
Functiontest_quality_check_uses_packaged_default_quality_config
( command: list[str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, )
tests/test_commands.py:936
Functiontest_quality_config_can_be_overridden
(tmp_path: Path)
tests/test_quality_config.py:46
Functiontest_quality_config_digest_covers_error_handling
(tmp_path: Path)
tests/test_quality_config.py:358
Functiontest_quality_config_digest_ignores_formatting_but_not_rules
(tmp_path: Path)
tests/test_quality_config.py:220
Functiontest_quality_config_errors_name_the_file_and_yaml_position
(tmp_path: Path)
tests/test_quality_config.py:151
Functiontest_quality_config_generates_message_and_suggestion_when_omitted
(tmp_path: Path)
tests/test_quality_config.py:100
Functiontest_quality_config_provenance_names_custom_file_without_its_directory
(tmp_path: Path)
tests/test_quality_config.py:238
Functiontest_quality_config_rejects_duplicate_keys_and_pathological_values
( tmp_path: Path, config: str, match: str )
tests/test_quality_config.py:210
Functiontest_quality_config_rejects_invalid_correctness
(tmp_path: Path, body: str, match: str)
tests/test_quality_config.py:780
Functiontest_quality_config_rejects_invalid_discoverability
(tmp_path: Path, body: str, match: str)
tests/test_quality_config.py:683
Functiontest_quality_config_rejects_invalid_efficiency
(tmp_path: Path, body: str, match: str)
tests/test_quality_config.py:544
Functiontest_quality_config_rejects_invalid_error_handling
(tmp_path: Path, body: str, match: str)
tests/test_quality_config.py:345
Functiontest_quality_config_rejects_invalid_expected_headers
(tmp_path: Path, config: str)
tests/test_quality_config.py:92
Functiontest_quality_config_rejects_non_utf8_file
(tmp_path: Path)
tests/test_quality_config.py:163
Functiontest_quality_config_rejects_unknown_keys_and_duplicate_headings
( tmp_path: Path, config: str, match: str )
tests/test_quality_config.py:141
Functiontest_quality_failure_does_not_invent_execution_failure
(execution_status)
tests/reporting/test_tier3_scoped_reports.py:121
Functiontest_quality_gate_stays_actionable_with_advisory_findings
()
tests/reporting/test_report_redesign.py:104
Functiontest_quality_grade_cannot_override_failed_incomplete_or_skipped_status
(passed, extra, expected)
tests/reporting/test_terminal_polish.py:244
Functiontest_quality_grades_and_minimum_match_constants
()
tests/test_metrics_reference_docs.py:385
Functiontest_quality_headline_fails_for_with_skill_required_check_failure_only
()
tests/test_tier3_attempt_verdict.py:483
Functiontest_quality_minimum_survives_merged_results
(below_threshold_skill: Path, merge_method: str)
tests/test_quality_policy_gate.py:165
Functiontest_quality_minimum_survives_recalculation
(below_threshold_skill: Path)
tests/test_quality_policy_gate.py:143
Functiontest_quality_score_and_threshold_in_system_out
()
tests/reporting/test_junit_reporter.py:362
Functiontest_quality_severity_waiver_still_passes_when_score_is_above_minimum
(below_threshold_skill: Path)
tests/test_quality_policy_gate.py:222
Functiontest_quality_threshold_failure_is_not_replaced_by_advisory_finding
(tmp_path)
tests/ci/test_report_views.py:917
Functiontest_quality_threshold_is_forwarded_to_evaluator
(tmp_path, score, workers)
tests/ci/test_run_tier1.py:1215
Functiontest_quality_threshold_is_wired_into_the_container
()
tests/ci/test_evaluate_workflow.py:177
Functiontest_quality_weights_match_constants
()
tests/test_metrics_reference_docs.py:369
Methodtest_query_returns_matches_above_threshold
(self)
tests/embedding/test_registry.py:585
Functiontest_quickstart_quality_check_matches_documented_result
Keep the bundled first-run result and its threshold behavior reproducible.
tests/test_cli.py:54
Functiontest_quiet_fallback_preserves_reason_in_returned_note
(generation_skill, monkeypatch)
tests/tier3/test_dataset_generation_progress.py:200
Functiontest_quiet_generation_has_no_provider_output
(generation_skill)
tests/tier3/test_dataset_generation_progress.py:61
Functiontest_quiet_negative_probe_is_not_failure
(checker, command, observation)
tests/tier3/test_error_recovery.py:645
Functiontest_quiet_probe_exception_does_not_hide_real_failures
(checker, command, observation)
tests/tier3/test_error_recovery.py:667
Functiontest_quiet_probe_exemption_does_not_depend_on_the_settled_field_name
(extractor, field)
tests/tier3/test_codex_tool_call_normalization.py:1752
Functiontest_quiet_probe_reads_only_a_whole_record
(checker, observation, clean)
tests/tier3/test_codex_tool_call_normalization.py:1634
Functiontest_quiet_probe_variants_are_not_failures
(checker, command, output)
tests/tier3/test_error_recovery.py:1229
Functiontest_quiet_probe_variants_keep_real_failures
(checker, command, output, exit_code)
tests/tier3/test_error_recovery.py:1259
Functiontest_quoted_cookie_credentials_are_absent_from_judge_evidence
(header: str, quote: str)
tests/tier3/test_security_finding_evidence.py:1832
Functiontest_quoted_cookie_credentials_are_absent_from_security_findings
(copy: str, header: str, quote: str)
tests/tier3/test_security_finding_evidence.py:1820
Functiontest_quoted_credential_values_are_masked_whole
(text: str, expected: str)
tests/tier3/test_security_finding_evidence.py:570
Functiontest_quoted_dataset_text_drops_terminal_controls_and_credentials
()
tests/tier3/test_dataset_baseline_fairness.py:2060
Methodtest_quoted_extra_keys_remain_strings
(self, sample_skill_dir, key)
tests/validators/test_schema.py:39
Functiontest_quoted_four_case_half_widths_match_the_interval
The 'a run on four cases is a smoke test' argument quotes real half-widths.
tests/test_metrics_reference_docs.py:271
Functiontest_quoted_pipe_argument_is_not_a_pipeline
()
tests/validators/test_plugin_hook_risk_bounds.py:461
Functiontest_quoted_separators_are_not_boundaries
()
tests/tier3/test_native_launch_insertion.py:77
Functiontest_quoted_shell_text_is_not_treated_as_docker_heredoc
(tmp_path: Path)
tests/test_harbor_runtime_skill_isolation.py:3824
Methodtest_quoted_unknown_key_remains_an_extra_field_error
(self, tmp_path)
tests/validators/test_plugin_schema.py:46
Functiontest_quoted_url_ampersand_is_not_treated_as_background_operator
(tmp_path: Path)
tests/test_harbor_local_mode.py:1822
Functiontest_quotes_in_description_are_escaped_in_copilot_and_openclaw_lists
(tmp_path: Path)
tests/harness/test_overlay.py:918
Functiontest_ragas_cleanup_consumes_late_task_exceptions
A task failing after the drain timeout must not leak credentials through asyncio's exception handler.
tests/tier3/test_judge_failure_artifacts.py:727
Functiontest_ragas_cleanup_does_not_wait_for_cancellation_acknowledgment
Cleanup returns before resistant work finishes, closes the client, and preserves the scorer's outcome.
tests/tier3/test_judge_failure_artifacts.py:623
Functiontest_ragas_client_close_cancellation_preserves_the_scorer_outcome
A cancelled close operation must not replace a completed score or the scorer's original exception.
tests/tier3/test_judge_failure_artifacts.py:586
Functiontest_ragas_evidence_failure_never_changes_the_score
( verifier: ModuleType, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture, broken: str )
tests/tier3/test_ragas_judge_evidence.py:672
Functiontest_ragas_evidence_helpers_reraise_the_judge_deadline
Pins the helpers only: they re-raise the deadline alarm instead of logging it as an evidence failure. In a real request the alarm leaves the http
tests/tier3/test_ragas_judge_evidence.py:725
Functiontest_ragas_evidence_masks_argv_lists_in_tool_calls
A tool call's argv list reaches the RAGAS prompt and reply as JSON in JSON, so its quotes are escaped twice. Each credential element is masked th
tests/tier3/test_ragas_judge_evidence.py:506
Functiontest_ragas_evidence_masks_consecutive_credential_lines_in_tool_calls
A tool call's arguments reach the RAGAS prompt and reply as JSON in JSON, so their line breaks are escaped twice. A flag value that ends one line
tests/tier3/test_ragas_judge_evidence.py:444
Functiontest_ragas_evidence_masks_credentials_at_json_escaped_line_starts
RAGAS writes the agent's messages into its prompt as JSON, and a JSON-mode reply quotes them as JSON too. Each line break in the agent's text is
tests/tier3/test_ragas_judge_evidence.py:409
Functiontest_ragas_evidence_masks_private_key_blocks_in_json_prompts_and_replies
A private key block in the agent's text spans "\\n" escapes in the RAGAS prompt and reply, and is masked whole.
tests/tier3/test_ragas_judge_evidence.py:608
Functiontest_ragas_evidence_masks_quoted_tool_call_values_escaped_twice
A double-quoted value in a tool call's arguments reaches the RAGAS prompt and reply with its quotes escaped twice. The whole value is masked, pas
tests/tier3/test_ragas_judge_evidence.py:561
Functiontest_ragas_evidence_reads_every_request_message_and_tool_call_replies
(verifier: ModuleType)
tests/tier3/test_ragas_judge_evidence.py:377
Functiontest_ragas_evidence_write_failure_is_logged_like_the_other_judges
_record_judge_attempt logs a failed judge_attempts.json write itself, so the RAGAS warning is not logged.
tests/tier3/test_ragas_judge_evidence.py:697
Functiontest_ragas_exchange_with_an_unreadable_request_names_the_selected_judge_model
A request body that is not a chat request names no model, so the entry names the selected judge model.
tests/tier3/test_ragas_judge_evidence.py:364
Functiontest_ragas_expired_setup_budget_does_not_create_a_scoring_coroutine
An exhausted setup deadline must not leave an unawaited scorer behind.
tests/tier3/test_judge_failure_artifacts.py:789
Functiontest_ragas_goal_judge_obeys_the_required_judge_time_budget
Ensure Ragas goal scorer returns before its Harbor budget expires and still closes its client.
tests/tier3/test_judge_failure_artifacts.py:831
Functiontest_ragas_goal_scorer_failure_warns_and_closes_its_client
A Ragas failure is logged as a redacted warning, and the client is closed before the custom judge runs.
tests/tier3/test_judge_failure_artifacts.py:528
Functiontest_ragas_goal_scorer_setup_failure_closes_its_client
A failure while building the scorer, before any score request, must not leak the client.
tests/tier3/test_judge_failure_artifacts.py:566
Functiontest_ragas_goal_scorer_uses_the_collections_keyword_contract
An eligible OpenAI judge scores goal_accuracy with Ragas instead of silently using the custom judge.
tests/tier3/test_judge_failure_artifacts.py:487
Functiontest_ragas_reply_behind_a_rejected_score_is_saved
(verifier: ModuleType, monkeypatch: pytest.MonkeyPatch)
tests/tier3/test_ragas_judge_evidence.py:295
Functiontest_ragas_reply_without_text_or_tool_calls_is_saved_as_the_reply_body
A refusal or a filtered reply is what makes RAGAS fail, so the entry keeps why there was no answer.
tests/tier3/test_ragas_judge_evidence.py:654
Functiontest_ragas_request_and_reply_are_saved_redacted
(verifier: ModuleType, monkeypatch: pytest.MonkeyPatch)
tests/tier3/test_ragas_judge_evidence.py:255
Functiontest_raising_efficiency_severity_keeps_cli_minimum_score_failure
( below_threshold_skill: Path, tmp_path: Path )
tests/test_quality_policy_gate.py:84
Functiontest_raw_docker_command_timeout_kills_and_reaps_client
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_secure_docker_environment.py:7707
Functiontest_raw_docker_creation_cancellation_resolves_process_race_and_reaps_client
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_secure_docker_environment.py:7579
Functiontest_raw_docker_creation_timeout_is_total_deadline_bounded_with_late_reaper
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_secure_docker_environment.py:7636
Functiontest_raw_docker_host_baseline_retains_windows_home_controls_without_home
( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, )
tests/test_harbor_secure_docker_environment.py:2529
Functiontest_raw_head_pins_address_checks_tls_and_ignores_ambient_auth
(er, monkeypatch)
tests/validators/test_private_endpoint_resolution.py:192
Methodtest_raw_html_encoded_trailing_space_remains_path_content
(self, tmp_path: Path)
tests/validators/test_quality_score.py:1269
← previousnext →20,601–20,700 of 24,741, ranked by callers