MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

Functiontest_a_native_tool_call_is_read_under_the_shell_it_names
A native call can name the shell that ran its command (Codex's ``exec_command`` keeps ``shell`` in its arguments). The rules that differ betwe
tests/tier3/test_script_invocation_evidence.py:2597
Functiontest_a_negated_skill_use_is_still_met_by_the_baseline
(text)
tests/tier3/test_dataset_baseline_fairness.py:1884
Functiontest_a_negation_over_another_action_still_requires_reading_skill_md
(field, text)
tests/tier3/test_dataset_baseline_fairness.py:1837
Functiontest_a_negation_over_another_action_still_requires_running_a_script_or_using_the_skill
(text)
tests/tier3/test_dataset_baseline_fairness.py:1856
Functiontest_a_negator_and_its_verb_modifiers_negate_the_removal_and_the_weakening
(known)
tests/test_tier3_case_summary.py:515
Methodtest_a_negator_and_its_verb_modifiers_negate_the_removal_and_the_weakening
(self, known: tuple)
tests/evaluation/test_judges.py:1223
Methodtest_a_nested_payload_carrier_reaches_the_top_level
A Tier 3 run records the identity on the payload, not on the result.
tests/test_benchmark_source_provenance.py:1432
Functiontest_a_netrc_password_after_a_value_less_flag_does_not_hide_a_later_credential
(flag: str, rest: str)
tests/tier3/test_security_finding_evidence.py:2412
Functiontest_a_network_client_that_another_command_starts_is_checked_for_uploads
(run, cmd)
tests/tier3/test_network_client_mentions.py:337
Functiontest_a_new_run_without_close_carries_nothing_from_a_refused_stop
(monkeypatch: pytest.MonkeyPatch)
tests/test_tier3_progress.py:1428
Methodtest_a_non_canonical_expectation_is_refused
(self, values: dict[str, object])
tests/test_benchmark_source_provenance.py:737
Methodtest_a_non_canonical_flag_is_a_usage_error
(self, tmp_path: Path, capsys: pytest.CaptureFixture[str])
tests/test_benchmark_source_provenance.py:784
Methodtest_a_non_pass_card_for_another_source_also_fails
(self, tmp_path: Path)
tests/test_benchmark_source_provenance.py:698
Functiontest_a_noun_between_the_negator_and_the_verb_breaks_the_negation
(known)
tests/test_tier3_case_summary.py:522
Methodtest_a_noun_between_the_negator_and_the_verb_breaks_the_negation
(self, known: tuple)
tests/evaluation/test_judges.py:1229
Functiontest_a_numeric_base_of_any_length_is_scored
zsh and ksh read a ``base#digits`` base as a decimal and drop its leading zeros, so ``016#ff`` and a base written after thousands of zeros are 16.
tests/tier3/test_script_invocation_evidence.py:1908
Functiontest_a_p_inside_a_flag_value_does_not_hide_the_password_after_a_masked_command_word
( flag: str, command: str, expected: str )
tests/tier3/test_security_finding_evidence.py:2282
Functiontest_a_p_inside_a_flag_value_escaped_twice_does_not_hide_the_password
(flag: str, shape: Any)
tests/tier3/test_security_finding_evidence.py:2323
Functiontest_a_part_loaded_nowhere_is_a_load_failure_that_drops_the_case_from_every_arm
(loose)
tests/tier3/test_private_loose_source.py:658
Methodtest_a_partial_carrier_and_the_options_publish_one_union
Attaching the supplied identity only where none was recorded dropped the rest of it.
tests/test_benchmark_source_provenance.py:1559
Functiontest_a_pass_at_k_case_named_by_its_cut_name_is_one_canonical_row
()
tests/tier3/test_token_usage.py:758
Functiontest_a_password_masked_behind_a_flag_value_keeps_the_escaped_backslash_after_it
(flag: str, tail: str)
tests/tier3/test_security_finding_evidence.py:2548
Methodtest_a_patch_that_deletes_or_moves_the_artifact_away_does_not_write_it
( self, headers: str, passed: int, form: str )
tests/tier3/test_plugin_signals.py:1032
Functiontest_a_path_list_names_each_referenced_path
(tmp_path: Path)
tests/tier3/test_private_loose_source.py:835
Functiontest_a_path_piped_to_xargs_is_undecidable
xargs builds its arguments from standard input, which the text lacks.
tests/tier3/test_script_invocation_evidence.py:323
Functiontest_a_pattern_search_that_raises_fails_the_check
(monkeypatch: pytest.MonkeyPatch, error: Exception)
tests/tier3/test_plugin_signals.py:1422
Functiontest_a_pattern_the_engine_cannot_compile_is_dropped_without_breaking_collection
(pattern: str)
tests/tier3/test_plugin_signals.py:1345
Functiontest_a_payload_over_the_limits_without_token_usage_keeps_no_token_record
( monkeypatch: pytest.MonkeyPatch, tmp_path: Path )
tests/reporting/test_tier3_token_usage_reports.py:734
Functiontest_a_payload_with_a_line_its_shell_refuses_stays_partial
A ``-c`` payload is refused by its own shell: a here-string under dash and busybox ash, a compound's opening word after an assignment under ea
tests/tier3/test_script_invocation_evidence.py:2542
Functiontest_a_pending_mcp_server_whose_calls_all_failed_is_not_loaded
(loose)
tests/tier3/test_private_loose_source.py:648
Functiontest_a_pending_mcp_server_with_a_successful_call_is_loaded_by_call_evidence
(loose)
tests/tier3/test_private_loose_source.py:636
Functiontest_a_piece_missing_from_the_loose_arm_is_a_mismatch
(loose)
tests/tier3/test_private_loose_source.py:369
Functiontest_a_pipeline_inside_a_subshell_still_isolates_its_own_stages
The group's scope is shared by the commands inside it, but a pipeline written inside the group still runs each stage in its own subshell: ``(f
tests/tier3/test_script_invocation_evidence.py:1435
Methodtest_a_placeholder_or_empty_line_is_reported_as_not_recorded
(self, tmp_path: Path, value: str)
tests/test_benchmark_source_provenance.py:642
Functiontest_a_plain_space_after_the_host_is_not_ambiguous
Clients percent-encode a plain space in the path; whitespace before it, or any other kind, still counts.
tests/validators/test_url_policy.py:150
Functiontest_a_plan_line_refused_by_a_buffered_terminal_stream_is_printed_once
( monkeypatch: pytest.MonkeyPatch, refused_writes: set[int], plan_line: str, notice: str, )
tests/test_tier3_progress.py:1766
Functiontest_a_planned_case_without_attempts_gets_a_row_with_no_tokens
()
tests/tier3/test_token_usage.py:305
Functiontest_a_plugin_with_nothing_loose_beyond_skills_gives_the_error
(tmp_path: Path)
tests/tier3/test_private_loose_source.py:385
Methodtest_a_policy_override_keeps_a_folder_scan_failed_beside_a_missing_reference
Each skill's gate failure is a confirmed failure that the folder result keeps. A ``--policy`` override lowers each Critical, and each skill's
tests/validators/test_skillspector_hook_surface.py:808
Functiontest_a_probe_with_one_safe_alternative_is_accepted
()
tests/tier3/test_f90_signals_conflict.py:156
Functiontest_a_quoted_config_value_after_a_separator_keeps_the_command_after_it
( copy: str, text: str, expected: str )
tests/tier3/test_security_finding_evidence.py:4109
Functiontest_a_quoted_heredoc_operator_is_not_a_heredoc
(check)
tests/tier3/test_script_invocation_evidence.py:397
Functiontest_a_quoted_or_escaped_word_is_not_shell_syntax
``'done'`` is a command named done, ``printf "("`` prints a parenthesis and ``python3 '|' run.py`` runs a script named ``|``. The tokenizer drops
tests/tier3/test_script_invocation_evidence.py:1239
Functiontest_a_quoted_run_of_metacharacters_is_one_word_and_a_mark_character_is_data
A quoted word made of metacharacters (``';|'``, ``'2>'``) would be split into operators after tokenizing, so it is marked like a quoted reserved
tests/tier3/test_script_invocation_evidence.py:1347
Functiontest_a_read_in_the_same_step_or_a_failed_read_does_not_count
check-18 e10: a parallel write and read, and a read that says "No such file".
tests/tier3/test_f90_signals_handoff.py:414
Functiontest_a_recorded_pass_without_passing_dimensions_does_not_invent_an_optional_policy
()
tests/reporting/test_verdict_evidence.py:270
Functiontest_a_recovered_timeout_attempt_is_expected_usage
()
tests/tier3/test_token_usage.py:364
Functiontest_a_reference_outside_the_plugin_root_is_never_copied
(tmp_path: Path)
tests/tier3/test_private_loose_source.py:424
Functiontest_a_reference_passes_what_is_done_to_it_on_to_the_name_it_refers_to
After ``declare -n f=g`` (``nameref`` in ksh and mksh), assigning, exporting, unexporting, reading into or unsetting ``f`` acts on ``g``, whic
tests/tier3/test_script_invocation_evidence.py:2197
Functiontest_a_reference_to_the_plugin_root_itself_copies_the_plugin_tree_like_the_plugin_arm
(tmp_path: Path)
tests/tier3/test_private_loose_source.py:762
Functiontest_a_referenced_directory_is_staged_with_the_plugin_copy_rules
(tmp_path: Path)
tests/tier3/test_private_loose_source.py:738
Functiontest_a_refused_attempt_fails_attempt_coverage_as_the_docs_say
(tmp_path: Path)
tests/test_harbor_claude_refusal_fallback.py:343
Functiontest_a_refused_plan_line_before_the_frame_turns_progress_off_with_a_notice
( monkeypatch: pytest.MonkeyPatch, refused_writes: set[int], notice: str, )
tests/test_tier3_progress.py:1550
Methodtest_a_registry_host_does_not_spend_the_path_budget
The reference grammar bounds the path once the host is split off, not the two together.
tests/test_benchmark_source_provenance.py:176
Methodtest_a_registry_host_does_not_spend_the_path_budget
The bound measures the path once the host is split off, as the reference grammar does.
tests/test_benchmark_source_provenance.py:994
Methodtest_a_registry_host_is_localhost_dotted_or_ported
Those are the three shapes the grammar reads as a host, and a host name may use any case.
tests/test_benchmark_source_provenance.py:200
Methodtest_a_registry_host_is_localhost_dotted_or_ported
Those are the three shapes the grammar reads as a host, and a host name may use any case.
tests/test_benchmark_source_provenance.py:1019
Methodtest_a_registry_port_and_tag_still_pin_by_digest
(self, tmp_path: Path)
tests/test_benchmark_source_provenance.py:1023
Methodtest_a_rendered_card_matches_the_identity_it_was_given
(self, tmp_path: Path, source: dict[str, str])
tests/test_benchmark_source_provenance.py:714
Functiontest_a_repeated_variable_word_is_resolved_once_per_assignment
(monkeypatch)
tests/tier3/test_network_client_mentions.py:648
Methodtest_a_repeated_warning_goes_back_once_unless_asked
(self, repeated: bool, copies: int)
tests/models/test_result.py:249
Methodtest_a_repository_differing_only_in_case_is_another_repository
The library merges identities case-sensitively, so the gate compares them the same way.
tests/test_benchmark_source_provenance.py:583
Functiontest_a_reused_baseline_is_labelled_and_otherwise_unchanged
()
tests/tier3/test_token_usage.py:410
Functiontest_a_reused_baseline_is_labelled_in_every_report
(tmp_path: Path)
tests/reporting/test_tier3_token_usage_reports.py:779
Functiontest_a_row_that_no_dataset_id_explains_is_marked_not_rendered_as_a_case
()
tests/tier3/test_token_usage.py:693
Methodtest_a_run_with_no_identity_records_null
The key is always present, so null means none was supplied, not an older reporter.
tests/test_benchmark_source_provenance.py:1437
Functiontest_a_run_without_a_baseline_reports_its_baseline_tokens_as_not_run
(tmp_path: Path)
tests/reporting/test_tier3_token_usage_reports.py:755
Methodtest_a_run_without_a_commit_does_not_borrow_the_container_revision
With no source revision the card says so, rather than showing the evaluator's.
tests/test_benchmark_source_provenance.py:339
Functiontest_a_scheme_name_after_a_config_separator_stays_masked_in_judge_evidence
(command: str)
tests/tier3/test_security_finding_evidence.py:4005
Functiontest_a_scored_attempt_whose_arm_failed_with_invalid_cached_tokens_is_not_measured
( tmp_path: Path, cached: object )
tests/reporting/test_tier3_token_usage_reports.py:258
Functiontest_a_scored_attempt_with_invalid_cached_tokens_is_not_measured_and_resources_keep_it
( tmp_path: Path, cached: object )
tests/reporting/test_tier3_token_usage_reports.py:283
Functiontest_a_script_fed_through_standard_input_is_still_unresolved
(check, command)
tests/tier3/test_script_invocation_evidence.py:826
Functiontest_a_script_named_after_the_interpreter_target_is_only_argv
Only the first non-option argument is the script the interpreter runs.
tests/tier3/test_script_invocation_evidence.py:211
Functiontest_a_script_path_before_a_comment_or_a_quoted_hash_is_still_data
(command: str)
tests/validators/test_plugin_static_risk_review_fixes.py:287
Functiontest_a_script_swapped_for_a_link_after_planning_is_never_copied
(tmp_path: Path)
tests/tier3/test_private_loose_source.py:456
Functiontest_a_self_contradicting_probe_is_rejected
(must_use: Any, must_not_use: Any)
tests/tier3/test_f90_signals_conflict.py:149
Functiontest_a_separator_grouped_with_punctuation_still_ends_a_command
The tokenizer returns ``));`` whole, which must not swallow what follows.
tests/tier3/test_script_invocation_evidence.py:538
Functiontest_a_separator_inside_a_here_string_operand_does_not_end_it
(check, command)
tests/tier3/test_script_invocation_evidence.py:610
Functiontest_a_session_start_hook_that_pipes_a_download_into_sh_with_a_comment_is_critical
( tmp_path: Path, command: str )
tests/validators/test_plugin_static_risk_review_fixes.py:298
Functiontest_a_shell_call_that_writes_no_file_is_not_a_write
Only real files count: ``2>/dev/null`` and a plain read are not writes.
tests/tier3/test_f90_signals_conflict.py:305
Functiontest_a_shell_option_change_leaves_the_pipeline_rule_unsettled
``emulate sh`` makes zsh fork the last stage (measured), and ``shopt -s lastpipe`` makes bash keep it. Which options are in force when the pip
tests/tier3/test_script_invocation_evidence.py:1260
Functiontest_a_shell_this_walk_does_not_model_leaves_the_command_unresolved
A shell outside the modelled ones reads the text by rules this walk does not know, so no definite score is given: a command that names the script
tests/tier3/test_script_invocation_evidence.py:2620
Functiontest_a_similar_filename_is_a_different_script
(check, command)
tests/tier3/test_script_invocation_evidence.py:82
Functiontest_a_skill_md_read_through_an_attached_input_redirection_is_a_read
``cat 0<SKILL.md`` reads the file just as ``cat 0< SKILL.md`` does, and ``cat 2>SKILL.md`` overwrites it with standard error and reads nothing.
tests/tier3/test_script_invocation_evidence.py:1090
Functiontest_a_skill_that_fails_on_its_own_keeps_its_reason_over_the_id_conflict
A skill with its own failure and a shared ID is counted and reported once. Without the `name not in failed` dedup guard in `_record_metadata_id_f
tests/golden/test_policy_behavior.py:173
Functiontest_a_skipped_baseline_is_not_run_and_has_no_change
()
tests/tier3/test_token_usage.py:424
Functiontest_a_slow_dns_lookup_counts_against_the_probe_deadline
()
tests/tier3/test_mcp_proof.py:608
Functiontest_a_spaced_call_value_escaped_twice_keeps_its_arguments_masked
(copy: str, command: str)
tests/tier3/test_security_finding_evidence.py:2359
Methodtest_a_standalone_run_persists_the_identity_in_run_config
A signature alone does not prove the value is written to disk. The standalone ``tier3 evaluate`` producer owns its own run directory, so
tests/test_benchmark_source_provenance.py:1257
Functiontest_a_stdin_driven_tool_is_never_a_full_credit_invocation
Whether xargs runs anything depends on input the command text lacks. ``printf '' | xargs -r python run.py`` runs nothing at all, and ``xargs -p``
tests/tier3/test_script_invocation_evidence.py:563
Functiontest_a_stop_notice_kept_by_a_buffered_terminal_stream_is_not_repeated_when_the_run_closes
( monkeypatch: pytest.MonkeyPatch, full_at_close: bool, )
tests/test_tier3_progress.py:1814
Functiontest_a_stop_notice_refused_after_a_refused_first_frame_is_not_taken_as_kept
( monkeypatch: pytest.MonkeyPatch, )
tests/test_tier3_progress.py:1876
Functiontest_a_stop_on_pass_trial_name_is_matched_to_its_case
()
tests/tier3/test_token_usage.py:223
Functiontest_a_stop_on_pass_unscored_attempt_takes_its_case_from_its_harbor_files
(tmp_path: Path)
tests/reporting/test_tier3_token_usage_reports.py:381
Functiontest_a_subagent_keeps_its_own_window_and_works_for_the_skill_that_started_it
Harbor 0.20+ shape: sidechain steps are the subagent's own calls (check-18 e04).
tests/tier3/test_f90_signals_handoff.py:197
Functiontest_a_subagent_producer_owns_the_writes_of_its_sidechain
()
tests/tier3/test_f90_signals_handoff.py:241
Functiontest_a_substitution_naming_a_client_resolves_to_that_command
(command, resolved)
tests/tier3/test_network_client_mentions.py:682
Functiontest_a_sum_above_the_exact_json_range_is_not_reported
()
tests/tier3/test_token_usage.py:579
Methodtest_a_summary_only_carrier_on_a_second_result_still_conflicts
The deepest carrier of the second payload counts as much as the first payload's.
tests/test_benchmark_source_provenance.py:1396
Functiontest_a_symlink_in_the_trace_narrows_the_exemption_to_the_top_level
()
tests/tier3/test_issue192_security_false_positives.py:212
← previousnext →12,201–12,300 of 24,741, ranked by callers