MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

↓ 3 callersFunction_is_native_openai_chat_url
(provider, request_url)
src/skillevaluator/tier3/harbor/templates/eval.py:1304
↓ 3 callersFunction_is_network_exfiltration_command
Inspect a shell command for network client exfiltration indicators.
src/skillevaluator/tier3/eval_core/checks.py:2459
↓ 3 callersFunction_is_non_bool_int
(value: object, *, minimum: int = 0)
src/skillevaluator/tier3/results_location.py:270
↓ 3 callersMethod_is_read_only_passwd_mount_issue
Suppress only a benign-tagged PE3 on an exact read-only passwd mount. SkillSpector tags ``docker run -v /etc/passwd:/etc/passwd:ro`` (and its
src/skillevaluator/validators/security.py:5413
↓ 3 callersFunction_is_relative_to
(path: Path, parent: Path)
src/skillevaluator/tier3/harbor/adapter.py:456
↓ 3 callersFunction_is_retriable_http_status
408, 429, and every 5xx except 501 and 505 (same rule as ``skillevaluator.inference.retry``).
src/skillevaluator/tier3/harbor/templates/eval.py:1653
↓ 3 callersFunction_is_reusable_workflow
Recognize workflow_call without YAML 1.1's unquoted-on ambiguity.
src/skillevaluator/_ci_onboarding_all_tiers.py:366
↓ 3 callersFunction_is_serialized_harbor_trial_result
Recognize Harbor's complete TrialResult envelope, not legacy fragments.
src/skillevaluator/tier3/harbor/collector.py:2837
↓ 3 callersFunction_is_tier2
(result: ValidationResult)
src/skillevaluator/reporting/benchmark.py:2146
↓ 3 callersFunction_is_tier3
(result: ValidationResult)
src/skillevaluator/reporting/benchmark.py:2153
↓ 3 callersMethod_is_workload_identity_enabled
Return True when the operator explicitly opted into GKE Workload Identity.
src/skillevaluator/tier3/harbor/gke_environment.py:799
↓ 3 callersFunction_iter_discovery_files
Yield bounded files from one pruned inter-skill traversal. A contained directory alias such as ``.agents -> .claude`` is never followed. It i
src/skillevaluator/embedding/extractor.py:458
↓ 3 callersFunction_job_result
(job: Path, *, errored: int)
tests/tier3/test_baseline_timeouts.py:146
↓ 3 callersFunction_jpeg_segment
Return one JPEG marker segment with its length.
tests/validators/test_security.py:8875
↓ 3 callersFunction_json_nesting_within_limit
Bound structural nesting without recursively parsing partial JSON.
src/skillevaluator/tier3/harbor/templates/eval.py:2561
↓ 3 callersFunction_json_safe_tier3_text
Normalize template-facing text and replace invalid UTF-8 surrogates.
src/skillevaluator/reporting/html.py:218
↓ 3 callersFunction_judge_attempt_model
Return the model that answered an attempt, or the requested model when none did.
src/skillevaluator/tier3/harbor/templates/eval.py:3138
↓ 3 callersFunction_judge_error
Return a bounded, redacted result that cannot be mistaken for a judged zero.
src/skillevaluator/tier3/eval_core/llm_judge.py:131
↓ 3 callersFunction_judge_fallback_policy
Return LLM_JUDGE_FALLBACK_POLICY: ``append`` (default) or ``replace``. The only fallback model this verifier adds on its own is the Claude Opus
src/skillevaluator/tier3/harbor/templates/eval.py:939
↓ 3 callersFunction_judge_mixed_cases
Map each judge metric to the cases whose two arms were judged by different models.
src/skillevaluator/tier3/harbor/collector.py:5583
↓ 3 callersFunction_judge_not_applicable
Return a scoreless result for a judge that has nothing to judge against.
src/skillevaluator/tier3/eval_core/llm_judge.py:144
↓ 3 callersFunction_label
(value: object)
src/skillevaluator/skillspector_provenance.py:107
↓ 3 callersFunction_launch_argv
The argv a launch config runs: ``command`` plus ``args``. ``command`` may name only the program, possibly as a path with spaces such as ``C:\
src/skillevaluator/validators/mcp_static.py:1701
↓ 3 callersFunction_leading_hash_or_slash_comment_lines
Lines whose first non-space text is `#` or `//`.
src/skillevaluator/validators/security.py:310
↓ 3 callersFunction_legacy_report
Return a minimal pre-completeness SkillSpector report the validator accepts.
tests/validators/test_skillspector_scope.py:1624
↓ 3 callersMethod_license_bucket
Classify a license value as allowed, blocked, unknown, or unsupported syntax.
src/skillevaluator/validators/license.py:415
↓ 3 callersFunction_lift_kpi_labels
(html: str)
tests/reporting/test_tier3_unavailable_lift_labels.py:69
↓ 3 callersFunction_lift_value
(metric: str, lift_data: dict[str, Any])
src/skillevaluator/evaluation/tier3_report.py:4621
↓ 3 callersFunction_link_manifest_to_another_name
Move *manifest*'s content to ``shared/name`` and leave a link under the manifest's own name.
tests/cli/test_validate_manifest_file.py:136
↓ 3 callersFunction_listed
Words as the docs list them: "a, b or c".
tests/tier3/test_dataset_baseline_fairness.py:1015
↓ 3 callersFunction_load_analyzer
()
tests/test_reference_text_analyzer.py:18
↓ 3 callersFunction_load_script
(name: str)
tests/test_reference_skill_scripts.py:56
↓ 3 callersFunction_load_suggestions_v2
Read evidence-backed suggestions from the best agent's findings.json.
src/skillevaluator/evaluation/tier3_report.py:5022
↓ 3 callersFunction_local_agent_credentials
Map the resolved provider to the env vars local-mode agent CLIs read. opencode/codex read OPENAI_API_KEY/OPENAI_BASE_URL; claude-code reads A
src/skillevaluator/tier3/harbor/runner.py:2072
↓ 3 callersFunction_lookalikes
(tmp_path: Path, first: str, second: str)
tests/harness/test_overlay.py:1018
↓ 3 callersFunction_lowercase
(variants: set[str], unbounded: set[str])
src/skillevaluator/validators/skillspector_scope.py:1118
↓ 3 callersFunction_lstat_output_child
Inspect one immediate output child relative to a pinned directory.
src/skillevaluator/tier3/output_provenance.py:87
↓ 3 callersFunction_make_plugin
(tmp_path: Path)
tests/cli/test_validate_plugin.py:21
↓ 3 callersFunction_manifest_label
Name a manifest read on its own by its directory, such as ``my-skill/SKILL.md``. The directory is made absolute and ``..`` is removed lexically,
src/skillevaluator/embedding/extractor.py:288
↓ 3 callersFunction_markdown_inline_text
Flatten untrusted metadata into one inert Markdown text fragment.
src/skillevaluator/reporting/markdown.py:97
↓ 3 callersFunction_markdown_safe_characters
Remove control and separator characters from untrusted Markdown text.
src/skillevaluator/reporting/markdown.py:61
↓ 3 callersFunction_mask_config_set_value
Mask the value of a config set command whose key names a credential (*values* reads the names of its text).
src/skillevaluator/tier3/eval_core/checks.py:4751
↓ 3 callersFunction_mask_text
(value: str, redactor: Redactor)
ci/diagnostics.py:72
↓ 3 callersFunction_match_skill_text
( text: str, skill_name: str | None, skill_scripts: Sequence[str], *, prompt: bool, pr
src/skillevaluator/tier3/evals_spec.py:854
↓ 3 callersFunction_mcnemar_exact_probability
Return the exact two-sided McNemar probability as a rational number.
src/skillevaluator/tier3/harbor/collector.py:4857
↓ 3 callersFunction_mean
(values: Sequence[float])
src/skillevaluator/tier3/harbor/stats.py:319
↓ 3 callersFunction_mean_scores
(rewards: list[dict[str, Any]])
tests/reporting/test_tier3_failure_reasons.py:89
↓ 3 callersFunction_measured_number
(value: Any)
src/skillevaluator/tier3/harbor/run_budget.py:420
↓ 3 callersFunction_merged_atif_schema_version
Label a merged trajectory with the newest source ATIF version, never below v1.7.
src/skillevaluator/tier3/harbor/collector.py:1973
↓ 3 callersFunction_messages
(result: ValidationResult)
tests/validators/test_plugin_component_risk.py:1144
↓ 3 callersFunction_methodology_row
(markdown: str, label: str)
tests/reporting/test_dimension_explanations.py:69
↓ 3 callersFunction_metric_benchmark_rows
(benchmark: str)
tests/reporting/test_activation_benchmark_html_regressions.py:66
↓ 3 callersFunction_metrics
(record: dict)
ci/tier_manifest.py:307
↓ 3 callersFunction_minimal_skill
(tmp_path: Path, name: str, description: str)
tests/test_benchmark_source_provenance.py:1452
↓ 3 callersFunction_multimodal_trajectory
()
tests/tier3/test_codex_tool_call_normalization.py:1009
↓ 3 callersFunction_namespace_chat_tool_name
( namespace_tools: dict[str, _NamespaceToolTarget], namespace: Any, name: str, kind: str,
src/skillevaluator/tier3/harbor/nvidia_build_bridge.py:1777
↓ 3 callersFunction_native_task
(skill: Path, extra: str = "")
tests/test_nemo_gym_backend.py:1979
↓ 3 callersFunction_no_ground_truth_low_efficiency_payload
Return a run with no ground_truth, Security 1.0 and Efficiency 0.1 that passes its required checks.
tests/test_public_benchmark_gate.py:1297
↓ 3 callersFunction_no_skill
The no-plugin arm: nothing to discover, so the skill metrics are N/A.
tests/tier3/test_private_ladder_stats.py:54
↓ 3 callersFunction_node_fingerprint
Return the fields that must remain stable while reading an artifact.
src/skillevaluator/tier3/results_location.py:160
↓ 3 callersFunction_norm_server
(value: str)
src/skillevaluator/tier3/eval_core/plugin_signals.py:2104
↓ 3 callersFunction_normalize_docker_context_source
Normalize a COPY/ADD source using Docker build-context traversal rules.
src/skillevaluator/tier3/harbor/adapter.py:2791
↓ 3 callersFunction_normalize_generated_cases
(cases: list[dict[str, Any]])
src/skillevaluator/tier3/generate_dataset.py:608
↓ 3 callersMethod_normalize_message
Normalize a finding message for deduplication. Strips trailing whitespace/newlines and collapses internal whitespace so that near-ide
src/skillevaluator/reporting/html.py:2572
↓ 3 callersFunction_normalized_key_parts
(key: str)
src/skillevaluator/utils/redaction.py:180
↓ 3 callersFunction_normalized_registry_vectors
Validate every vector against the registry width and normalize each once.
src/skillevaluator/embedding/registry.py:848
↓ 3 callersFunction_normalized_reward_numbers
Return JSON-safe reward data and whether an invalid number was replaced.
src/skillevaluator/tier3/harbor/collector.py:3865
↓ 3 callersFunction_normalized_trajectory_reference
(reference: Any)
src/skillevaluator/tier3/harbor/collector.py:1884
↓ 3 callersFunction_normalized_vector
(vector: object, expected_dimension: int | None)
src/skillevaluator/embedding/registry.py:828
↓ 3 callersFunction_not_applicable_metrics
(data: Mapping[str, Any], variant: str)
src/skillevaluator/tier3/result_display.py:91
↓ 3 callersMethod_notify
(t)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 3 callersFunction_npm_pin
(name: Any, version: Any)
src/skillevaluator/plugin_dependency_audit.py:195
↓ 3 callersMethod_nvidia_provider_config_command
(self, model_id: str)
src/skillevaluator/tier3/harbor/local_agents.py:575
↓ 3 callersFunction_object
(value, allowed, required, where)
src/skillevaluator/tier3/grader_validation.py:65
↓ 3 callersFunction_observed_path_matches
``observed`` and ``artifact`` are resolved (:func:`_resolve_path`); ``observed`` may be a glob. Two absolute paths must be equal. A relative path
src/skillevaluator/tier3/eval_core/plugin_signals.py:4337
↓ 3 callersFunction_only_adds_markers
Whether *new* is *base* with more of its text replaced by markers, and none taken back.
tests/tier3/test_security_finding_evidence.py:1568
↓ 3 callersFunction_only_llm_stages_missing
True when the check itself ran and only its ``--llm`` / ``--llm-verify`` stage did not. ``skillspector-llm`` is also recorded when SkillSpector i
src/skillevaluator/reporting/console_ui.py:1372
↓ 3 callersFunction_open_absolute_directory
Open one absolute path component-by-component without following links.
src/skillevaluator/tier3/harbor/secure_copy.py:318
↓ 3 callersFunction_open_fallback_regular
Open without blocking, then verify identity before any fallback read. ``O_NOFOLLOW`` is used where the platform provides it. Elsewhere a link
src/skillevaluator/tier3/harbor/secure_copy.py:524
↓ 3 callersFunction_open_relative_directory
( root_descriptor: int, parts: tuple[str, ...], *, manifest: _TreeManifest | None = None, )
src/skillevaluator/tier3/harbor/secure_copy.py:718
↓ 3 callersFunction_open_tree_root
(source: Path, allowed_root: Path, *, role: str)
src/skillevaluator/tier3/harbor/secure_copy.py:364
↓ 3 callersMethod_open_windows
( self, relative_path: Path, expected: os.stat_result | None, *, allow_hardlinks: bool = False )
src/skillevaluator/utils/secure_fs.py:483
↓ 3 callersFunction_opencode_trajectory
Convert OpenCode bash events with Harbor's converter, which keeps only the output.
tests/tier3/test_error_recovery.py:1046
↓ 3 callersFunction_ordered_step_trajectory_paths
(trial_root: Path)
src/skillevaluator/tier3/harbor/collector.py:1710
↓ 3 callersFunction_outcome_behaviors
Behaviors about the task outcome, which both arms can be graded on. Reading SKILL.md or running the skill's script is scored by skill_execution;
src/skillevaluator/tier3/generate_dataset.py:561
↓ 3 callersFunction_outcome_counts
()
src/skillevaluator/tier3/eval_core/plugin_signals.py:4145
↓ 3 callersFunction_outcome_ground_truth
(outcome: str | None)
src/skillevaluator/tier3/generate_dataset.py:557
↓ 3 callersFunction_outer_tool_call_observation
( step: dict[str, Any], tool_call: dict[str, Any], *, include_unscoped: bool = True )
src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py:78
↓ 3 callersFunction_parse_license_expression
Check SPDX expression syntax and collect the names it references. Supports ``AND``, ``OR`` and ``WITH`` (matched case-insensitively) and pare
src/skillevaluator/validators/license.py:76
↓ 3 callersFunction_parse_limit_skill
The skill of the v2.13.3 report, whose helper SkillSpector cannot parse within its span limit.
tests/validators/test_skillspector_reference_diagnostics.py:102
↓ 3 callersFunction_parse_ref
(raw: str)
src/skillevaluator/tier3/eval_core/plugin_signals.py:3435
↓ 3 callersFunction_pass_at_k_branch
(agent: Mapping[str, Any], condition: str)
src/skillevaluator/tier3/token_usage.py:201
↓ 3 callersMethod_passes_luhn
Validate a number string using the Luhn algorithm. Returns True if the digit string passes the Luhn check, meaning it could be a vali
src/skillevaluator/validators/security.py:6407
↓ 3 callersFunction_path_args
(args: Mapping[str, Any])
src/skillevaluator/tier3/eval_core/plugin_signals.py:2266
↓ 3 callersFunction_path_is_link_or_reparse
Return whether a result path is a symlink, junction, or reparse point.
src/skillevaluator/tier3/results_location.py:146
↓ 3 callersMethod_path_with_evaluator_python
Ensure local verifier scripts use the evaluator's Python runtime.
src/skillevaluator/tier3/harbor/local_environment.py:1888
↓ 3 callersFunction_pattern_matches
Return the matches of *pattern* that finditer finds in *text*, searching only a text it can match.
src/skillevaluator/tier3/eval_core/checks.py:4619
↓ 3 callersFunction_pattern_may_match
Return whether *text* holds a word of each group that _FINDING_EVIDENCE_PATTERN_WORDS lists for *pattern*.
src/skillevaluator/tier3/eval_core/checks.py:4614
↓ 3 callersFunction_payload
(findings=None)
tests/tier3/test_findings_severity.py:32
↓ 3 callersFunction_percentage_point_units
Round a [0, 1] score half-up to whole units of ``10**-decimal_places`` percentage points.
src/skillevaluator/reporting/benchmark.py:1934
← previousnext →2,801–2,900 of 24,741, ranked by callers