MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

↓ 2 callersFunction_positionals
Yield ``(index, token)`` for positional arguments, skipping flags and their values.
src/skillevaluator/validators/mcp_static.py:1993
↓ 2 callersFunction_positive
(value)
ci/comment_report.py:58
↓ 2 callersFunction_positive_attempt_ordinal
Return a positive integer attempt ordinal without accepting booleans.
src/skillevaluator/tier3/harbor/collector.py:4257
↓ 2 callersFunction_powershell_switch
``"command"`` or ``"encoded"`` for a PowerShell switch that runs an inline program, else ``None``. PowerShell accepts any prefix of a switch name
src/skillevaluator/validators/mcp_static.py:529
↓ 2 callersFunction_prefixed_scope
This shell's bindings with a command's ``NAME=value`` prefix over them. The values are assigned in order, so a later one sees an earlier one,
src/skillevaluator/tier3/eval_core/script_invocation.py:1362
↓ 2 callersFunction_preflight_environment
Check installed runtimes and backend configuration without an agent run.
src/skillevaluator/tier3/workflow.py:236
↓ 2 callersFunction_prepare_fallback_moved_backup
( backup: Path, expected: os.stat_result, *, kind: str, )
src/skillevaluator/tier3/harbor/secure_copy.py:1961
↓ 2 callersFunction_prepared_skill_name
Prepare an identity once per case, without retaining contributor aliases globally.
src/skillevaluator/tier3/evals_spec.py:817
↓ 2 callersFunction_present
(value: object)
src/skillevaluator/reporting/native_evidence.py:70
↓ 2 callersFunction_prevalidate_baseline_skill_candidates
Validate configured baseline sources once before a multi-agent run.
src/skillevaluator/tier3/harbor/adapter.py:1026
↓ 2 callersFunction_preview
Short redacted rendering of a dataset-authored value (never of agent arguments). Token shapes the general redactor does not know (GitHub ``ghp_``
src/skillevaluator/tier3/eval_core/plugin_signals.py:3854
↓ 2 callersFunction_print_catalog_summary
The catalog scoreboard: per-skill verdict counts and where reports live.
src/skillevaluator/cli.py:1323
↓ 2 callersFunction_print_gitlab_admission_prerequisite
(wizard, requirement)
src/skillevaluator/_ci_onboarding_all_tiers.py:241
↓ 2 callersFunction_print_metadata_id_conflicts
Show advisory (non-blocking) catalog ID conflicts under the scoreboard.
src/skillevaluator/cli.py:1756
↓ 2 callersFunction_print_report
(report: dict[str, Any])
src/skillevaluator/tier3/interaction_commands.py:112
↓ 2 callersMethod_print_success_details
Print success details.
src/skillevaluator/reporting/cli.py:746
↓ 2 callersMethod_print_summary_table
Print summary table for all validators.
src/skillevaluator/reporting/cli.py:845
↓ 2 callersFunction_private_decoy_workspace_snapshot
Keep the same verified source bytes available to every paired task.
src/skillevaluator/tier3/harbor/runner.py:3492
↓ 2 callersFunction_private_sandbox_name
Build the retired private label without embedding it in public source.
tests/golden/test_benchmark_card.py:63
↓ 2 callersFunction_private_task_staging_parent
Choose a temp parent outside every runtime skill source. Preserve the normal system temporary location unless TMPDIR places task staging insi
src/skillevaluator/tier3/harbor/adapter.py:6019
↓ 2 callersFunction_private_url
(value: str)
src/skillevaluator/_ci_onboarding_pages.py:50
↓ 2 callersFunction_probability_text
Return a bounded nonzero decimal representation for an exact probability.
src/skillevaluator/tier3/harbor/collector.py:4885
↓ 2 callersMethod_process_findings
Report an already validated, nonempty Gitleaks findings list.
src/skillevaluator/validators/secrets.py:230
↓ 2 callersMethod_process_pip_audit
Parse a completed pip-audit process and report trustworthy evidence.
src/skillevaluator/validators/dependencies.py:579
↓ 2 callersMethod_profile
(self, tmp_path: Path, *, allow_net: bool = False)
tests/test_harbor_local_sandbox.py:650
↓ 2 callersFunction_program_from_stdin
Whether an interpreter with these arguments runs its standard input as a program. It does when it reads its program from standard input (no progr
src/skillevaluator/plugin_component_risk.py:2363
↓ 2 callersFunction_project_root
The root a project skill is installed in: a harness root, or an OpenClaw workspace ``skills/`` folder.
src/skillevaluator/harness/overlay.py:690
↓ 2 callersFunction_prompt_example
(prompt: str)
tests/tier3/test_judge_contracts.py:82
↓ 2 callersMethod_protected_home_usernames
Identities whose ``/home/<user>/`` paths should be flagged as PII. The union of the submitter identity (see :meth:`_submitter_usernames`)
src/skillevaluator/validators/security.py:3159
↓ 2 callersFunction_protected_write_entry
Return the protected-path entry a written *target* is or is inside, on path-segment boundaries. Entries starting with "/" or "~" are anchored at
src/skillevaluator/tier3/eval_core/checks.py:1424
↓ 2 callersFunction_provenance
(current_identity: dict[str, Any], source: dict[str, Any], source_dir: Path)
src/skillevaluator/tier3/version_comparison.py:81
↓ 2 callersFunction_provider
(*, model: str = "nvidia/nemotron-3-nano-30b-a3b")
tests/test_models_command.py:37
↓ 2 callersFunction_provider_error_fields
Return the code, type and message of a JSON error body, or None when the body has none.
src/skillevaluator/tier3/harbor/templates/eval.py:1050
↓ 2 callersFunction_provisional_agent_eval
()
tests/reporting/test_evidence_volume_rendering.py:137
↓ 2 callersFunction_public_failure_list
Return a bounded, redacted sample of trial-level failures.
src/skillevaluator/tier3/harbor/collector.py:464
↓ 2 callersFunction_publication_ready_oversized_condition_payload
(*, extra_condition: bool = False)
tests/reporting/test_unified_tier3_report.py:2372
↓ 2 callersFunction_publication_safe_environment
(value: object)
src/skillevaluator/reporting/benchmark.py:2408
↓ 2 callersFunction_publish_transactional
( parent_descriptor: int, destination_name: str, staged: _StagedNode, *, expected_existing
src/skillevaluator/tier3/harbor/secure_copy.py:1433
↓ 2 callersMethod_python_child
Run a real Python child that writes raw bytes, with ``codec`` as the selected text codec.
tests/utils/test_tool_runner.py:310
↓ 2 callersFunction_python_comment_line_numbers
Line numbers whose first non-space token is a Python comment.
src/skillevaluator/validators/security.py:319
↓ 2 callersFunction_qualified
(target: str, name: str)
src/skillevaluator/reporting/junit_reporter.py:150
↓ 2 callersMethod_qualified
(self, noun: int)
src/skillevaluator/evaluation/dataset_case_removal.py:407
↓ 2 callersFunction_quality
(agent: Mapping[str, Any], key: str)
src/skillevaluator/tier3/case_summary.py:598
↓ 2 callersFunction_quality_check_messages
( command: list[str], tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *extra_args: str, )
tests/test_commands.py:913
↓ 2 callersFunction_quality_score_functions
()
tests/test_metrics_reference_docs.py:406
↓ 2 callersFunction_quiet_mcp_logs
Keep ``mcp`` log records (tracebacks included) off the console during a probe; errors fail it.
src/skillevaluator/tier3/mcp_proof.py:593
↓ 2 callersFunction_quoted
A bounded, printable and credential-redacted diagnostic excerpt.
src/skillevaluator/tier3/evals_spec.py:958
↓ 2 callersFunction_quoted_skill
(tmp_path: Path)
tests/harness/test_plugin_compat.py:216
↓ 2 callersFunction_ragas_goal_accuracy_enabled
RAGAS is an OpenAI-only optimization, never an agent-key fallback. Reasoning judges reject the sampling settings RAGAS sends, so they use the pro
src/skillevaluator/tier3/harbor/templates/eval.py:3454
↓ 2 callersFunction_ragas_tool_call
(tool: str, arguments: dict[str, str])
tests/tier3/test_security_finding_evidence.py:1350
↓ 2 callersMethod_raise_path_count_limit
(self, relative: Path, actual: int)
src/skillevaluator/utils/secure_fs.py:1106
↓ 2 callersFunction_raw_cell
(table: str, condition: str, column: str)
tests/reporting/test_tier3_baseline_trial_views.py:319
↓ 2 callersMethod_raw_docker_action
( self, action: list[str], identities: Iterable[str], )
src/skillevaluator/tier3/harbor/secure_docker_environment.py:1712
↓ 2 callersMethod_raw_filtered_service_container_ids
Resolve exact IDs from Docker's authoritative Compose-label index.
src/skillevaluator/tier3/harbor/secure_docker_environment.py:1576
↓ 2 callersFunction_raw_lifecycle_deadline_scope
Share one monotonic deadline across containment, reap, and restore.
src/skillevaluator/tier3/harbor/secure_docker_environment.py:705
↓ 2 callersFunction_read
(path: str)
tests/tier3/test_security_home_paths.py:54
↓ 2 callersFunction_read_bounded_bytes
( path: Path, diagnostics: list[dict[str, Any]], *, artifact: str, )
src/skillevaluator/tier3/harbor/report_data.py:381
↓ 2 callersFunction_read_bounded_catalog_descriptor
(descriptor: int, catalog_path: Path)
src/skillevaluator/embedding/registry.py:1232
↓ 2 callersFunction_read_bounded_rubric_file
Read a contained regular file without following its final symlink.
src/skillevaluator/validators/rubric_eval.py:75
↓ 2 callersFunction_read_descriptor
(path: Path, max_bytes: int, **options: object)
tests/utils/test_secure_fs.py:549
↓ 2 callersFunction_read_failed_judge_sidecar
Read one sidecar, distinguishing safe absence from an unreadable candidate.
src/skillevaluator/tier3/harbor/collector.py:962
↓ 2 callersFunction_read_js_escape
(source: str, start: int)
src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py:370
↓ 2 callersFunction_read_json
(path: Path, root: Path)
src/skillevaluator/tier3/interactions.py:65
↓ 2 callersFunction_read_json_text
(text: str)
src/skillevaluator/tier3/harbor/collector.py:713
↓ 2 callersFunction_read_marker
(path: Path, expected_size: int)
src/skillevaluator/tier3/output_provenance.py:543
↓ 2 callersFunction_read_native_json
(path: Path, run_root: Path)
src/skillevaluator/reporting/dashboard_data.py:583
↓ 2 callersFunction_read_result
(run_dir: Path)
src/skillevaluator/tier3/version_comparison.py:34
↓ 2 callersFunction_read_stable_json
Read one JSON artifact through the pinned run root.
src/skillevaluator/tier3/results_location.py:210
↓ 2 callersFunction_read_trial_evidence
Read one agent-written trial file safely; None when it does not exist. Raises ``SecurePathError`` or ``OSError`` when the file exists but cannot
src/skillevaluator/tier3/harbor/collector.py:7303
↓ 2 callersMethod_reading_end
Return where the value read as *reading* (see _value_reading) ends, or None when there is none.
src/skillevaluator/tier3/eval_core/checks.py:4881
↓ 2 callersFunction_reads_program_from_stdin
An interpreter given no script, no inline code and no terminal option reads its program from standard input; ``-`` names standard input.
src/skillevaluator/tier3/eval_core/script_invocation.py:2519
↓ 2 callersFunction_record
(tool: str, arg: str, rule_name: str, detail: str)
src/skillevaluator/tier3/eval_core/plugin_signals.py:4061
↓ 2 callersMethod_record_error
(self, exc: Exception, *, artifact: str = "")
src/skillevaluator/tier3/plugin_comparisons.py:330
↓ 2 callersFunction_record_metadata_id_failures
Add blocking ID conflicts to *failures*, keeping a skill's own failure reason first.
src/skillevaluator/cli.py:1771
↓ 2 callersFunction_record_regex_timeouts
Record the deadline of every dataset-pattern search (a search without one fails).
tests/tier3/test_plugin_signals.py:83
↓ 2 callersFunction_record_skipped_trajectory
Persist a value-free explanation when canonical trajectory output is omitted.
src/skillevaluator/tier3/harbor/collector.py:1416
↓ 2 callersMethod_record_unrecognized_license_file
Fail closed when a LICENSE/COPYING file exists but cannot be identified.
src/skillevaluator/validators/license.py:455
↓ 2 callersFunction_recorded_case_id
Return the case ID the collector recorded in an unscored attempt's ``failure.json``. Older runs have no such record. The file is optional, so an
src/skillevaluator/tier3/harbor/report_data.py:778
↓ 2 callersFunction_recorded_evidence_refs
Resolve LLM-cited references, keeping only those the run recorded, once each.
src/skillevaluator/tier3/harbor/report.py:807
↓ 2 callersFunction_recorded_path_matches
Match absolute paths canonically and relative paths by safe suffix.
src/skillevaluator/tier3/results_location.py:245
↓ 2 callersFunction_redact_evidence_text
(text: str)
src/skillevaluator/tier3/eval_core/atif_helpers.py:599
↓ 2 callersFunction_redact_exec_stderr
Scrub kubelet exec query strings, encoded secrets, and credential patterns from stderr.
src/skillevaluator/tier3/harbor/gke_environment.py:429
↓ 2 callersFunction_redact_harbor_diagnostic
Normalize a complete diagnostic, then remove synthesized exact secrets.
src/skillevaluator/tier3/harbor/runner.py:943
↓ 2 callersFunction_redact_network_evidence
Sanitize secrets from network action evidence text.
src/skillevaluator/tier3/eval_core/checks.py:2040
↓ 2 callersFunction_redact_spans
Replace each span of *text* in *spans* with "<redacted>", and spans that overlap or touch with one.
src/skillevaluator/tier3/eval_core/checks.py:4624
↓ 2 callersFunction_ref_label
Same label Tier 3 uses for unresolved refs (canonical ID or trailing name).
src/skillevaluator/plugin_discovery.py:201
↓ 2 callersFunction_ref_label_from
:func:`ref_label` of a ref whose :func:`normalize_ref` result is ``canonical``.
src/skillevaluator/plugin_dependencies.py:326
↓ 2 callersFunction_ref_unavailable
Whether ``ref`` names only component types this arm cannot carry. A typed ref names its own type. An untyped ref names the type of each decla
src/skillevaluator/tier3/eval_core/plugin_signals.py:3617
↓ 2 callersFunction_reference_target
``(target, disposition, structured)`` of an AE1 issue, else ``None``. *structured* says whether the target came from ``evidence`` (v2.12.0 and
src/skillevaluator/skillspector_references.py:213
↓ 2 callersFunction_reference_union_redaction
(text: str, secrets: set[str])
tests/test_harbor_secure_docker_environment.py:1024
↓ 2 callersFunction_references_exact_target_artifact
Return whether one lexical path references an exact target artifact.
src/skillevaluator/tier3/eval_core/checks.py:1676
↓ 2 callersFunction_refine_from_trajectory_template
Refine all eval fields from trajectory + developer guidance without LLM. Priority order for expected_behavior: 1. EVAL.md ## Behaviors (devel
src/skillevaluator/tier3/generate_dataset.py:1523
↓ 2 callersMethod_refresh_live
(self)
src/skillevaluator/tier3/harbor/progress.py:895
↓ 2 callersFunction_refusal_doc_texts
Return each place that documents the switch, whitespace-normalized.
tests/test_harbor_claude_refusal_fallback.py:243
↓ 2 callersFunction_refusal_fallback_target
Return the refusal fallback target for ``model``, or None for no fallback. SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODEL replaces the built-in target; ``
src/skillevaluator/tier3/harbor/templates/eval.py:977
↓ 2 callersFunction_regex_search
Search a bounded subject under a deadline; returns whether it matched, or why it was not checked. Patterns are dataset-authored and subjects agen
src/skillevaluator/tier3/eval_core/plugin_signals.py:3881
↓ 2 callersFunction_reject_baseline_pre_agent_setup
Keep skill-owned setup code out of the observational baseline arm.
src/skillevaluator/tier3/harbor/adapter.py:2328
↓ 2 callersFunction_reject_catalog_report_collisions
( catalog_path: Path | None, *, report_formats: tuple[str, ...], output_dir: Path, basenam
src/skillevaluator/cli.py:415
↓ 2 callersFunction_reject_legacy_datasets
Refuse legacy discovery before it can escape the evaluator snapshot.
src/skillevaluator/tier3/workflow.py:42
↓ 2 callersFunction_reject_unknown_keys
(mapping: dict, allowed: frozenset[str], source: str, location: str)
src/skillevaluator/quality_config.py:145
← previousnext →4,301–4,400 of 24,741, ranked by callers