MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

↓ 1 callersFunction_read_result_entry_id
Resolve case id from Harbor ``result.json`` when reward metadata is absent.
src/skillevaluator/tier3/generate_dataset.py:1009
↓ 1 callersFunction_read_reward_entry_id
(trial_dir: Path)
src/skillevaluator/tier3/generate_dataset.py:994
↓ 1 callersFunction_read_run_config
(run_dir: Path)
src/skillevaluator/evaluation/tier3_report.py:4915
↓ 1 callersFunction_read_secure_manifest
( secure_file: SecureFile, declared_path: Path, *, encoding: str, max_bytes: int, )
src/skillevaluator/plugin_manifest.py:134
↓ 1 callersFunction_read_source
(source: Path)
src/skillevaluator/tier3/dataset_migration.py:57
↓ 1 callersFunction_read_static_js_template
Read a template with no substitutions as a string-like property key.
src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py:403
↓ 1 callersFunction_read_text
Size and lines of a text file a harness could read; None for binary data and files that can't be read. The check is advisory, so anything odd is
src/skillevaluator/harness/overlay.py:285
↓ 1 callersFunction_reader_scope
The bindings whatever reads a command's text next may expand. ``eval`` reads it in this shell, and inline code hands it on with the command's
src/skillevaluator/tier3/eval_core/script_invocation.py:1455
↓ 1 callersFunction_reason
(record: dict)
ci/comment_summary.py:57
↓ 1 callersFunction_reason_candidates
Yield candidate reason strings across the known grader detail shapes.
src/skillevaluator/evaluation/tier3_findings.py:166
↓ 1 callersFunction_reason_details
One text per distinct reason, with the analyzers of rows that differ only in them joined. The run time and the time budget (``observed_seconds``,
src/skillevaluator/skillspector_references.py:346
↓ 1 callersFunction_reason_from_value
(value: Any)
src/skillevaluator/tier3/harbor/collector.py:721
↓ 1 callersFunction_reasons
The first reasons of *rows* with only known, valid fields, without repeats.
src/skillevaluator/skillspector_references.py:174
↓ 1 callersFunction_rebuilt_cases
(agent: Mapping[str, Any], key: str)
src/skillevaluator/tier3/case_summary.py:198
↓ 1 callersFunction_recommendation_category_from
Heuristic category derived from the imperative verb of a suggestion.
src/skillevaluator/evaluation/tier3_report.py:4276
↓ 1 callersFunction_recommendation_title_from
Short title for a deterministic recommendation (first sentence, truncated).
src/skillevaluator/evaluation/tier3_report.py:4414
↓ 1 callersFunction_record_builtin_reassignments
Update ``assignments`` for the variables ``read``, ``mapfile`` or ``printf -v`` set. ``read X <<< curl`` and ``printf -v X curl`` set ``X`` to ``
src/skillevaluator/tier3/eval_core/checks.py:2730
↓ 1 callersMethod_record_coverage
(registry: EmbeddingRegistry, result: ValidationResult)
src/skillevaluator/validators/similarity.py:237
↓ 1 callersMethod_record_judge_failure
( result: ValidationResult, *, report: Any, skill_path: Path, manifest
src/skillevaluator/validators/rubric_eval.py:721
↓ 1 callersFunction_record_line
(name: str, data: bytes)
tests/test_custom_backend_package.py:100
↓ 1 callersMethod_record_matches
Convert SimilarityMatch objects into structured Findings.
src/skillevaluator/validators/similarity.py:266
↓ 1 callersFunction_record_output
(observation: str)
src/skillevaluator/tier3/eval_core/checks.py:7500
↓ 1 callersFunction_record_ragas_exchange
Save one RAGAS judge request and its reply, or the HTTP error, as a goal_accuracy attempt. _record_judge_attempt redacts the full texts before it
src/skillevaluator/tier3/harbor/templates/eval.py:3508
↓ 1 callersFunction_record_token_usage_omission
Count a token usage drop in ``report_truncation``; ``False`` when its record cannot take it.
src/skillevaluator/evaluation/tier3_report.py:1828
↓ 1 callersFunction_record_validate_json_report
(report_name: str | None)
src/skillevaluator/cli.py:716
↓ 1 callersFunction_recorded_image
(summary: dict, fallback: str)
ci/publish_reports.py:898
↓ 1 callersFunction_recorded_score_policies
(agents: dict[str, dict[str, Any]])
src/skillevaluator/evaluation/tier3_report.py:2005
↓ 1 callersFunction_records
(value: Any)
src/skillevaluator/reporting/insights_view.py:17
↓ 1 callersFunction_records
(value: object)
src/skillevaluator/reporting/diagnostic_view.py:16
↓ 1 callersFunction_recover_interrupted_key_publish
Remove only same-inode private temp links left by an interrupted publish.
src/skillevaluator/tier3/output_provenance.py:406
↓ 1 callersFunction_recovered_job_failure
Only dismiss aggregate error counts fully explained by retained timeouts.
src/skillevaluator/tier3/harbor/collector.py:6893
↓ 1 callersFunction_recovered_timeout_attempts
Bind retained failures to later scoreable trials of the same task and agent.
src/skillevaluator/tier3/harbor/collector.py:6851
↓ 1 callersFunction_recovered_timeout_lines
One line per agent whose agent timeouts a later scored attempt recovered.
src/skillevaluator/tier3/result_display.py:365
↓ 1 callersFunction_redact_absolute_paths
Reduce absolute POSIX and Windows paths embedded in free text to basenames.
src/skillevaluator/reporting/benchmark.py:2342
↓ 1 callersFunction_redact_command_line_credentials
Mask password, token and cookie arguments of quoted shell command lines.
src/skillevaluator/evaluation/tier3_findings.py:110
↓ 1 callersFunction_redact_detail
(value: str, environment: Mapping[str, str])
src/skillevaluator/tier3/harbor/runtime_preflight.py:882
↓ 1 callersFunction_redact_judge_evidence
Mask credentials in a judge prompt, reply, or problem before it is saved.
src/skillevaluator/tier3/harbor/templates/eval.py:3030
↓ 1 callersFunction_redact_judge_evidence_as_written
Mask credentials in a judge prompt, reply, or problem as written.
src/skillevaluator/tier3/harbor/templates/eval.py:3044
↓ 1 callersFunction_redact_result
( result: ExecResult, secret_values: set[str], *, replacement: str | None = None, include_
src/skillevaluator/tier3/harbor/secure_docker_environment.py:646
↓ 1 callersFunction_redact_truncated_secret_suffix
Scrub trailing secret prefixes when CPython int() error formatting truncates stderr at 200 chars.
src/skillevaluator/tier3/harbor/gke_environment.py:410
↓ 1 callersFunction_redact_unterminated_private_key
Redact from the first private-key header with no later ``-----END `` to the end of the text.
src/skillevaluator/utils/redaction.py:247
↓ 1 callersFunction_redacted_match_excerpt
(match: re.Match[str])
scripts/check_oss_boundary.py:238
↓ 1 callersFunction_redacted_pin
The classification with credentials in its detail removed, and the detail bounded. Each of ``credentials`` (a declaration's literal credentials)
src/skillevaluator/validators/mcp_static.py:2701
↓ 1 callersMethod_redacted_prompt_fields
Message, line content, and line number placeholders for unsafe files.
src/skillevaluator/inference/finding_verifier.py:100
↓ 1 callersFunction_redacted_reference_url
:func:`redacted_url` for a URL that holds ``${NAME}`` or ``${NAME:-default}`` references. What is hidden is decided on the URL the client builds,
src/skillevaluator/validators/mcp_static.py:326
↓ 1 callersFunction_redacted_row
Redact the failure text a rollout row carries before it reaches trial and result files.
src/skillevaluator/tier3/nemo_gym/runner.py:741
↓ 1 callersFunction_redaction_markers
(value: Any)
src/skillevaluator/tier3/trace_import.py:66
↓ 1 callersFunction_redirects_script_to_stdin
Whether the script is fed to a command's standard input. ``python <run.py`` runs the script even though it never appears as an argument, and
src/skillevaluator/tier3/eval_core/script_invocation.py:1498
↓ 1 callersFunction_reference_anchor_words
Return words, one of which appears in any text that references *file_path*. Every reference variant contains the file's stem or a whole directory
src/skillevaluator/validators/skillspector_scope.py:748
↓ 1 callersFunction_reference_canary_spec
Return the validated canary spec from a task entry, or ``None`` when no canary was planted.
src/skillevaluator/tier3/eval_core/plugin_security.py:620
↓ 1 callersFunction_reference_check_canary
Scan agent-issued tool calls for canary exfiltration. Returns ``{"planted", "leaked", "sinks", "sink_kinds", "file_present"}``, plus ``read_b
src/skillevaluator/tier3/eval_core/plugin_security.py:1954
↓ 1 callersFunction_reference_contexts
(tokens: list[tuple[str, str]], references: list[tuple[int, int]])
src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py:707
↓ 1 callersFunction_reference_file_role
(file_path: str)
src/skillevaluator/validators/skillspector_scope.py:1073
↓ 1 callersFunction_reference_index
Index *text* for :func:`_reference_variant_may_match`. Returns the sorted path tokens and the sorted pairs of adjacent ``/``-separated segmen
src/skillevaluator/validators/skillspector_scope.py:236
↓ 1 callersFunction_reference_kwarg_error
Validate Harbor fields whose values name provider-managed secrets.
src/skillevaluator/tier3/evals_config.py:626
↓ 1 callersFunction_reference_variant_may_match
Return False only when *variant* cannot match the indexed reference text. A bare variant made of token characters can match only at the start of
src/skillevaluator/validators/skillspector_scope.py:252
↓ 1 callersFunction_reference_within
(path, directory)
tests/ci/test_select_changed_skills.py:246
↓ 1 callersMethod_references_readme
Return whether SKILL.md contains a parsed Markdown link to README.md.
src/skillevaluator/validators/quality_score.py:547
↓ 1 callersFunction_refine_with_llm
Refine eval cases using agent trajectory evidence via LLM.
src/skillevaluator/tier3/generate_dataset.py:1296
↓ 1 callersMethod_refresh
()
src/skillevaluator/reporting/assets/chart.umd.min.js:7
↓ 1 callersFunction_refresh_adc_credentials
Mint a fresh Vertex ADC token for one Gym launch and cap the launch to its lifetime. Mirrors the Harbor runner: the token is refreshed before eve
src/skillevaluator/tier3/nemo_gym/runner.py:781
↓ 1 callersFunction_refuse_gitlab_root_replacement
(mode)
src/skillevaluator/_ci_onboarding_all_tiers.py:571
↓ 1 callersFunction_regex_compile_problem
Bound counted expansion before entering the engine's untimed compiler. Escapes and simple character classes are single tokens. Braces combined
src/skillevaluator/tier3/eval_core/plugin_signals.py:775
↓ 1 callersFunction_regular_root
(path: Path)
ci/diagnostics.py:143
↓ 1 callersFunction_regular_targets
(entries, targets)
ci/benchmark_publish.py:75
↓ 1 callersMethod_reject_candidate
(self, pending: str, offset: int, emitted: list[str])
src/skillevaluator/tier3/harbor/stream_redaction.py:417
↓ 1 callersFunction_reject_copy_repo_root_output
Reject or reserve report storage inside any Tier 3 repo context.
src/skillevaluator/cli.py:648
↓ 1 callersFunction_reject_native_baseline_setup
(task_toml: Path, config: Any, paths: Any, steps: list[Any])
src/skillevaluator/tier3/harbor/adapter.py:5409
↓ 1 callersFunction_reject_native_standard_grading_overrides
( task_toml: Path, grading_mode: str, config: Any, paths: Any, steps: list[Any], verif
src/skillevaluator/tier3/harbor/adapter.py:5459
↓ 1 callersFunction_reject_native_windows_execution
( task_toml: Path, config: Any, verifier_passes: list[_NativeVerifierPass], )
src/skillevaluator/tier3/harbor/adapter.py:5388
↓ 1 callersFunction_related_paths
Return distinct path-like string values carried in finding metadata.
src/skillevaluator/reporting/cli.py:77
↓ 1 callersFunction_related_paths
Return distinct path-like string values carried in finding metadata.
src/skillevaluator/reporting/markdown.py:162
↓ 1 callersFunction_relative_dotfile_entry
Return the relative dotfile entry a relative *target* itself starts with, whatever its directory. A relative header such as ".aws/credentials" or
src/skillevaluator/tier3/eval_core/checks.py:1471
↓ 1 callersFunction_relative_file
Report a file the CLI names relative to the plugin root, never as an absolute path.
src/skillevaluator/validators/claude_plugin_validate.py:208
↓ 1 callersFunction_relative_label
(skill_dir: Path, root: Path)
src/skillevaluator/validators/schema.py:134
↓ 1 callersFunction_relative_path
(path: Path)
src/skillevaluator/utils/secure_fs.py:205
↓ 1 callersFunction_relative_posix
Return *path* relative to *root* as POSIX text, or None when outside it.
src/skillevaluator/reporting/junit_reporter.py:113
↓ 1 callersFunction_relative_reference
Return a relative URI reference whose first segment cannot read as a scheme.
src/skillevaluator/reporting/sarif_reporter.py:117
↓ 1 callersFunction_relative_run_paths
Relative paths a hook command runs, other than after ``cd <plugin root>``. Claude Code and Codex run a hook command in the session's working dire
src/skillevaluator/plugin_component_risk.py:1825
↓ 1 callersFunction_relative_to_directory
Find containment through existing ancestors, including case aliases on macOS.
src/skillevaluator/tier3/decoy_commands.py:106
↓ 1 callersFunction_relative_variants
(candidate: str, source_directory: Path)
src/skillevaluator/validators/skillspector_scope.py:1182
↓ 1 callersFunction_relative_word
Whether ``word`` names a file by a path relative to the working directory. ``./x`` always does. A program with a ``/`` (``scripts/check.sh``) doe
src/skillevaluator/plugin_component_risk.py:1807
↓ 1 callersFunction_relativize_text
(text: str, roots: tuple[str, ...])
src/skillevaluator/validators/claude_plugin_validate.py:222
↓ 1 callersMethod_removal
(self, start: int, end: int, kind: str)
src/skillevaluator/evaluation/dataset_case_removal.py:494
↓ 1 callersMethod_removeElements
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 1 callersMethod_removeUnreferencedMetasets
()
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 1 callersFunction_remove_staged_path
(path: Path)
src/skillevaluator/tier3/harbor/adapter.py:4167
↓ 1 callersFunction_remove_stale_findings_artifact
(results_dir: Path, agent: str)
src/skillevaluator/tier3/harbor/report.py:102
↓ 1 callersFunction_remove_token_usage_omissions
Take token usage's keys out of ``report_truncation``, and the record when only they made it.
src/skillevaluator/evaluation/tier3_report.py:1842
↓ 1 callersFunction_rename_after_public_mutation
Rename during commit/rollback without ever enabling fallback.
src/skillevaluator/tier3/harbor/secure_copy.py:1414
↓ 1 callersFunction_render
(skill_name: str, results: list[ValidationResult])
tests/reporting/browser_navigation_fixtures.py:105
↓ 1 callersFunction_render_current_report
()
src/skillevaluator/tier3/harbor/runner.py:6938
↓ 1 callersFunction_render_dimensions
( *, console: Console, agents: Mapping[str, Any], safe: Any, )
src/skillevaluator/tier3/result_display.py:903
↓ 1 callersFunction_render_environment_script
Render a sourceable script after validating every name and value.
src/skillevaluator/tier3/harbor/secure_docker_environment.py:827
↓ 1 callersFunction_render_feedback_and_suggestions
( *, console: Console, result: Mapping[str, Any], safe: Any, excluded_messages: set[str] |
src/skillevaluator/tier3/result_display.py:1034
↓ 1 callersMethod_render_finding_detail
Render detailed information for a single finding.
src/skillevaluator/reporting/markdown.py:700
↓ 1 callersMethod_render_findings
( self, lines: list[str], results: list[ValidationResult], private_labels: tup
src/skillevaluator/reporting/benchmark.py:879
↓ 1 callersFunction_render_findings_body
Render findings list into a Rich Text body (icon/label/score/question/reasons/evidence).
src/skillevaluator/tier3/harbor/report.py:384
↓ 1 callersMethod_render_freshness
(lines: list[str])
src/skillevaluator/reporting/benchmark.py:1105
↓ 1 callersFunction_render_html
(payload: dict)
ci/tier_manifest.py:430
← previousnext →7,301–7,400 of 24,741, ranked by callers