MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

↓ 2 callersFunctionissue_diagnostics
Finding metadata that explains an AE1 *issue*: the target, its reasons and the reference lines. ``skillspector_evidence`` holds ``target_path``,
src/skillevaluator/skillspector_references.py:287
↓ 2 callersFunctioniter_config_strings
Yield ``(json_path, string)`` for string leaves of a config value (bounded, iterative).
src/skillevaluator/plugin_permissions.py:86
↓ 2 callersFunctioniter_hook_handlers
Yield ``(event, group_index, matcher, handler_index, handler)`` for a hooks config. Accepts the ``hooks.json`` / settings shape ``{"hooks": {Even
src/skillevaluator/plugin_component_risk.py:1147
↓ 2 callersFunctioniter_hook_handlers
Yield ``(event, group_index, group, handler_index, handler)`` for a hooks config. Accepts the ``hooks.json`` / settings shape ``{"hooks": {Event:
src/skillevaluator/tier3/plugin_native.py:372
↓ 2 callersFunctionj
(t,e,i)
src/skillevaluator/reporting/assets/chart.umd.min.js:7
↓ 2 callersFunctionje
(t,e=[""],i,s,n=(()=>t[0]))
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersFunctionjudge
(*_args: object, **_kwargs: object)
tests/tier3/test_view_results_feedback.py:102
↓ 2 callersFunctionjunit_target_identity
Return a stable identity for the evaluated target. The target's path relative to the repository root comes first, then its path relative to t
src/skillevaluator/reporting/junit_reporter.py:125
↓ 2 callersFunctionke
(t,e,i)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersFunctionkn
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersFunctionlabel
(relative: Path)
ci/diagnostics.py:229
↓ 2 callersMethodlabel
(self, value: object)
src/skillevaluator/reporting/plugin_static.py:181
↓ 2 callersMethodlinkScales
()
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersFunctionln
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersFunctionload_allowed_runtime_env
Load declared harbor.runtime_env mapping from a skill's evals config if present.
src/skillevaluator/tier3/evals_config.py:867
↓ 2 callersFunctionload_allowlist
Load a strict anchor-specific allowlist; unknown metadata fails closed.
scripts/check_oss_boundary.py:656
↓ 2 callersFunctionload_pii_waivers
Load a version-1 PII waiver file, rejecting any entry that is not exactly valid. Expired waivers load (they are reported, not applied); a waiver
src/skillevaluator/validators/pii_waivers.py:151
↓ 2 callersFunctionload_plugin_runtime_report
Replay a selected snapshot; missing older snapshots are explicitly unavailable.
src/skillevaluator/reporting/plugin_runtime_io.py:156
↓ 2 callersFunctionload_staged_harbor_dataset
Load and deduplicate dataset entries staged into Harbor task trees.
src/skillevaluator/tier3/harbor/report_data.py:1308
↓ 2 callersMethodlocation_occurrence_id
Return the occurrence ID this finding has at another location. A check that folds repeats into one finding, such as the PII scan, lis
src/skillevaluator/models/result.py:209
↓ 2 callersFunctionloose_parts_agent_import_path
The Harbor wrapper that runs the loose arm's setup before ``agent`` starts, or raise ``LoosePartsUnsupported``.
src/skillevaluator/tier3/loose_parts.py:644
↓ 2 callersMethodlsp_servers_for
The LSP servers ``agent``'s ``arm`` stages, with their file extensions (a native with-plugin arm only).
src/skillevaluator/tier3/eval_core/plugin_signals.py:1399
↓ 2 callersFunctionlt
(t)
src/skillevaluator/reporting/assets/chart.umd.min.js:7
↓ 2 callersFunctionluminance
(channels)
tests/reporting/test_report_browser.py:466
↓ 2 callersFunctionmain
Render the app, accepting CLI paths or a supplied path list for embedding.
src/skillevaluator/reporting/dashboard_app.py:310
↓ 2 callersFunctionmake_collected_file
Factory for CollectedFile objects with sensible defaults.
tests/deduplication/conftest.py:26
↓ 2 callersFunctionmake_request
Make an HTTP request and return the response.
src/skillevaluator/tier3/reference_skills/api-caller/scripts/call_api.py:19
↓ 2 callersFunctionmake_skills_only_plugin
A plugin whose only loadable parts are skills (plus glue): loose parts equal the skills alone.
tests/tier3/private_loose_fixtures.py:121
↓ 2 callersFunctionmanifest_relative_path
Return the root-relative manifest path that *path* names, if it names one. ``agent_plugin.yaml``/``.yml``, a root ``plugin.json`` (Agent Plugins
src/skillevaluator/plugin_manifest.py:379
↓ 2 callersFunctionmask
(view: str)
src/skillevaluator/tier3/eval_core/checks.py:3992
↓ 2 callersFunctionmask_line
(line)
src/skillevaluator/tier3/harbor/templates/eval.py:3052
↓ 2 callersMethodmatcher
Return a check for PII candidates in ``file_path``, or ``None`` when no waiver names the file. The check takes a candidate's category, line,
src/skillevaluator/validators/pii_waivers.py:321
↓ 2 callersFunctionmatcher_sensitive_tools
The write, fetch, and MCP tools (``Write``, ``Edit``, ``MultiEdit``, ``NotebookEdit``, ``WebFetch``, ``mcp__*``) a matcher matches, read the same
src/skillevaluator/plugin_component_risk.py:1596
↓ 2 callersFunctionmcp_container_image
Return the image reference a ``docker|podman|nerdctl run`` MCP server launches, if any. Read through :func:`parse_mcp_runner` (wrappers such as `
src/skillevaluator/validators/mcp_static.py:2661
↓ 2 callersFunctionmcp_finding
An ``MCP_DECLARATION`` finding; with ``name``, the message starts with ``mcpServers['<name>']: ``.
src/skillevaluator/validators/mcp_static.py:368
↓ 2 callersFunctionmcp_servers_error
Return why ``evals/environment/mcp_servers.toml`` blocks a Gym run, or None when it does not.
src/skillevaluator/tier3/nemo_gym/backend.py:323
↓ 2 callersFunctionmember_cost_rows
Selected effective members' text and static first-turn hook output, without reading runtime state.
src/skillevaluator/plugin_context_cost.py:414
↓ 2 callersFunctionmerge_catalog_sarif_documents
Merge per-skill SARIF payloads into one upload-ready document. GitHub Code Scanning rejects SARIF with more than 20 runs, so child reports ar
src/skillevaluator/reporting/sarif_reporter.py:279
↓ 2 callersFunctionmerge_finding_summaries
Merge per-result summaries without hashing the same findings again.
src/skillevaluator/reporting/finding_summary.py:160
↓ 2 callersFunctionmetadata_id_duplicate_severity
Severity of a duplicate ``metadata.id`` under *policy* (HIGH only when IDs are required).
src/skillevaluator/validators/schema.py:168
↓ 2 callersFunctionmetadata_id_problem
Return why *value* breaks the ``metadata.id`` contract, or ``None``.
src/skillevaluator/validators/schema.py:118
↓ 2 callersFunctionmetrics_for_agents
Return the canonical default or legacy metric set represented by agents.
src/skillevaluator/tier3/harbor/report_data.py:570
↓ 2 callersFunctionmodel_hidden_traits
``{"model_hidden"}`` when ``disable-model-invocation`` is on (as Claude Code reads it), else empty.
src/skillevaluator/plugin_context_cost.py:204
↓ 2 callersFunctionmore_case_rows_note
Say how many case rows a table left out; ``code`` formats the payload key.
src/skillevaluator/reporting/tier3_token_usage.py:144
↓ 2 callersFunctionmr_checkout
Give an MR route the real checkout and pinned tools its skip check reads.
tests/ci/test_gitlab_enriched_admission.py:73
↓ 2 callersFunctionms
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersMethodnames
(self, value: object, limit: int = 12)
src/skillevaluator/reporting/plugin_static.py:185
↓ 2 callersFunctionnative_agent_import_path
Return the native Harbor wrapper for ``agent`` over its routing wrapper, or raise ``PluginLoadError``.
src/skillevaluator/tier3/plugin_native.py:2312
↓ 2 callersFunctionnative_mcp_servers_declared
Return True when a native ``evals/harbor/*/task.toml`` declares ``[[environment.mcp_servers]]``.
src/skillevaluator/tier3/nemo_gym/backend.py:337
↓ 2 callersFunctionnemo_gym_sandbox_error
Return an error when ``SKILL_EVAL_NEMO_GYM_SANDBOX`` names a sandbox the backend does not support.
src/skillevaluator/tier3/nemo_gym/backend.py:265
↓ 2 callersFunctionnemo_gym_timeout_s
Return ``SKILL_EVAL_NEMO_GYM_TIMEOUT_S`` in seconds, or raise ``ValueError``.
src/skillevaluator/tier3/nemo_gym/backend.py:253
↓ 2 callersFunctionnemo_gym_version
Return the installed NeMo Gym version from the worker's own interpreter. ``-P`` keeps the caller's directory off ``sys.path``, so a ``nemo_gym``
src/skillevaluator/tier3/nemo_gym/backend.py:84
↓ 2 callersFunctionneutral_selection_summary
()
tests/ci/test_report_views.py:853
↓ 2 callersFunctionnewer_than_0_9_0
Whether ``release`` is ``## Unreleased`` or the heading of a release after 0.9.0.
tests/changelog_entries.py:35
↓ 2 callersFunctionnormalize_census
Validate a census object's shape; ``None`` when it is not a census. Keeps the ``loaded``, ``listed``, ``staged``, and ``not_loaded`` lists (b
src/skillevaluator/tier3/plugin_native.py:2412
↓ 2 callersFunctionnormalized_tool_call_error_flag
Return an attributable result's error flag, including metadata-free skill lookup failures. ``True`` if any attributable result reports an error,
src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py:101
↓ 2 callersFunctionnot_applicable_reason
Return why *metric* can be N/A in an arm with or without the skill under test.
src/skillevaluator/tier3/harbor/metrics.py:135
↓ 2 callersFunctionoa
(t)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersFunctionobservation_usage
Normalize observed counters once; absent counters and actual zero remain distinct.
src/skillevaluator/tier3/plugin_comparisons.py:48
↓ 2 callersFunctionobservations
(case, reward, thresholds)
src/skillevaluator/tier3/grader_validation.py:312
↓ 2 callersFunctionobserve
(**kwargs: Any)
tests/test_tier3_config_key_parity.py:348
↓ 2 callersMethodof
``entries`` itself when already parsed, else parsed from policy strings.
src/skillevaluator/validators/mcp_static.py:3007
↓ 2 callersFunctionon_output
(text: str, _stream: str)
tests/test_harbor_local_mode.py:305
↓ 2 callersFunctionoutput_provenance_key_path
Return the configured private-key path without creating the key.
src/skillevaluator/tier3/output_provenance.py:345
↓ 2 callersMethodparse
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersMethodparse_response
(self, response_text: str, **_kwargs: Any)
src/skillevaluator/validators/rubric_eval.py:351
↓ 2 callersMethodparse_response
Parse the raw LLM response text. Override in subclasses.
src/skillevaluator/inference/client.py:895
↓ 2 callersFunctionparse_rules_manifest
Parse .mdc rule file into RulesManifest (utility function).
src/skillevaluator/validators/rules_schema.py:249
↓ 2 callersMethodpatch
(self, path: str, payload: dict)
ci/github_api.py:109
↓ 2 callersFunctionper_agent_token_usage
Return ``usage`` without its per-case rows, for the terminal summary of a run.
src/skillevaluator/tier3/token_usage.py:644
↓ 2 callersMethodpinning
(self)
src/skillevaluator/plugin_mcp.py:111
↓ 2 callersFunctionplanned_attempts
Return the upper bound used for fail-fast admission checks.
src/skillevaluator/tier3/harbor/run_budget.py:94
↓ 2 callersFunctionplugin_mcp_tool_prefix
Claude Code's permission name for one plugin MCP server's tools: ``mcp__plugin_<plugin>_<server>``. A ``query`` tool on server ``db`` of plugin `
src/skillevaluator/plugin_component_risk.py:776
↓ 2 callersMethodpoll_cases
Push live per-case rows for the current plan to the view.
src/skillevaluator/reporting/console_ui.py:174
↓ 2 callersMethodpost
(self, path, payload)
tests/ci/test_feedback.py:136
↓ 2 callersFunctionprebuild_task_environments
Pre-build Docker images for generated task environments. After ``generate_harbor_tasks`` has written all task directories, this function buil
src/skillevaluator/tier3/harbor/adapter.py:1660
↓ 2 callersFunctionpreferred_skill_manifests
Pick one manifest per skill folder, ``SKILL.md`` over ``skill.md``. ``files`` are skill manifests (each named like one of :data:`~skillevalua
src/skillevaluator/plugin_member_paths.py:35
↓ 2 callersFunctionprefix_finding
Copy *finding* with *prefix* applied to every location it names. Used when a sub-result is folded into a parent result (a skill inside a coll
src/skillevaluator/models/result.py:351
↓ 2 callersFunctionpreflight_qualification
Check label coverage only; scored counts and rate checks remain unobserved.
src/skillevaluator/tier3/grader_policy.py:258
↓ 2 callersFunctionprepare_skillspector
Download the SkillSpector wheel and build its uv environment; return an error or None.
src/skillevaluator/utils/scanner_install.py:113
↓ 2 callersFunctionprivate_scan_pids
Read the PIDs of the scan for report directory ``name`` from its private directory under TMPDIR.
tests/ci/test_run_tier1.py:1511
↓ 2 callersFunctionprivilege_summary
The ``plugin.privileges`` metadata block.
src/skillevaluator/plugin_component_risk.py:3948
↓ 2 callersFunctionprobe_interrupted_process
(pid_path: str)
tests/test_harbor_secure_docker_environment.py:8095
↓ 2 callersMethodprocess
Ask the judge, and ask once more, naming the problem, if its reply cannot be scored. A reply that ``json`` cannot parse (any ``ValueError`` o
src/skillevaluator/validators/rubric_eval.py:354
↓ 2 callersFunctionprocess_exists
(pid: int)
tests/test_harbor_local_mode.py:2839
↓ 2 callersFunctionprompt
(command: str)
tests/tier3/test_security_finding_evidence.py:1471
↓ 2 callersMethodprovenance
(self)
src/skillevaluator/tier3/plugin_evidence.py:586
↓ 2 callersFunctionpublication_evidence_identity
Parse a marker only when every field exactly matches the built-in contract.
src/skillevaluator/publication_evidence.py:64
↓ 2 callersFunctionpublication_free_text
Normalize free text like :func:`publication_semantic_text`, keeping layout whitespace. Every other control, format, or invisible identity charact
src/skillevaluator/publication_text.py:362
↓ 2 callersFunctionpublication_paths_skipped_by_scan
Return digest-covered paths that a name-based producer scan skips. The Tier 1 file-walk checks and Tier 2 skip generated-artifact names (``re
src/skillevaluator/publication_identity.py:221
↓ 2 callersFunctionpublication_source_path_is_excluded
Return whether a path is at or below an entry excluded by v2.
src/skillevaluator/publication_identity.py:208
↓ 2 callersFunctionpublication_target_dict
Project an untrusted target claim to its canonical three fields.
src/skillevaluator/reporting/base.py:834
↓ 2 callersFunctionpull_request
Return whether the CI event evaluates a pull or merge request diff.
ci/select_changed_skills.py:299
↓ 2 callersFunctionpypi_pin
The version a PyPI runner requirement pins, else ``None``. Reads ``pkg==V``, ``pkg===V``, uv's ``pkg@V``, extras (``pkg[x]==V``), and the PEP
src/skillevaluator/validators/mcp_static.py:2302
↓ 2 callersFunctionq
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:7
↓ 2 callersFunctionqe
(t,e,i)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersFunctionqs
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersFunctionqt
(t,e)
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 2 callersFunctionread
(descriptor, amount)
tests/tier3/test_private_plugin_security.py:439
↓ 2 callersFunctionread_hook_script
Text of a script a hook runs from the plugin root (``${CLAUDE_PLUGIN_ROOT}``), for the hook risk analyzer. Returns the whole file (at most ``MAX_
src/skillevaluator/plugin_config_risk.py:267
← previousnext →5,101–5,200 of 24,741, ranked by callers