MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

↓ 1 callersFunction_isolated_skill_read
True if ``cmd`` is one plain read of one SKILL.md, also inside ``bash -c '...'`` wrappers.
src/skillevaluator/tier3/eval_core/checks.py:1743
↓ 1 callersFunction_issue_field
Return a getter that retrieves a field from *issue*, treating None as empty string.
src/skillevaluator/validators/security.py:1110
↓ 1 callersFunction_iter_custom_metric_candidates
(reward: dict[str, Any])
src/skillevaluator/tier3/harbor/metrics.py:233
↓ 1 callersFunction_iter_jsonl_dicts
Parse JSONL lines, skipping non-JSON noise (e.g. stderr mixed into tee logs).
src/skillevaluator/tier3/eval_core/log_converters.py:585
↓ 1 callersFunction_iter_paths
(root: Path)
src/skillevaluator/deduplication/utils/skill_collector.py:493
↓ 1 callersFunction_iter_repo_context_files
Enumerate repository files without inspecting excluded output trees.
src/skillevaluator/tier3/harbor/adapter.py:1255
↓ 1 callersFunction_jaccard
(a: frozenset[str], b: frozenset[str])
src/skillevaluator/harness/overlay.py:586
↓ 1 callersFunction_job
( job_dir: Path, scores: dict[str, float], cases: int, not_applicable: tuple[str, ...] = (),
tests/test_overall_cross_format_consistency.py:29
↓ 1 callersFunction_job_has_unrecovered_agent_timeout
Return whether a trial of the job, other than a recovered attempt, recorded ``AgentTimeoutError``. Checks the trial and each native step. A basel
src/skillevaluator/tier3/harbor/collector.py:7379
↓ 1 callersFunction_job_path_is_link_or_reparse
Return whether an attempt-job root is a symlink, junction, or reparse point.
src/skillevaluator/tier3/harbor/runner.py:4311
↓ 1 callersFunction_job_root_fingerprint
(metadata: os.stat_result)
src/skillevaluator/tier3/harbor/runner.py:4325
↓ 1 callersFunction_join_continuations
``text`` with each backslash-newline replaced by spaces of the same length (offsets stay the same).
src/skillevaluator/validators/mcp_static.py:3217
↓ 1 callersFunction_join_interrupted_agents
Wait while running agents stop their Harbor jobs after an interrupt. A second Ctrl+C ends the cleanup grace period; a third one propagates.
src/skillevaluator/tier3/harbor/runner.py:6699
↓ 1 callersFunction_joined
Bound a list before joining; the section's text cleaner bounds the cell.
src/skillevaluator/reporting/native_evidence.py:212
↓ 1 callersFunction_jpeg_tables_are_valid
Whether a DQT (0xDB), DHT (0xC4) or DAC (0xCC) segment payload holds whole, valid tables.
src/skillevaluator/validators/security.py:1366
↓ 1 callersFunction_js_code_tokens
Tokenize enough JavaScript to distinguish code references from prose.
src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py:445
↓ 1 callersFunction_js_match
Match ``pattern`` at ``index``, keeping only a match whose names JavaScript accepts.
src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py:279
↓ 1 callersFunction_json
(value: Any, depth: int = 0)
src/skillevaluator/reporting/retained.py:37
↓ 1 callersFunction_json
(value, depth: int = 0, budget: list[int] | None = None)
ci/tier_manifest.py:25
↓ 1 callersFunction_json
(raw: bytes)
ci/comment_report.py:44
↓ 1 callersFunction_json_keys
(path: Path, key: str | None = None)
src/skillevaluator/harness/plugin.py:78
↓ 1 callersFunction_json_safe_benchmark_policy
Project untrusted per-result policy metadata onto its two typed keys.
src/skillevaluator/reporting/json_reporter.py:62
↓ 1 callersFunction_json_safe_tier3_mapping_keys
Yield proof keys first without materializing an untrusted wide mapping.
src/skillevaluator/reporting/html.py:224
↓ 1 callersFunction_json_safe_tier3_value
Copy untrusted Tier 3 metadata into a finite JSON-compatible shape.
src/skillevaluator/reporting/html.py:235
↓ 1 callersFunction_json_string_length
Return the decoded length of a JSON string body: one character per escape sequence.
src/skillevaluator/utils/structured_data.py:270
↓ 1 callersFunction_json_text
(value: str)
src/skillevaluator/tier3/plugin_native.py:870
↓ 1 callersFunction_json_values
(payload: object, key: str)
tests/test_overall_cross_format_consistency.py:132
↓ 1 callersMethod_judge_attempts
Count the times the judge was asked for this skill, once a first reply came back.
src/skillevaluator/validators/rubric_eval.py:715
↓ 1 callersFunction_judge_aware_lift
Rebuild judge exclusions, falling back to recorded lift for incomplete loads.
src/skillevaluator/evaluation/tier3_report.py:2200
↓ 1 callersFunction_judge_evidence_line_value_spans
Return *text* as the line pass read it, and the spans of the values it replaced there to write *masked*. The line pass only replaces values with
src/skillevaluator/tier3/harbor/templates/eval.py:2922
↓ 1 callersFunction_judge_evidence_value_sites
Map the text that each of *lines* can leave after a marker to the spans of the values to mask in it. When an unquoted value that ends the line be
src/skillevaluator/tier3/harbor/templates/eval.py:2946
↓ 1 callersFunction_judge_goal_accuracy_custom
Fallback: two-step custom prompt mirroring RAGAS logic, with the same binary score.
src/skillevaluator/tier3/harbor/templates/eval.py:3658
↓ 1 callersFunction_judge_goal_accuracy_ragas
Use RAGAS AgentGoalAccuracyWithReference for high-quality two-step evaluation.
src/skillevaluator/tier3/harbor/templates/eval.py:3560
↓ 1 callersMethod_judge_labels
Name the provider and model that gave the score, when the judge answered. On the native Anthropic API the model can be the server-side fallba
src/skillevaluator/validators/rubric_eval.py:704
↓ 1 callersFunction_judge_mixed_lift_exclusions
Name the cases each judge metric's lift left out because the arms' judge models differed. Reads the logical attempts the lift rebuild compares, s
src/skillevaluator/evaluation/tier3_report.py:2164
↓ 1 callersFunction_judge_model_env_controls
Return reserved judge-model names used as keys or value references.
src/skillevaluator/tier3/harbor/adapter.py:5052
↓ 1 callersFunction_judge_models_by_case
Map each case to the judge models that scored ``metric`` in these rewards.
src/skillevaluator/tier3/harbor/collector.py:5565
↓ 1 callersFunction_judge_policies_valid
Return whether LLM_JUDGE_FALLBACK_POLICY and LLM_JUDGE_RETRY_POLICY both hold a known value.
src/skillevaluator/tier3/harbor/templates/eval.py:955
↓ 1 callersFunction_judge_retry_feedback
Retry instruction that names what was wrong with the previous reply.
src/skillevaluator/tier3/eval_core/llm_judge.py:556
↓ 1 callersFunction_judge_retry_feedback
Retry instruction that names what was wrong with the previous reply.
src/skillevaluator/tier3/harbor/templates/eval.py:2870
↓ 1 callersFunction_judgment
( requirement: dict[str, Any], reward: dict[str, Any], *, ambiguous: bool = False, missing: bool = False )
src/skillevaluator/evaluation/tier3_expectations.py:87
↓ 1 callersFunction_junit_file_path
Return a percent-encoded POSIX path relative to the scanned workspace, matching SARIF.
src/skillevaluator/reporting/junit_reporter.py:89
↓ 1 callersFunction_jwt
(payload_chars: int)
tests/reporting/test_tier3_failure_reasons.py:313
↓ 1 callersFunction_jwt_continued
A JWT split by a shell line continuation, inside a JSON string escaped *times* times, without its quotes.
tests/tier3/test_security_finding_evidence.py:1512
↓ 1 callersFunction_kata_environment_kwarg_error
Keep Kata's runtime overlay pointed at a Kata handler and literal DNS servers.
src/skillevaluator/tier3/harbor/runner.py:1774
↓ 1 callersFunction_keep_selected
Record which skills the tier selected, so a summary can tell 'not selected' from missing.
ci/run_tiers.py:682
↓ 1 callersFunction_keeps_patch_header
Return whether a header *path* stays in the command-text path checks of a shell apply_patch. A path holding shell syntax stays (``_APPLY_PATCH_SH
src/skillevaluator/tier3/eval_core/checks.py:1511
↓ 1 callersFunction_kept_with_failed_baseline
Return whether the in-memory or persisted data of ``agent`` keeps it with a failed baseline.
src/skillevaluator/tier3/harbor/report.py:181
↓ 1 callersFunction_keyed_client
Embed each known skill name as its own unit vector, whatever else its text says.
tests/embedding/test_catalog.py:628
↓ 1 callersFunction_keyword_rule_entry
Return the longest entry the keyword rule finds in lowercased command text. An entry counts where a path starts, except inside the file a source
src/skillevaluator/tier3/eval_core/checks.py:1181
↓ 1 callersFunction_known_cases
Return the case names an attempt may be matched to. The evaluation dataset decides when the report knows it: its IDs are the only canonical o
src/skillevaluator/tier3/token_usage.py:205
↓ 1 callersFunction_known_identity
(identity: dict, agent: str | None)
src/skillevaluator/tier3/history.py:111
↓ 1 callersFunction_known_mismatches
(current: dict, previous: dict, agent: str | None)
src/skillevaluator/tier3/history.py:127
↓ 1 callersMethod_lacks_geospatial_context
Return whether GPS-shaped pairs on *line* have no geospatial cue. Precise vectors, poses, quaternions, and waypoints have the same shape
src/skillevaluator/validators/security.py:6243
↓ 1 callersFunction_ladder_glue
Full plugin minus loose parts on the cross-component common cases, on all five dimensions. The cases are the ladder's own (so a case dropped anyw
src/skillevaluator/tier3/harbor/stats.py:1028
↓ 1 callersFunction_ladder_rungs
The four steps and the common-basis Bundle lift over one case set, plus the first step's full result. *split* marks a case-type group: as in :fun
src/skillevaluator/tier3/harbor/stats.py:994
↓ 1 callersFunction_late_observation_excerpts
Most-recent tool observations first (end-state evidence), each clipped.
src/skillevaluator/tier3/eval_core/atif_helpers.py:936
↓ 1 callersFunction_later_gate_fixes
One item for each gate failure after the first of a check that failed beside an incomplete scan. The check's own item names only its first gate f
src/skillevaluator/reporting/console_ui.py:1303
↓ 1 callersFunction_launch_cwd
The plugin folder an MCP server runs in, from its ``cwd``, or ``None``. A root variable or a ``./`` path names a plugin folder in any format.
src/skillevaluator/plugin_mcp.py:496
↓ 1 callersMethod_launcher_env
Return the minimal environment visible before confinement starts.
src/skillevaluator/tier3/harbor/local_environment.py:1501
↓ 1 callersFunction_layer_llm_insights
Append LLM-as-Judge conclusions/recommendations on top of the deterministic baselines. The judge never raises; when the LLM is unavailable the
src/skillevaluator/evaluation/tier3_report.py:1309
↓ 1 callersMethod_layers
()
src/skillevaluator/reporting/assets/chart.umd.min.js:13
↓ 1 callersFunction_lead_lines
The lines of one result block that can say the call failed: its first and last lines. Harness status lines are skipped, and an MCP content wrappe
src/skillevaluator/tier3/eval_core/plugin_signals.py:1831
↓ 1 callersFunction_leak_rate
Leaked trials over trials that carried a canary, or ``None`` when none did.
src/skillevaluator/tier3/eval_core/runtime_evidence.py:383
↓ 1 callersFunction_left_out
(arm_name: str, turns: _FirstTurns)
src/skillevaluator/tier3/harbor/stats.py:1328
↓ 1 callersFunction_legacy_completion_mtime
Return completion time for an unmarked pre-result run, if safely readable.
src/skillevaluator/tier3/results_location.py:832
↓ 1 callersFunction_legacy_dimensions_are_valid
(payload: object)
src/skillevaluator/tier3/results_location.py:715
↓ 1 callersFunction_legacy_summary_is_complete
Return whether a pre-result summary satisfies an authentic era schema.
src/skillevaluator/tier3/results_location.py:733
↓ 1 callersFunction_license_notice_comments
Read full comment lines, preserving shell/file boundaries around notices. A deliberately small file-write grammar admits source comments in cat/t
src/skillevaluator/tier3/eval_core/checks.py:2344
↓ 1 callersFunction_license_shell_line_contexts
Line beginnings proven outside shell quotes, continuations and substitutions. Substitution/parameter grammars can nest quotes, so once encountere
src/skillevaluator/tier3/eval_core/checks.py:2309
↓ 1 callersFunction_lift_excluded_case_count
Return how many cases a judge metric's lift left out because the arms' judge models differed.
src/skillevaluator/evaluation/tier3_report.py:2711
↓ 1 callersFunction_lift_overall_active_metrics
Return the active metric set when each of its metrics is scored by both arms or explained by N/A.
src/skillevaluator/tier3/harbor/collector.py:5696
↓ 1 callersFunction_lift_text
(delta: float)
src/skillevaluator/tier3/commands.py:1708
↓ 1 callersFunction_line_failed
Whether one lead line says the call failed (see the marker notes above).
src/skillevaluator/tier3/eval_core/plugin_signals.py:1860
↓ 1 callersFunction_link_directory
Create a directory link: a junction on native Windows, a symlink elsewhere.
tests/utils/test_secure_fs_platform.py:380
↓ 1 callersFunction_link_file
(link: Path, target: str)
tests/utils/test_secure_fs_platform.py:395
↓ 1 callersFunction_link_leaves_root
Whether a link (or reparse point) leads outside the plugin root; unresolvable links count as outside. An absolute target is outside: it names a h
src/skillevaluator/plugin_paths.py:528
↓ 1 callersFunction_linked_directory_contains_skill
Inspect a linked target for manifests without following further links.
src/skillevaluator/utils/helpers.py:29
↓ 1 callersFunction_links
( directory: Path | None, report_url: str | None = None, *, anchor: str | None = None )
ci/report_views.py:156
↓ 1 callersMethod_lint_folder
(self, root: Path)
src/skillevaluator/validators/script_lint.py:115
↓ 1 callersMethod_lint_script
Run all lint checks on a single Python script.
src/skillevaluator/validators/script_lint.py:173
↓ 1 callersFunction_list_posix_directory
Return one held directory's entries with their no-follow metadata, sorted by name.
src/skillevaluator/utils/secure_fs.py:1222
↓ 1 callersFunction_listed
(items: list[str])
src/skillevaluator/tier3/next_steps.py:66
↓ 1 callersFunction_listing_grid
(rows: list[dict[str, Any]])
src/skillevaluator/harness/report.py:166
↓ 1 callersFunction_litellm_model
(provider: str, model: str)
src/skillevaluator/inference/client.py:949
↓ 1 callersFunction_literal_compose_build_args
(build: dict[str, Any])
src/skillevaluator/tier3/harbor/adapter.py:3080
↓ 1 callersFunction_literal_deduct_amounts
Every literal ``deduct(<int>, ...)`` amount in one function, in source order.
tests/test_metrics_reference_docs.py:411
↓ 1 callersMethod_live_capable
Live redraws need a real, capable TTY, as Harbor progress requires. FORCE_COLOR/TTY_COMPATIBLE make Rich treat a CI pipe as a terminal; that
src/skillevaluator/reporting/console_ui.py:566
↓ 1 callersFunction_live_row
Run one live request; report only a bounded diagnostic, never a provider body.
src/skillevaluator/doctor.py:125
↓ 1 callersFunction_llm_actions
( issues: list[dict[str, Any]], skill_path: Path | None, dataset: Mapping[str, Mapping[str, Any]] )
src/skillevaluator/tier3/case_summary.py:1268
↓ 1 callersFunction_llm_scope
(models: list[dict], context: dict)
ci/report_views.py:220
↓ 1 callersFunction_llm_verdicts
(profile, candidates, *, llm_model, result)
src/skillevaluator/deduplication/plugin/catalog_checks.py:141
↓ 1 callersFunction_load
(import_path: str)
tests/test_harbor_claude_refusal_fallback.py:522
↓ 1 callersFunction_load
()
tests/test_openclaw_plugin_smoke.py:18
↓ 1 callersMethod_load
(self, position: int)
src/skillevaluator/plugin_component_risk.py:3015
↓ 1 callersFunction_load_atomic_rename_backend
()
src/skillevaluator/tier3/harbor/secure_copy.py:136
↓ 1 callersFunction_load_baseline
The host checker and the template as they stood at a git ref.
scripts/script_invocation_differential.py:1159
↓ 1 callersFunction_load_bounded_jsonl
(raw: bytes, diagnostics: list[dict[str, Any]])
src/skillevaluator/tier3/harbor/report_data.py:502
↓ 1 callersFunction_load_custom_backend
(env_mode: str)
src/skillevaluator/tier3_environments.py:396
↓ 1 callersFunction_load_entries_by_id
(skill_path: Path)
src/skillevaluator/tier3/harbor/adapter.py:5710
← previousnext →6,701–6,800 of 24,741, ranked by callers