MCPcopy Create free account

hub / github.com/NVIDIA/SkillEvaluator / functions

Functions24,741 in github.com/NVIDIA/SkillEvaluator

↓ 1 callersFunction_artifact_loading_reasons
Collect only bounded, schema-checked loader diagnostics for the report.
src/skillevaluator/evaluation/tier3_report.py:222
↓ 1 callersFunction_artifact_redirect
Resolve one object-storage redirect to an HTTPS URL without credentials.
ci/feedback.py:564
↓ 1 callersFunction_artifact_snapshot_is_unchanged
Re-read one artifact to detect ordinary changes during validation.
src/skillevaluator/tier3/results_location.py:227
↓ 1 callersFunction_assemble
Join (title, body) sections under a char budget. Returns (text, dropped, truncated).
src/skillevaluator/tier3/eval_core/atif_helpers.py:949
↓ 1 callersFunction_assert_output_parent_still_declared
Reject a lexical output parent that no longer names the pinned directory.
src/skillevaluator/tier3/output_provenance.py:130
↓ 1 callersFunction_assert_real_hermes_artifacts
(results_dir)
tests/test_hermes_agent.py:506
↓ 1 callersFunction_assert_what_to_fix_lists_the_hook_gate
The "what to fix" items of the Tier 1 terminal summary show the hook gate failure. The check's own item names its first gate failure, which besid
tests/validators/test_skillspector_hook_surface.py:378
↓ 1 callersFunction_assign_case_identity
(data: dict[str, Any], result: dict[str, Any], trial_name: str)
src/skillevaluator/tier3/harbor/collector.py:3705
↓ 1 callersFunction_assignment_value_match
Return the value that starts at *start* in *text*, read as a call only when that runs past the word reading. A call whose arguments hold a quote
src/skillevaluator/tier3/eval_core/checks.py:5244
↓ 1 callersMethod_atom
One atom, and whether a quantifier may follow it.
src/skillevaluator/plugin_component_risk.py:1310
↓ 1 callersFunction_attach_activation_expectations
Compare authored expectations with observed use, independently of correctness.
src/skillevaluator/evaluation/tier3_report.py:3724
↓ 1 callersFunction_attach_agent_report_details
Populate bounded diagnostic details after the best agent is known. Global report limits intentionally prioritize the best-scoring agent. This
src/skillevaluator/evaluation/tier3_report.py:2628
↓ 1 callersFunction_attach_baseline_pairs
Pair with-skill trials to their baseline counterparts by ``entry_id``. Adds ``baseline_overall`` / ``baseline_scores`` / ``lift_scores`` to each
src/skillevaluator/evaluation/tier3_report.py:3795
↓ 1 callersFunction_attach_tool_result
( steps: list[dict[str, Any]], tool_use_id: str, content_text: str, *, is_error: bool = False )
src/skillevaluator/tier3/eval_core/log_converters.py:57
↓ 1 callersFunction_attach_truncation
(target: dict[str, Any], diagnostics: list[dict[str, Any]])
src/skillevaluator/tier3/harbor/report_data.py:281
↓ 1 callersFunction_attempt
(root, case, repeat, output, thresholds, timeout, provider_env=None)
src/skillevaluator/tier3/grader_validation.py:422
↓ 1 callersFunction_attempt_coverage
(data: Mapping[str, Any])
src/skillevaluator/tier3/result_display.py:358
↓ 1 callersFunction_attempt_job_stats
Read one per-attempt Harbor job result for merging; ``None`` when unreadable.
src/skillevaluator/tier3/harbor/runner.py:4261
↓ 1 callersFunction_attempt_summary
Overall and per-metric scores, measured resources and the first warning of one attempt.
src/skillevaluator/evaluation/tier3_feedback_chat.py:801
↓ 1 callersFunction_attempt_tokens
Normalize a report trial's or usage-only attempt's ``tokens`` and its ``invalid_token_counters``.
src/skillevaluator/tier3/token_usage.py:196
↓ 1 callersFunction_attempts
Whether ``call`` tried to use something ``refs`` names, whatever the outcome. A ``SKILL.md`` read that failed by itself is not an attempt (the fi
src/skillevaluator/tier3/eval_core/plugin_signals.py:4223
↓ 1 callersFunction_attribute_changes
The attribute letters (``_ATTRIBUTE_LETTERS``) a declaration's options add with ``-`` and remove with ``+``.
src/skillevaluator/tier3/eval_core/script_invocation.py:1006
↓ 1 callersFunction_attribution
(with_skill_command: str, baseline_command: str)
tests/tier3/test_security_finding_evidence.py:890
↓ 1 callersMethod_audit_dependencies
Check for unpinned or banned packages in dependency files.
src/skillevaluator/validators/hygiene.py:122
↓ 1 callersMethod_audit_requirements
Audit a requirements.txt file.
src/skillevaluator/validators/dependencies.py:188
↓ 1 callersFunction_audit_row
(member: PluginMember)
src/skillevaluator/tier3/plugin_prepare.py:373
↓ 1 callersFunction_augment
( reporter: ReporterBase, document: str, results: list[ValidationResult], projection: StaticPl
src/skillevaluator/reporting/plugin_static_render.py:107
↓ 1 callersFunction_augment
(reporter: ReporterBase, document: str, payload: dict)
src/skillevaluator/reporting/plugin_runtime_render.py:31
↓ 1 callersMethod_author_usernames
Usernames derived from the skill's declared ``author`` frontmatter. Reads the SKILL.md manifest in ``skill_path`` (if any) and extracts the
src/skillevaluator/validators/security.py:3149
↓ 1 callersFunction_authored_field
The field name the author wrote: the agentskills.io name when the runtime value came from it.
src/skillevaluator/tier3/evals_spec.py:911
↓ 1 callersFunction_authoritative_step_names
Return ordered authoritative Harbor step names, if this is multi-step.
src/skillevaluator/tier3/harbor/collector.py:6305
↓ 1 callersMethod_auto_approve
(facts: list[_ShellFacts], site: _HookSite)
src/skillevaluator/plugin_component_risk.py:3709
↓ 1 callersFunction_auto_trial_inputs_are_regular
Check every identity/trajectory input before automatic discovery reads it.
src/skillevaluator/tier3/generate_dataset.py:1034
↓ 1 callersFunction_available
(module: str)
src/skillevaluator/cli.py:764
↓ 1 callersFunction_backend_config_prerequisite_errors
Check Runta and Mosaic credential inputs against what Harbor's child will see.
src/skillevaluator/tier3/harbor/runner.py:2354
↓ 1 callersFunction_backend_may_forward
(name: str)
src/skillevaluator/tier3/harbor/runner.py:1355
↓ 1 callersMethod_background_command
(command: str)
src/skillevaluator/tier3/harbor/local_environment.py:1696
↓ 1 callersFunction_balanced_json_container_end
Return the exclusive end of one bounded structural container.
src/skillevaluator/tier3/harbor/templates/eval.py:2508
↓ 1 callersFunction_bare_name_variants
Return the bare file name of a nested file, such as ``helper.md`` for ``lib/helper.md``. Guidance often names a helper by file name alone. A name
src/skillevaluator/validators/skillspector_scope.py:663
↓ 1 callersMethod_base
(self)
src/skillevaluator/validators/pii_waivers.py:276
↓ 1 callersFunction_base64_texts
Text decoded from the first base64 runs in ``text`` (an allow piped through ``base64 -d``).
src/skillevaluator/plugin_component_risk.py:1888
↓ 1 callersFunction_base_tool_name
(fn: str)
src/skillevaluator/tier3/eval_core/plugin_signals.py:2096
↓ 1 callersFunction_baseline_glob_has_literal_scope
Return whether a SkillSpector suppression glob has a literal run of ``min_run`` characters. Wildcards, negated classes, and multi-character class
src/skillevaluator/validators/security.py:805
↓ 1 callersFunction_baseline_only_dimension
Describe one dimension of a baseline-only run (--skip-with-skill). The baseline is the measurement here, not a comparison point: it drives the
src/skillevaluator/evaluation/tier3_report.py:2820
↓ 1 callersFunction_baseline_saturated
Flag agents that pass every case without the skill: the eval cannot show what the skill adds.
src/skillevaluator/tier3/case_summary.py:1541
↓ 1 callersFunction_baseline_timeout_failures
Gather the typed failures of a short baseline arm and check T2 to T7. Returns ``(failures, None)`` when every rule holds, otherwise ``([], co
src/skillevaluator/tier3/harbor/collector.py:7431
↓ 1 callersFunction_bash_patterns
OpenCode bash patterns for a Claude ``Bash(<spec>)`` rule (``prefix:*`` is a command prefix).
src/skillevaluator/tier3/plugin_native.py:1900
↓ 1 callersFunction_batches
(pins: tuple[tuple[str, str], ...])
src/skillevaluator/plugin_dependency_audit.py:455
↓ 1 callersFunction_bedrock_active_credential_service_models
(session: object | None)
src/skillevaluator/tier3/harbor/runtime_preflight.py:232
↓ 1 callersFunction_bedrock_client_error_kind
Classify a sanitized AWS service error, including credential providers.
src/skillevaluator/tier3/harbor/runtime_preflight.py:1172
↓ 1 callersFunction_bedrock_client_uses_tls
(client: object)
src/skillevaluator/tier3/harbor/runtime_preflight.py:374
↓ 1 callersFunction_bedrock_endpoint_metadata_is_builtin
Require endpoint authority to come from Botocore's packaged data.
src/skillevaluator/tier3/harbor/runtime_preflight.py:403
↓ 1 callersFunction_bedrock_exception_result
Convert a Bedrock SDK failure to a redacted, policy-safe result.
src/skillevaluator/tier3/harbor/runtime_preflight.py:1216
↓ 1 callersFunction_bedrock_refusal_fallback_model
Return the Bedrock refusal fallback ID for a Claude 5.5 or later model, else None. A bare name such as ``claude-opus-4-8`` gets the request model
src/skillevaluator/tier3/harbor/templates/eval.py:2102
↓ 1 callersFunction_behavior_evidence_completeness
Report which behavior evidence sections reached the judge whole. Every section is compared against its complete source rather than against an
src/skillevaluator/tier3/eval_core/atif_helpers.py:1173
↓ 1 callersFunction_behavior_expectations
( expected_behaviors: list[str], results: list[dict[str, Any]], error: str = "" )
src/skillevaluator/tier3/eval_core/llm_judge.py:1133
↓ 1 callersFunction_behavior_prompt_example
One entry per expected behavior, so the example is a valid reply if a judge copies it.
src/skillevaluator/tier3/eval_core/llm_judge.py:1026
↓ 1 callersFunction_behavior_prompt_example
One entry per expected behavior, so the example is a valid reply if a judge copies it.
src/skillevaluator/tier3/harbor/templates/eval.py:3718
↓ 1 callersFunction_behavior_user_request_limit
Return the behavior evidence allowance for the user request section. The old fixed 800 characters cut ordinary Tier 3 task prompts, but the w
src/skillevaluator/tier3/eval_core/atif_helpers.py:296
↓ 1 callersFunction_behavior_warning
(score: float, result: Mapping[str, Any])
src/skillevaluator/tier3/attempt_verdict.py:114
↓ 1 callersFunction_benchmark_policy
Resolve the persisted publication policy for all configurable tiers.
src/skillevaluator/reporting/benchmark.py:1138
↓ 1 callersFunction_best_elapsed
(pattern, text: str, repeats: int = 5)
tests/tier3/test_github_token_redaction.py:76
↓ 1 callersFunction_bind
(key: str, entry_id: str)
src/skillevaluator/tier3/harbor/collector.py:3576
↓ 1 callersFunction_bind_evaluated_source
Bind trusted run context; retain conflicting claims separately and fail closed. Reporters reject contradictory source carriers. The working rende
ci/benchmark_reports.py:89
↓ 1 callersFunction_bind_identity
Preserve claims exactly and fail closed even on case-only source conflicts.
ci/benchmark_reports.py:71
↓ 1 callersFunction_bind_unsettled
Bind ``name`` to a value the text does not settle: a declaration whose options the walk does not model.
src/skillevaluator/tier3/eval_core/script_invocation.py:940
↓ 1 callersFunction_binding_readings
The binding rules a text is walked under. The tool's own shell is read as bash, as elsewhere in this walk. ``sh`` is dash on some systems, ba
src/skillevaluator/tier3/eval_core/script_invocation.py:883
↓ 1 callersFunction_blob
(content)
ci/benchmark_publish.py:105
↓ 1 callersFunction_blob
(workspace: Path, revision: str, path: str, limit: int)
ci/security_baselines.py:65
↓ 1 callersFunction_blocked
(name: str)
tests/test_precommit.py:486
↓ 1 callersFunction_blocking_finding
(result: ValidationResult)
src/skillevaluator/reporting/run_status.py:105
↓ 1 callersFunction_bool
(value: Any, config_path: Path, field: str)
src/skillevaluator/tier3/evals_config.py:390
↓ 1 callersFunction_bound
(value: float)
src/skillevaluator/reporting/html.py:1923
↓ 1 callersFunction_bound_launch_errors
Return whether ``errors`` are exactly the runner messages the timed-out attempts produce. An arm-level job reports one ``<agent> without-skill Ha
src/skillevaluator/tier3/harbor/baseline_timeouts.py:172
↓ 1 callersFunction_bounded_cases
Keep case rows up to the evidence budget; return them and the rows left out per agent. The budget is spent one case at a time across agents, so a
src/skillevaluator/evaluation/tier3_feedback_chat.py:901
↓ 1 callersMethod_bounded_cells
(self, values: list)
src/skillevaluator/reporting/plugin_static.py:219
↓ 1 callersFunction_bounded_custom_metric_names
Return a bounded custom-name sample and whether more names may exist.
src/skillevaluator/evaluation/tier3_report.py:2969
↓ 1 callersFunction_bounded_dataset_payload
Slice list-like dataset shapes before normalization copies their records.
src/skillevaluator/tier3/harbor/report_data.py:474
↓ 1 callersFunction_bounded_findings_payload
Project a findings artifact into a fixed, browser-loadable schema.
src/skillevaluator/tier3/harbor/report.py:1139
↓ 1 callersFunction_bounded_pieces
Cap provider inputs, keeping meaningful tails and source containment. The minimum-length filter applies to the original chunk. Its final short
src/skillevaluator/deduplication/utils/chunker.py:72
↓ 1 callersFunction_bounded_raw_metric_mapping
Keep a deterministic representative slice of raw custom metric maps.
src/skillevaluator/evaluation/tier3_report.py:1450
↓ 1 callersFunction_bounded_rows_payload
()
tests/reporting/test_private_runtime_replay.py:134
↓ 1 callersFunction_bounded_smallest
Select deterministic paths while bounding both visits and retained paths.
src/skillevaluator/tier3/harbor/report_data.py:297
↓ 1 callersFunction_bounded_staged_entry_files
Find staged entry files without allowing ``rglob`` to hide unbounded visits.
src/skillevaluator/tier3/harbor/report_data.py:331
↓ 1 callersFunction_bounded_tier3_preview_value
(value: Any, budget: _Tier3PreviewBudget)
src/skillevaluator/reporting/html.py:1661
↓ 1 callersMethod_braced
(self)
src/skillevaluator/plugin_component_risk.py:1287
↓ 1 callersMethod_bridge_client_environment
(self)
src/skillevaluator/tier3/harbor/local_agents.py:691
↓ 1 callersMethod_bridge_client_environment
(self)
src/skillevaluator/tier3/harbor/local_agents.py:1025
↓ 1 callersMethod_bridge_result_stdout
(result: object)
src/skillevaluator/tier3/harbor/local_agents.py:747
↓ 1 callersFunction_broad
(values: Iterable[str])
src/skillevaluator/plugin_permissions.py:301
↓ 1 callersFunction_budget_usage
Read attempt usage from Harbor's retained trajectory and judge artifacts.
src/skillevaluator/tier3/harbor/runner.py:4768
↓ 1 callersFunction_build
(directory: Path)
scripts/script_invocation_differential.py:1150
↓ 1 callersFunction_build_anthropic_output_config
Build Anthropic Messages API output_config payload.
src/skillevaluator/inference/client.py:182
↓ 1 callersFunction_build_anthropic_output_config
(schema)
src/skillevaluator/tier3/harbor/templates/eval.py:1425
↓ 1 callersFunction_build_baseline_only_conclusions
Insights for a baseline-only run: the calibration result, never a lift claim.
src/skillevaluator/evaluation/tier3_report.py:4776
↓ 1 callersFunction_build_catalog
(workspace: Path, selected: list[Path], path: Path)
ci/run_tiers.py:408
↓ 1 callersFunction_build_dimensions
( with_scores: dict[str, Any], without_scores: dict[str, Any], precomputed_with: dict[str, Any],
src/skillevaluator/evaluation/tier3_report.py:2720
↓ 1 callersFunction_build_evaluators
( metrics: list[str], with_scores: dict[str, Any], without_scores: dict[str, Any], lift_data:
src/skillevaluator/evaluation/tier3_report.py:2664
↓ 1 callersFunction_build_file_manifest_fallback
(source: Path, allowed_root: Path)
src/skillevaluator/tier3/harbor/secure_copy.py:1595
↓ 1 callersFunction_build_file_manifest_posix
(source: Path, allowed_root: Path)
src/skillevaluator/tier3/harbor/secure_copy.py:1569
← previousnext →5,601–5,700 of 24,741, ranked by callers