MCPcopy Create free account
hub / github.com/crewAIInc/crewAI / eval_command

Function eval_command

lib/cli/src/crewai_cli/cli.py:819–877  ·  view source on GitHub ↗

Evaluate this crew through CrewAI AMP, or compare models on the project's deployment (--models). It grades the project's last traced run. When there is none, or the project changed since it, it runs the crew first and grades that run: the run is traced and its trace sent to CrewAI A

(
    ctx: click.Context,
    run_id: str | None,
    models: str | None,
    deployment_id: str | None,
    no_open: bool,
)

Source from the content-addressed store, hash-verified

817)
818@click.pass_context
819def eval_command(
820 ctx: click.Context,
821 run_id: str | None,
822 models: str | None,
823 deployment_id: str | None,
824 no_open: bool,
825) -> None:
826 """Evaluate this crew through CrewAI AMP, or compare models on the project's
827 deployment (--models).
828
829 It grades the project's last traced run. When there is none, or the project
830 changed since it, it runs the crew first and grades that run: the run is
831 traced and its trace sent to CrewAI AMP, which grades it. The first time
832 tracing is turned on for the project, a person at a terminal is asked
833 first; with no terminal (a coding agent, a script) nothing is asked and the
834 command says it turned tracing on, so `crewai eval` alone takes an agent
835 from a change to its verdict. A project that turned tracing off is asked
836 at a terminal, and told how to turn it on without one.
837
838 Run by a script or coding agent (no terminal), it prints a markdown brief
839 after the verdict — what failed and the change to make — so an agent can act
840 on it directly. That is when AMP provides the brief: otherwise its link, when
841 AMP sends only that, else nothing more.
842
843 A run's evaluation exits 0 only when the goal gate PASSED, and 1 otherwise — a
844 failed gate, no verdict, or an evaluation that could not run — so a CI job can
845 gate on it. A comparison exits 0 when it finished, 1 when it failed or could
846 not start.
847
848 `crewai eval export ID` downloads an evaluation's full data as files, for a
849 coding agent to dig into.
850 """
851 if ctx.invoked_subcommand is not None:
852 # The subcommand does the work; an option meant for this body would
853 # otherwise be dropped without a word.
854 if (
855 run_id is not None
856 or models is not None
857 or deployment_id is not None
858 or no_open
859 ):
860 raise click.UsageError(
861 f"`crewai eval {ctx.invoked_subcommand}` takes no --run, --models, "
862 "--deployment or --no-open; give those to `crewai eval` alone."
863 )
864 return
865 if models is not None:
866 if run_id is not None:
867 raise click.UsageError(
868 "--run grades a run that already happened; --models runs the "
869 "deployment again. Give one of them."
870 )
871 eval_models(models, deployment_id=deployment_id, open_report=not no_open)
872 return
873 if deployment_id is not None:
874 raise click.UsageError(
875 "--deployment names the deployment --models runs; add --models LIST."
876 )

Callers

nothing calls this directly

Calls 2

eval_modelsFunction · 0.70
eval_crewFunction · 0.70

Tested by

no test coverage detected