Evaluate this crew through CrewAI AMP, or compare models on the project's deployment (--models). It grades the project's last traced run. When there is none, or the project changed since it, it runs the crew first and grades that run: the run is traced and its trace sent to CrewAI A
(
ctx: click.Context,
run_id: str | None,
models: str | None,
deployment_id: str | None,
no_open: bool,
)
| 817 | ) |
| 818 | @click.pass_context |
| 819 | def eval_command( |
| 820 | ctx: click.Context, |
| 821 | run_id: str | None, |
| 822 | models: str | None, |
| 823 | deployment_id: str | None, |
| 824 | no_open: bool, |
| 825 | ) -> None: |
| 826 | """Evaluate this crew through CrewAI AMP, or compare models on the project's |
| 827 | deployment (--models). |
| 828 | |
| 829 | It grades the project's last traced run. When there is none, or the project |
| 830 | changed since it, it runs the crew first and grades that run: the run is |
| 831 | traced and its trace sent to CrewAI AMP, which grades it. The first time |
| 832 | tracing is turned on for the project, a person at a terminal is asked |
| 833 | first; with no terminal (a coding agent, a script) nothing is asked and the |
| 834 | command says it turned tracing on, so `crewai eval` alone takes an agent |
| 835 | from a change to its verdict. A project that turned tracing off is asked |
| 836 | at a terminal, and told how to turn it on without one. |
| 837 | |
| 838 | Run by a script or coding agent (no terminal), it prints a markdown brief |
| 839 | after the verdict — what failed and the change to make — so an agent can act |
| 840 | on it directly. That is when AMP provides the brief: otherwise its link, when |
| 841 | AMP sends only that, else nothing more. |
| 842 | |
| 843 | A run's evaluation exits 0 only when the goal gate PASSED, and 1 otherwise — a |
| 844 | failed gate, no verdict, or an evaluation that could not run — so a CI job can |
| 845 | gate on it. A comparison exits 0 when it finished, 1 when it failed or could |
| 846 | not start. |
| 847 | |
| 848 | `crewai eval export ID` downloads an evaluation's full data as files, for a |
| 849 | coding agent to dig into. |
| 850 | """ |
| 851 | if ctx.invoked_subcommand is not None: |
| 852 | # The subcommand does the work; an option meant for this body would |
| 853 | # otherwise be dropped without a word. |
| 854 | if ( |
| 855 | run_id is not None |
| 856 | or models is not None |
| 857 | or deployment_id is not None |
| 858 | or no_open |
| 859 | ): |
| 860 | raise click.UsageError( |
| 861 | f"`crewai eval {ctx.invoked_subcommand}` takes no --run, --models, " |
| 862 | "--deployment or --no-open; give those to `crewai eval` alone." |
| 863 | ) |
| 864 | return |
| 865 | if models is not None: |
| 866 | if run_id is not None: |
| 867 | raise click.UsageError( |
| 868 | "--run grades a run that already happened; --models runs the " |
| 869 | "deployment again. Give one of them." |
| 870 | ) |
| 871 | eval_models(models, deployment_id=deployment_id, open_report=not no_open) |
| 872 | return |
| 873 | if deployment_id is not None: |
| 874 | raise click.UsageError( |
| 875 | "--deployment names the deployment --models runs; add --models LIST." |
| 876 | ) |
nothing calls this directly
no test coverage detected