Function evaluate_item

evaluate_score.py:1–91

(i, query, answer, benchmark, tokenizer)

def evaluate_item(i, query, answer, benchmark, tokenizer):
    DEEPSEEK_MODEL = "deepseek-v3-250324"
    DEEPSEEK_URL = "xxxx"
    client = OpenAI(api_key='xxxx',base_url=DEEPSEEK_URL)
    sys_prompt="""
        ---Role---
        You are an expert tasked with evaluating answer to the question based on four criteria: **Comprehensiveness**, **Diversity**, **Directness** and **Empowerment**.
        """

    prompt = f"""
        Your task is to evaluate the following answer based on four criteria. For each criterion, assign a score from 1 to 10 , following the detailed scoring rubric.

        When explaining your score, you must refer directly to specific parts of the answer to justify your reasoning. Avoid general statements — your explanation must be grounded in the content provided.

        - **Comprehensiveness**: 
        How much detail does the answer provide to cover all aspects and details of the question?
        
        - **Diversity**:
        How varied and rich is the answer in providing different perspectives and insights on the question?
      
        - **Empowerment**: 
        How well does the answer help the reader understand and make informed judgments about the topic?

        - **Overall Quality**:
        Provide an overall evaluation based on the combined performance across all four dimensions. Consider both content quality and answer usefulness to the question.
        
        Scoring Guidelines:

        "1-2": "Low score description: Clearly deficient in this aspect, with significant issues.", 
        "3-4": "Below average score description: Lacking in several important areas, with noticeable problems.", 
        "5-6": "Average score description: Adequate but not exemplary, meets basic expectations with some minor issues.",
        "7-8": "Above average score description: Generally strong but with minor shortcomings.", 
        "9-10": "High score description: Outstanding in this aspect, with no noticeable issues."

        Here is the question:
        {query}

        Here are the  answer:

        {answer}

        Evaluate the answer using the  criteria listed above and provide detailed explanations for each criterion with reference to the text. 
        Output your evaluation in the following JSON format:

        {{
            "Comprehensiveness": {{
                "score": "[1-10]",
                "Explanation": "[Provide explanation here]"
            }},
            "Empowerment": {{
                "score": "[1-10]",
                "Explanation": "[Provide explanation here]"
            }},
            "Diversity": {{
                "score": "[1-10]",
                "Explanation": "[Provide explanation here]"
            }},
            "Overall Quality": {{
                "score": "[1-10]",
                "Explanation": "[Summarize why this answer is the overall winner based on the three criteria]"