Method count_bytes

lm-eval-harness/lm_eval/api/task.py:545–547

Used for byte-level perplexity metrics in rolling loglikelihood

(cls, doc)

    @classmethod
    def count_bytes(cls, doc):
        """Used for byte-level perplexity metrics in rolling loglikelihood"""
        return len(doc.encode("utf-8"))

    @classmethod
    def count_words(cls, doc):