A class for reordering and batching elements of an array. This class allows for sorting an array based on a provided sorting function, group

TaskManager indexes all tasks from the default `lm_eval/tasks/` and an optional directory if provided.

Data collator used for language modeling. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The

FilterEnsemble creates a pipeline applying multiple filters. Its intended usage is to stack multiple post-processing steps in order. `tas

The layer implementaion for [MoM: Linear Sequence Modeling with Mixture-of-Memories](https://arxiv.org/abs/2502.13685).

The layer implementaion for [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://arxiv.org/abs/2412.06464). # noqa Similar to M

r""" The layer implementaion for [Gated Linear Attention Transformers with Hardware-Efficient Training](https://arxiv.org/abs/2312.06635). # noqa

Criteria to stop on the specified multi-token sequence.

ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts https://arxiv.org/abs/1712.07040

NeuronModelForCausalLM with `stopping_criteria` in `generate`

A filter which evaluates

Filter classes operate on a per-task level. They take all model outputs (`instance.resps` for all `task.instances`) across all instances

Efficient Attentions for Long Document Summarization https://arxiv.org/abs/2104.02112 Note: The average length of the reference summaries is

takes an array `arr` and function `fn` and returns a dictionary with keys fn(ob) for each ob in `arr` and with values `self.arr[key]` a list

An abstracted Huggingface model class. Enables usage with both models of `transformers.AutoModelForCausalLM` and `transformers.AutoModelForSe