Mixed Dataset Sampler Core concept: - Within each GPU: samples in one batch must come from the same dataset - Across different GPUs:

An implementation of the PRO loss from "Large Language Model based Long-tail Query Rewriting in Taobao Search", adapted for Semantic Tex

Calculates KL divergence loss based on fair ranking (handles ties correctly), aligning better with Spearman's rank correlation. Tied sim

Sampler used when training on multiple datasets to ensure each batch only contains samples from one dataset, discarding any leftover samp

Arguments.

Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.