Tokenize each text into sentences, batch the sentences through Sonar, and regroup them per text. Returns a list of dicts, each dict has:

Saves a dictionary containing: - Model, optimizer, scheduler states - epoch, step_in_epoch (the iteration index in that epoch)

Return the METEOR score for a pair of strings.

Return BLEU and ROUGE-L_f scores for a pair of strings.

Run one forward step and return next sentence + its embedding.

Loads the saved state dicts and RNG states. Returns a dict: { "epoch": epoch, "step_in_epoch": step_in_epoch, "glob