Way faster than text.strip(strip_characters) since strip_characters is a set instead of a str, and it contains a lot of elements (all

Under zero.Init returns a context manager that will gather the sharded param, otherwise returns an empty list If `modify` is `True`, gather

hist_data is a list of statistics lists (works up to 3, if more is needed, add some colors and annotation colors)

delta since last delta call

Compute the score of a generative task using a llm as a judge. The generative task can be multiturn with 2 turns max, in that case, w

Process the input text and yield results as they're generated

converts seconds delta as generated by 2 snapshots of time.time() into hh:mm:ss

converts seconds delta as generated by 2 snapshots of time.time() into ss.msec

We artificially separate `tokenizer_add_tokens` and `tokenizer_add_special_tokens` is a dictionary whose keys only takes into account special tok

Get words from a text. Non reversible since the text is split on multiple characters, words are stripped of special characters and cha

Get words from a text. Non reversible since the text is split on multiple characters, words are stripped of special characters and cha

Helper method to create chat completions with standard parameters

Searches for the first occurrence of 'pattern' in 'sequence' starting at offset 'start'. Returns the index of that occurrence, or -1 if n

Searches for the first occurrence of 'pattern' in 'sequence' starting at offset 'start'. Returns the index of that occurrence, or -1 if n

Ceil 'number' to the nearest integer multiple of 'factor'.

`split` is "validation" or "test".

compact formatting of the logs with pre-specified formatter for each log entry, plus a catch-all if new log entries are added but for

Args details: `images_to_pack` -> # Each tensor is of size (3, im_height, im_width) `output_input_ids` -> # Each tensor is of size (max_s

This function returns the end dictionary at the exit of the dataloader. Mostly batchify things and pad accordingly.

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he