Encodes a string into a list of token IDs. Args: s (str): The input string to be encoded. bos (bool): W

Run a system command and ensure it succeeds.

Decodes a list of token IDs into a string. Args: t (List[int]): The list of token IDs to be decoded. Retu

Restore original configuration file

Parallel map, but with backpressure. If the caller doesn't call `next` fast enough, this will stop calling `func` at some point rather than l

Parallel map, but with backpressure. If the caller doesn't call `next` fast enough, this will stop calling `func` at some point rather than l

Create a temporary dataset with only the first N characters for quick testing.

Terminate the current phase.

Generate new configuration file with simplified format

Start a new phase, and terminate the current one, if one is ongoing.

Rebuild project

Run perplexity test on a single dataset.

stride * 8 * (tx // HALF_WARP_expr) + (tx % 8) * stride + 16 * ((tx % HALF_WARP_expr) // 8)

Splits the string `s` so that each substring contains no more than `max_consecutive_slice_len` consecutive whitespaces or consecuti

Backup current configuration file

Benchmark model Args: output_suffix: Output file suffix (lowercase format, e.g., q6_k) Returns: