Make causal mask used for bi-directional self-attention.

Make causal mask used for bi-directional self-attention.

Make causal mask used for bi-directional self-attention.

Validate the `rope_scaling` configuration.

Validate the `rope_scaling` configuration.

Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function i

Adds sentinel tokens and padding token (if missing). Expands the tokenizer vocabulary to include sentinel tokens used in mixture-of-denoiser

Check if the given string a number.

How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom b

Compute the value. When current unit is larger than previous unit, current unit * all previous units will be used as all previous uni

Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be loaded with ``torch.load(file)`` + ``load_stat

一百八 to 一百八十 一亿一千三百万 to 一亿 一千万 三百万

Disable the redundant torch default initialization to accelerate model creation.

Disable the redundant torch default initialization to accelerate model creation.

Disable the redundant torch default initialization to accelerate model creation.