compute pd_x, pd_gamma, pd_beta according to data, params and shape_x

Set autoresume start time.

Initialize torch.distributed and mpu.

Initialize manually allocated static memory.

Create the all-reduce hook for backprop.

Calculate the epoch needed for so many sample.

Parse entire arguments.

Print arguments.

Initialize ADLR autoresume.

Set random seed for reproducability.

Set tensorboard writer.

Initialize timers.

r""" Add checkpoint policy to callback.

r""" Add checkpoint policy to callback.

Add inference params

Add parameters about retrain.

Add training params

Reduce gradients across data parallel ranks.

Beam search. Note that this function does not support model parallel!

Initialize tokenizer.

Initialize tokenizer.

Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you

Calculate l2 norm of parameters

Check for autoresume signal and exit if it is received.

Ensure fixed arguments for a model are the same for the input arguments and the one retrieved from checkpoint.

Checkpoint exception checkpoints size. Args: ckpt_file_list: exception checkpoints Returns: result of exception checkpoints size

cleaning wikitext dataset

Clear the internal evaluation result.

Clips gradient norm of an iterable of parameters whose gradients are in fp32. This is adapted from torch.nn.utils.clip_grad.clip_grad_norm

Converts a sequence of tokens into ids using the vocab.

Create dataset Inputs: batch_size: batch size data_path: path of your MindRecord files device_num: total device numbe

Decode the whitespace-encoded strings produced by encode_whitespace. >>> text = 'a\\n b\\n c' >>> s, l = 10, 10 >>> text == decode_whi

Decode the whitespace-encoded strings produced by encode_whitespace. >>> text = 'a\\n b\\n c' >>> s, l = 10, 10 >>> text == decode_wh

Encode whitespaces to extra tokens. >>> encode_whitespaces('a\\n b\\n c', 10, 10) 'a\\n<|extratoken_10|>b\\n<|extratoken_11|>c'

Encode whitespaces to extra tokens. >>> encode_whitespaces('a\\n b\\n c', 10, 10) 'a\\n<|extratoken_10|>b\\n<|extratoken_11|>c'

Calculates 1 - comb(n - c, k) / comb(n, k).

Helper function to evaluate and dump results on screen.

Export mindir model

Export mindir model

Export mindir model

Export mindir model

Export mindir model

Export mindir model

Export mindir model

Fix up query/key/value matrix ordering if checkpoint version is smaller than 2.0

Convert fp16/bf16 `val` to fp32

Convert fp32 `val` to fp16/bf16

Opposite of above function, gather values from model parallel ranks.

OpenAI's gelu implementation.

生成代码。 @param prompt 待补全的代码

Generate batch from context tokens.

Generate a batch

Generate batch from context tokens.

Generate batch from context tokens.

Provide extra arguments required for merging.

Return my rank for the data parallel group.

Get exception checkpoint based on restore ranks Args: args_param: training model parameters Returns: exception checkpoint list

Generate position_id and attention_mask according to input_ids considering eod reset Inputs: input_ids: the input token ids e

Generate position_id and attention_mask according to input_ids considering eod reset Inputs: input_ids: the input token ids l

Build language model and return along with the key to save.

Build the learning rate scheduler.

The output should be a cuda tensor of size 1.

Build masks and position id for left to right model.

Build masks and position id for left to right model.

Build masks and position id for left to right model.