PanGUConfig config class which defines the model size

The PanguAlpha network consisting of two parts the backbone and the head Args: config(PanguAlphaConfig): the config of network In

Code Generation Model for Multilingual Program Synthesis.

Clip grads by global norm

PanguAlpha evaluation net Args: backbone: backbone network of PanguAlpha generate: enable generate mode Inputs: i

Monitor the loss in training. If the loss in NAN or INF terminating training.

PanguAlpha training loss for generation. Args: config(PanGUConfig) Inputs: input_ids: the tokenized inputs past:

Implements the Adam algorithm to fix the weight decay. It is a complete operator, not a combination of other ops. Note: When separat

CodeGeeX: A Multilingual Code Generation Model.

Dataset to concatenate multiple datasets. Purpose: useful to assemble different existing datasets, possibly large-scale datasets as the c

Monitor the ppl loss in evaluating. Note: If per_print_times is 0, do NOT print loss. Args: print_per_step (int): Print

PanguAlpha evaluation net Args: backbone: backbone network of PanguAlpha generate: enable generate mode Inputs: i

Calculate the global norm value of given tensors

Warmup-decay learning rate for PanguAlpha network.

Ppl metric

The PanguAlpha network consisting of two parts the backbone and the head Args: config(PanguAlphaConfig): the config of network In

Pipeline version of CodeGeeX.

r""" This class is almost same with the mindspore's AdamWeightDecay implements, the only difference is the optimizer's state will be a

Loader for IndexedDataset

MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the s

MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the st

MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the s

PanguAlpha training loss for generation. Args: config(PanGUConfig) Inputs: input_ids: the tokenized inputs past:

MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the s

Anneals the learning rate.

CodeGeeX: A Multilingual Code Generation Model.

CodeGeeX: A Multilingual Code Generation Model.

Tracker for the cuda RNG states. Using the `add` method, a cuda rng state is initialized based on the input `seed` and is assigned to `name`.

Language model embeddings. Arguments: hidden_size: hidden size vocab_size: vocabulary size max_sequence_length: maximum s

Language model embeddings. Arguments: hidden_size: hidden size vocab_size: vocabulary size max_sequence_length: maximum s

Language model embeddings. Arguments: hidden_size: hidden size vocab_size: vocabulary size max_sequence_length: maximum si

Language model embeddings. Arguments: hidden_size: hidden size vocab_size: vocabulary size max_sequence_length: maximum s

r"""Embedding layer of the PanGUAlpha Model

r"""Embedding layer of the PanGUAlpha Model

Float16 optimizer for fp16 and bf16 data types. Arguments: optimizer: base optimizer such as Adam or SGD clip_grad: clip gradeint

GPT-2 BPE tokenizer. Peculiarities: - Byte-level BPE

Wrapper for Hugging Face tokenizer.

r""" Jieba Tokenizer

PanguAlpha evaluation net Args: backbone: backbone network of PanguAlpha generate: enable generate mode Inputs: i

Contiguous memory buffer. Allocate a contiguous memory of type `dtype` and size `numel`. It is used to reduce memory fragmentation. Usage

Head to get the logits of each token in the vocab Args: config(): the config of network Inputs: state: the output of the

Head to get the logits of each token in the vocab Args: config(): the config of network Inputs: state: the output of the

Encapsulation class of PanguAlpha network training. Append an optimizer to the training network after that the construct function can be

Encapsulation class of PanguAlpha network training. Append an optimizer to the training network after that the construct function can be

Encapsulation class of PanguAlpha network training. Append an optimizer to the training network after that the construct function can be

r"""The base backbone of the PanGuAlpha model

r"""The base backbone of the PanGuAlpha model

Parallel self-attention layer abstract class. Self-attention layer takes input with size [b, s, h] and returns output of the same size.

A single top query layer. Top query layer takes input with size [b, s, h] and returns an output of the same size.

Parallel top query self-attention layer abstract class. Self-attention layer takes input with size [b, s, h] and returns output of the same s

Transformer class.

A single transformer layer. Transformore layer takes input with size [b, s, h] and returns an output of the same size.

Language model embeddings. Arguments: hidden_size: hidden size vocab_size: vocabulary size max_sequence_length: maximum s

Language model embeddings. Arguments: hidden_size: hidden size vocab_size: vocabulary size max_sequence_length: maximum s

Language model embeddings. Arguments: hidden_size: hidden size vocab_size: vocabulary size max_sequence_length: maximum si

Language model embeddings. Arguments: hidden_size: hidden size vocab_size: vocabulary size max_sequence_length: maximum s

r"""Query Layer at the final layer.

r"""Query Layer at the final layer.

Dataset wrapper to randomly mapping indices to original order. Will also enlarge the length

self-attention layer abstract class. Self-attention layer takes input with size [b, s, h] and returns output of the same size.

self-attention layer abstract class. Self-attention layer takes input with size [b, s, h] and returns output of the same size.

self-attention layer abstract class. Self-attention layer takes input with size [b, s, h] and returns output of the same size.

Group of timers.

A single top query layer. Top query layer takes input with size [b, s, h] and returns an output of the same size.

A single top query layer. Top query layer takes input with size [b, s, h] and returns an output of the same size.

A single top query layer. Top query layer takes input with size [b, s, h] and returns an output of the same size.

Top query self-attention layer abstract class. Self-attention layer takes input with size [b, s, h] and returns output of the same size.