The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added betwee

A single transformer layer. Transformer layer takes input with size [s, b, h] and returns an output of the same size.

MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the s

The torch.nn model to encode the prefix Input shape: (batch-size, prefix-length) Output shape: (batch-size, prefix-length, 2*layers*hidde

Parallel self-attention layer abstract class. Self-attention layer takes input with size [s, b, h] and returns output of the same size.

The base class of each model worker.

r""" This is the configuration class to store the configuration of a [`~ChatGLMModel`]. It is used to instantiate an ChatGLM model according t

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Language model embeddings.

Transformer class.