A class that keeps all conversation history.

Causal Language Modeling head. Reference: Improving Language Understanding by Generative Pre-Training. https://cdn.openai.com/res

Causal Language Modeling loss. Reference: Improving Language Understanding by Generative Pre-Training. https://cdn.openai.com/res

Fake file-like stream object that redirects writes to a logger instance.

Collate examples for supervised fine-tuning.

Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).

Token embedding with dropout.

Inference parameters passed to model to efficiently calculate and store context during inference. Reference: https://github.com/Dao-A

Dataset for supervised fine-tuning.

Multi-head attention layer.

Multi-Layer Perceptron. Reference: Attention Is All You Need. https://arxiv.org/pdf/1706.03762.pdf.

Parallel block. This block applies parallel mixer and MLP layers to the input (used in GPT-J and CodeGen).

Image to Patch Embedding

Phi model.

Vision Transformer with support for patch or hybrid CNN input stage

Cross-attention layer (compatible with PyTorch). Reference: https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/modules/mha

Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).

Abstract base class for all logit processors that can be applied during generation.

Phi configuration.

Phi for Causal Language Modeling.

Phi pre-trained model.

Rotary positional embedding (RoPE). Reference: RoFormer: Enhanced Transformer with Rotary Position Embedding. https://arxiv.org/p

Self-attention layer (compatible with PyTorch). Reference: https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/modules/mha.

Different separator style.