Splits multiple sentences into words and flattens the result

Decodes a list of token IDs into a string. Args: t (List[int]): The list of token IDs to be decoded. Returns:

Build a Llama instance by initializing and loading a pre-trained model. Args: ckpt_dir (str): Path to the directory cont

Generate assistant responses for a list of conversational dialogs using the language generation model. Args: dialogs (Li

Initialize the RMSNorm normalization layer. Args: dim (int): The dimension of the input tensor. eps (float,

Forward pass through the RMSNorm layer. Args: x (torch.Tensor): The input tensor. Returns: torch.Te

Encodes a string into a list of token IDs. Args: s (str): The input string to be encoded. bos (bool): Whethe

Computes the LCS-based F-measure score Source: http://research.microsoft.com/en-us/um/people/cyl/download/papers/ rouge-working-note-v1.3.1.pdf

Extracts all n-grams upto a given maximum order from an input segment. Args: segment: text segment from which n-grams will be extracted. ma

Calculates word n-grams for multiple sentences.

Computes the length of the longest common subsequence (lcs) between two strings. The implementation below uses a DP programming algorithm and run

a list of lists of tokens

Generate text sequences based on provided prompts using the language generation model. Args: prompt_tokens (List[List[in

torch.repeat_interleave(x, dim=2, repeats=n_rep)

Computes ROUGE-N of two text collections of sentences. Sourece: http://research.microsoft.com/en-us/um/people/cyl/download/ papers/rouge-workin

Calcualtes n-grams. Args: n: which n-grams to calculate text: An array of tokens Returns: A set of n-grams

Returns the length of the Longest Common Subsequence between sequences x and y. Source: http://www.algorithmist.com/index.php/Longest_Common_Su

Apply the RMSNorm normalization to the input tensor. Args: x (torch.Tensor): The input tensor. Returns:

private recon calculation

Returns the Longest Subsequence between x and y. Source: http://www.algorithmist.com/index.php/Longest_Common_Subsequence Args: x: sequenc

Returns LCS_u(r_i, C) which is the LCS score of the union longest common subsequence between reference sentence ri and candidate summary C. For e

Apply rotary embeddings to input tensors using the given frequency tensor. This function applies rotary embeddings to the given query 'xq' a

Computes BLEU score of translated segments against one or more references. Args: reference_corpus: list of lists of references for each transla

Precompute the frequency tensor for complex exponentials (cis) with given dimensions. This function calculates a frequency tensor with compl

Reshape frequency tensor for broadcasting it with another tensor. This function reshapes the frequency tensor to have the same shape as the

Calculates average rouge scores for a list of hypotheses and references

Computes ROUGE-L (sentence level) of two text collections of sentences. http://research.microsoft.com/en-us/um/people/cyl/download/papers/ roug

both are a list of strings

Perform top-p (nucleus) sampling on a probability distribution. Args: probs (torch.Tensor): Probability distribution tensor.

Perform text completion for a list of prompts using the language generation model. Args: prompts (List[str]): List of te

Initializes the Tokenizer with a SentencePiece model. Args: model_path (str): The path to the SentencePiece model file.

Initialize the Attention module. Args: args (ModelArgs): Model configuration parameters. Attributes:

Initialize the FeedForward module. Args: dim (int): Input dimension. hidden_dim (int): Hidden dimension of t

Initialize a TransformerBlock. Args: layer_id (int): Identifier for the layer. args (ModelArgs): Model confi

Initialize a Transformer model. Args: params (ModelArgs): Model configuration parameters. Attributes:

Forward pass of the attention module. Args: x (torch.Tensor): Input tensor. start_pos (int): Starting positi

Perform a forward pass through the TransformerBlock. Args: x (torch.Tensor): Input tensor. start_pos (int):

Perform a forward pass through the Transformer model. Args: tokens (torch.Tensor): Input token indices. star

Entry point of the program for generating text using a pretrained model. Args: ckpt_dir (str): The directory containing checkpoint f

Entry point of the program for generating text using a pretrained model. Args: ckpt_dir (str): The directory containing checkpoint f