Allocates a batch of tokens on the heap that can hold a maximum of n_tokens Each token can be assigned up to n_seq_max sequence ids The batch

Apply chat template. Args: tmpl: Template to use. If None, uses model's default chat: Array of chat messages n_msg: Numbe

Get list of built-in chat templates. Args: output: Output buffer to store template names. len: Length of the output buffer.

Get default parameters for llama_context

Copies the state to the specified destination address (DEPRECATED)

Process a batch of tokens. 0 - success 1 - could not find a KV slot for the batch (try reducing the size of the batch or increase the context)

Convert the provided tokens into text (inverse of llama_tokenize()). Args: vocab: The vocabulary to use for tokenization. tokens:

Process a batch of tokens using the encoder. 0 - success < 0 - error

Get the flash attention type name.

Frees all allocated memory

Get the context linked through llama_context_params.ctx_other.

Get the embeddings for the input shape: [n_embd] (1-dimensional)

Get the embeddings for the ith sequence llama_get_embeddings(ctx) + i*n_embd

Get the nextn embeddings from the last evaluation.

Get the nextn embeddings for the ith output row from the last evaluation.

Get the embeddings for a sequence id Returns NULL if pooling_type is LLAMA_POOLING_TYPE_NONE shape: [n_embd] (1-dimensional)

Token logits obtained from the last call to llama_decode() The logits for which llama_batch.logits[i] != 0 are stored contiguously in the orde

Logits for the ith token. Equivalent to: llama_get_logits(ctx) + i*n_vocab

Get the memory for the context

Get the backend sampled candidate count for the ith token.

Get the backend sampled candidates for the ith token.

Get the backend sampled logit count for the ith token.

Get the backend sampled logits for the ith token.

Get the backend sampled probability count for the ith token.

Get the backend sampled probabilities for the ith token.

Get the backend sampled token for the ith token.

Returns the size in bytes of the state (DEPRECATED)

Get the current logging callback and user data.

Set callback for all future logging events. If this is not called, or NULL is supplied, everything is output on stderr.

Get the maximum number of tensor buffer type overrides.

Check if the memory supports shifting

Clear the memory contents If data == true, the data buffers will also be cleared together with the metadata

Adds relative position "delta" to all tokens that belong to the specified sequence and have positions in [p0, p1) p0 < 0 : [0, p1] p1 < 0 : [

Copy all tokens that belong to the specified sequence to another sequence p0 < 0 : [0, p1] p1 < 0 : [p0, inf)

Integer division of the positions by factor of `d > 1` p0 < 0 : [0, p1] p1 < 0 : [p0, inf)

Removes all tokens that do not belong to the specified sequence

Returns the largest position present in the memory for the specified sequence Return -1 if the sequence is empty

Returns the smallest position present in the memory for the specified sequence This is typically non-zero only for SWA caches Return -1 if the

Removes all tokens that belong to the specified sequence and have positions in [p0, p1) Returns false if a partial sequence cannot be removed. Re

Get the default chat template. Returns None if not available If name is None, returns the default chat template

Returns label of classifier output by index. Returns None if no label provided

For encoder-decoder models, this function returns id of the token that must be provided to the decoder to start generating output sequence. For ot

Get default parameters for llama_model

Get a string describing the model type

Returns true if the model contains a decoder that requires llama_decode() call

Returns true if the model contains an encoder that requires llama_encode() call

Initialize a model from user-provided metadata and tensor data.

Returns true if the model is diffusion-based (like LLaDA, Dream, etc.)

Returns true if the model is hybrid (like Jamba, Granite, etc.)

Returns true if the model is recurrent (like Mamba, RWKV, etc.)

Load the model from a file If the file is split into multiple parts, the file name must follow this pattern: <name>-%05d-of-%05d.gguf If the

Load a model from an open FILE pointer.

Load the model from multiple splits (support custom naming scheme) The paths must be in the correct order

Get the number of metadata key/value pairs

Get metadata key name by index

Get sampling metadata key name. Returns None if the key is invalid.

Get metadata value as a string by key name

Get metadata value as a string by index

Returns the number of classifier outputs (only valid for classifier models)

Get the model input embedding size.

Get the model output embedding size.

Returns the total number of parameters in the model

Returns 0 on success

Get default parameters for llama_model_quantize

Save the model to a file

Returns the total size of all the tensors in the model in bytes

Get the context size per sequence.

Get the number of threads used for generation of a single token