Exception raised when there is insufficient space for allocating a block.

A version of AttentionInfo that stores the data as Python lists rather than tensors, making it more efficient for passing over multiproce

For flags that we define that aren't in the hf config.

RMSNorm for head dimension (used in q_norm and k_norm)

Convention: tokens are in order of prefill, hydragen, decode. Hydragen = the shared prefix part of hydragen (only one layer deep for now)

Models will make the size of the embedding/unembedding matrices be a clean number for efficiency reasons. However, the actual number of t

Wraps the queues to interact with a manager and one or more model processes.

NOTE: this keeps track of the state ACCORDING TO THE SCHEDULER - may not actually reflect the true state of requests. E.g., a "finished" sequ

Request model for creating a batch

Multi-headed attention from 'Attention Is All You Need' paper

Request model for synchronous batch completions