reference implementation for deterministic creation of model files

if there's a pending marker from the entropy stream, return that otherwise, fetch from the stream and get a marker. if there's no marker, return 0xff,

Decode processed id or token sequences. Args: out_type: output type. str, bytes or 'serialized_proto' or 'immutable_proto' (Default

Encode text input to segmented ids or tokens. Args: input: input string. accepsts list of string. out_type: output type.

Returns true if `id` is user defined symbol.

size is in number of elements

Helper for decoding scales and mins of Q4_K and Q5_K block formats

shuffles to pick the required scales in dot products

ne2 is passed explicitly to help compiler optimize repeated calls

same as above, for sequential indices starting at node_idx

Returns true if the subgraph formed by {node_idxs} can be fused checks whethers all nodes which are not part of outputs can be elided by checking if t