Class TurbomindEngineConfig

lmdeploy/messages.py:206–330

TurboMind Engine config. Args: dtype: data type for model weights and activations. It can be one of the following values, ['auto', 'float16', 'bfloat16'] The `auto` option will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 m

@pydantic_dataclass
class TurbomindEngineConfig:
    """TurboMind Engine config.

    Args:
        dtype: data type for model weights and activations. It can be
            one of the following values, ['auto', 'float16', 'bfloat16']
            The `auto` option will use FP16 precision for FP32 and FP16
            models, and BF16 precision for BF16 models.
        model_format: the layout of the deployed model. It can be one
            of the following values [hf, awq, gptq, compressed-tensors,
            fp8, mxfp4]. `hf` means a Hugging Face model (.bin,
            .safetensors), `awq` and `gptq` mean grouped 4-bit
            weight-only checkpoints, `compressed-tensors` means
            pack-quantized grouped int4 checkpoints and is usually
            auto-detected from the input model config, `fp8` means
            blocked fp8 checkpoints, and `mxfp4` means MXFP4 expert
            weights. If it is not specified, i.e. None, it will be
            extracted from the input model
        tp: the number of GPU cards used in tensor parallelism,
            default to 1
        session_len: the max session length of a sequence, default to
            None
        max_batch_size: the max batch size during inference. If it is
            not specified, the engine will automatically set it according to
            the device
        cache_max_entry_count: the percentage of gpu memory occupied
            by the k/v cache.
            For versions of lmdeploy between `v0.2.0` and `v0.2.1`, it
            defaults to 0.5, depicting the percentage of TOTAL GPU memory to
            be allocated to the k/v cache.
            For lmdeploy versions greater than `v0.2.1`, it defaults to 0.8,
            signifying the percentage of FREE GPU memory to be reserved for
            the k/v cache.
            When it's an integer > 0, it represents the total number of k/v
            blocks.
        cache_chunk_size: The policy to apply for KV block from
            the block manager, default to -1.
        cache_block_seq_len: the length of the token sequence in
            a k/v block, default to 64
        enable_prefix_caching: enable cache prompts for block reuse,
            default to False
        quant_policy: default to 0. For TurboMind, when k/v is quantized
            into int4 or int8, set it to 4 or 8, respectively
        rope_scaling_factor: scaling factor used for dynamic ntk,
            default to 0. TurboMind follows the implementation of transformer
            LlamaAttention
        use_logn_attn: whether or not to use log attn: default to False
        download_dir: Directory to download and load the weights,
            default to the default cache directory of huggingface.
        revision: The specific model version to use. It can be a branch
            name, a tag name, or a commit id. If unspecified, will use the
            default version.
        max_prefill_token_num: the number of tokens each iteration during
            prefill, default to 8192
        num_tokens_per_iter: the number of tokens processed in each
            forward pass. Working with `max_prefill_iters` enables the
            "Dynamic SplitFuse"-like scheduling
        max_prefill_iters: the max number of forward pass during prefill