Class PytorchEngineConfig

lmdeploy/messages.py:334–491

PyTorch Engine Config. Args: dtype: data type for model weights and activations. It can be one of the following values, ['auto', 'float16', 'bfloat16'] The `auto` option will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 mod

@dataclass
class PytorchEngineConfig:
    """PyTorch Engine Config.

    Args:
        dtype: data type for model weights and activations. It can be
            one of the following values, ['auto', 'float16', 'bfloat16']
            The `auto` option will use FP16 precision for FP32 and FP16
            models, and BF16 precision for BF16 models.
        tp: Tensor Parallelism. default 1.
        dp: Data Parallelism. default 1.
        dp_rank: rank of dp.
        ep: Expert Parallelism. default 1.
        session_len: Max session length. Default None.
        max_batch_size: Max batch size. If it is not specified,
            the engine will automatically set it according to the device
        attn_tp_size: tp size for attention, only works for dp>1
        mlp_tp_size: tp size for mlp, only works for dp>1
        moe_tp_size: tp size for moe, only works for dp>1
        cache_max_entry_count: the percentage of gpu memory occupied
            by the k/v cache. For lmdeploy versions greater than `v0.2.1`,
            it defaults to 0.8, signifying the percentage of FREE GPU memory
            to be reserved for the k/v cache
        prefill_interval: Interval to perform prefill,
            Default 16.
        block_size: paging cache block size, default 64.
        num_cpu_blocks: Num cpu blocks. If num is 0, cache
            would be allocate according to current environment.
        num_gpu_blocks: Num gpu blocks. If num is 0, cache
            would be allocate according to current environment.
        adapters: The path configs to lora adapters.
        max_prefill_token_num: tokens per iteration.
        thread_safe: thread safe engine instance.
        enable_prefix_caching: Enable token match and sharing caches.
        device_type: The inference device type, options ['cuda']
        eager_mode: Enable "eager" mode or not
        custom_module_map: nn module map customized by users. Once
            provided, the original nn modules of the model will be
            substituted by the mapping ones
        download_dir: Directory to download and load the weights,
            default to the default cache directory of huggingface.
        revision: The specific model version to use.
            It can be a branch name, a tag name, or a commit id.
            If unspecified, will use the default version.
        quant_policy: default to 0. When k/v is quantized into int4,
            int8, fp8, or fp8_e5m2, set it to 4, 8, 16, or 17,
            respectively
        distributed_executor_backend: backend of distributed backend,
            options: ['uni', 'mp', 'ray']
        empty_init: Whether to load the model weights, you should set
            it to True if you want to update weights after create the pipeline
        enable_microbatch: enable microbatch for specified model
        enable_eplb: enable eplb for specified model
        enable_metrics: enable metrics system
        role: role of engin, options: ['Hybrid', 'Prefill',
            'Decode']. Default to `EngineRole.Hybrid`.
        migration_backend: migration backend. options: ['DLSlime'].
            Default to `MigrationBackend.DLSlime`.
        enable_mp_engine: run engine in multi-process mode.