Configuration for the KV cache. Attributes: block_size (int): Size of a cache block in number of tokens. gpu_memory_utilization (float): Fraction of GPU memory to use for model execution. cache_dtype (str): Data type for kv cache storage. Default is 'bfloat16'.
| 1378 | |
| 1379 | |
| 1380 | class CacheConfig: |
| 1381 | """ |
| 1382 | Configuration for the KV cache. |
| 1383 | |
| 1384 | Attributes: |
| 1385 | block_size (int): Size of a cache block in number of tokens. |
| 1386 | gpu_memory_utilization (float): Fraction of GPU memory to use for model execution. |
| 1387 | cache_dtype (str): Data type for kv cache storage. Default is 'bfloat16'. |
| 1388 | num_gpu_blocks_override (Optional[int]): Number of GPU blocks to use. |
| 1389 | Overrides profiled num_gpu_blocks if provided. |
| 1390 | kv_cache_ratio (float): Ratio for calculating the maximum block number. |
| 1391 | enc_dec_block_num (int): Number of encoder-decoder blocks. |
| 1392 | prealloc_dec_block_slot_num_threshold (int): Number of token slot threadshold to allocate next blocks for decoding. |
| 1393 | enable_prefix_caching (bool): Flag to enable prefix caching. |
| 1394 | enable_output_caching (bool): Flag to enable kv cache output tokens, only works in V1 scheduler. |
| 1395 | """ |
| 1396 | |
| 1397 | def __init__(self, args): |
| 1398 | """ |
| 1399 | Initialize the CacheConfig class. |
| 1400 | |
| 1401 | Args: |
| 1402 | block_size (int): Size of a cache block in number of tokens. |
| 1403 | gpu_memory_utilization (float): Fraction of GPU memory to use. |
| 1404 | cache_dtype (str): Data type for cache storage. Default is 'bfloat16'. |
| 1405 | num_gpu_blocks_override (Optional[int]): Override for number of GPU blocks. |
| 1406 | num_cpu_blocks (Optional[int]): Number of CPU blocks. |
| 1407 | kv_cache_ratio (float): Ratio for max block calculation. |
| 1408 | enc_dec_block_num (int): Number of encoder-decoder blocks. |
| 1409 | prealloc_dec_block_slot_num_threshold (int): Number of token slot threshold to allocate next blocks for decoding, used when ENABLE_V1_KVCACHE_SCHEDULER=1. |
| 1410 | enable_prefix_caching (bool): Enable prefix caching. |
| 1411 | max_encoder_cache(int): Maximum number of tokens in the encoder cache. |
| 1412 | max_processor_cache(int): Maximum number of bytes in the processor cache. |
| 1413 | """ |
| 1414 | self.block_size = 64 |
| 1415 | self.gpu_memory_utilization = 0.9 |
| 1416 | self.num_gpu_blocks_override = None |
| 1417 | if envs.ENABLE_V1_KVCACHE_SCHEDULER: |
| 1418 | self.kv_cache_ratio = 1.0 |
| 1419 | else: |
| 1420 | self.kv_cache_ratio = 0.75 |
| 1421 | self.enc_dec_block_num = envs.FD_ENC_DEC_BLOCK_NUM |
| 1422 | self.prealloc_dec_block_slot_num_threshold = 12 |
| 1423 | self.cache_dtype = "bfloat16" |
| 1424 | self.model_cfg = None |
| 1425 | self.enable_chunked_prefill = False |
| 1426 | self.rdma_comm_ports = None |
| 1427 | self.local_rdma_comm_ports = None |
| 1428 | self.cache_transfer_protocol = None |
| 1429 | self.pd_comm_port = None |
| 1430 | self.local_pd_comm_port = None |
| 1431 | self.enable_prefix_caching = False |
| 1432 | self.enable_ssd_cache = False |
| 1433 | self.cache_queue_port = None |
| 1434 | self.local_cache_queue_port = None |
| 1435 | self.swap_space = None |
| 1436 | self.max_encoder_cache = None |
| 1437 | self.max_processor_cache = None |
no outgoing calls