MCPcopy Create free account
hub / github.com/LMCache/LMCache / _normalize_paged_layers

Function _normalize_paged_layers

lmcache/python_ops_fallback.py:906–1031  ·  view source on GitHub ↗

Normalize paged buffer input based on GPU KV format. Accepts either tensor-form inputs (list / Tensor) or a 1-D pointer tensor (int64 / uint64). When a pointer tensor is provided *shape_desc*, *device*, and *dtype* must be supplied so the tensors can be reconstructed via :func:`_te

(
    paged_buffer_ptrs_tensor: "torch.Tensor | list",
    engine_kv_format: EngineKVFormat,
    shape_desc: "PageBufferShapeDesc | None" = None,
    device: "torch.device | str | None" = None,
    dtype: "torch.dtype | None" = None,
)

Source from the content-addressed store, hash-verified

904
905
906def _normalize_paged_layers(
907 paged_buffer_ptrs_tensor: "torch.Tensor | list",
908 engine_kv_format: EngineKVFormat,
909 shape_desc: "PageBufferShapeDesc | None" = None,
910 device: "torch.device | str | None" = None,
911 dtype: "torch.dtype | None" = None,
912) -> "torch.Tensor | list[torch.Tensor] | list[list[torch.Tensor]]":
913 """Normalize paged buffer input based on GPU KV format.
914
915 Accepts either tensor-form inputs (list / Tensor) or a 1-D pointer tensor
916 (int64 / uint64). When a pointer tensor is provided *shape_desc*, *device*,
917 and *dtype* must be supplied so the tensors can be reconstructed via
918 :func:`_tensor_from_ptr`.
919
920 Returns:
921 - Single ``torch.Tensor`` for cross-layer formats.
922 - ``list[list[torch.Tensor]]`` (2 x NL) for SGLang MHA formats.
923 - ``list[torch.Tensor]`` (per-layer) for all other formats.
924 """
925 if _is_cross_layer_format(engine_kv_format):
926 if isinstance(paged_buffer_ptrs_tensor, torch.Tensor):
927 if _is_ptr_tensor(paged_buffer_ptrs_tensor):
928 # 1-D pointer tensor with a single entry → reconstruct full tensor.
929 if shape_desc is None or device is None or dtype is None:
930 raise ValueError(
931 "_normalize_paged_layers: shape_desc, device, and dtype are "
932 "required when paged_buffer_ptrs_tensor is a pointer tensor"
933 )
934 nb = int(shape_desc.nb)
935 nl = int(shape_desc.nl)
936 bs = int(shape_desc.bs)
937 nh = int(shape_desc.nh)
938 hs = int(shape_desc.hs)
939 if int(engine_kv_format) == int(EngineKVFormat.NB_NL_TWO_NH_BS_HS):
940 shape: tuple[int, ...] = (nb, nl, 2, nh, bs, hs)
941 else:
942 shape = (nb, nl, 2, bs, nh, hs)
943 ptr = int(paged_buffer_ptrs_tensor[0].item())
944 return _tensor_from_ptr(ptr, shape, dtype, device)
945 return paged_buffer_ptrs_tensor
946 raise TypeError(
947 "Cross-layer formats require a single torch.Tensor input; "
948 "got: " + type(paged_buffer_ptrs_tensor).__name__
949 )
950 if _is_sglang_mha_format(engine_kv_format):
951 if _is_ptr_tensor(paged_buffer_ptrs_tensor):
952 # 1-D pointer tensor [K_L0,...,K_LN-1, V_L0,...,V_LN-1] → nested list.
953 if shape_desc is None or device is None or dtype is None:
954 raise ValueError(
955 "_normalize_paged_layers: shape_desc, device, and dtype are "
956 "required when paged_buffer_ptrs_tensor is a pointer tensor"
957 )
958 nb = int(shape_desc.nb)
959 nl = int(shape_desc.nl)
960 bs = int(shape_desc.bs)
961 nh = int(shape_desc.nh)
962 hs = int(shape_desc.hs)
963 is_flat = int(engine_kv_format) == int(EngineKVFormat.TWO_X_NL_X_NBBS_NH_HS)

Callers 1

Calls 5

_is_cross_layer_formatFunction · 0.85
_is_ptr_tensorFunction · 0.85
_tensor_from_ptrFunction · 0.85
_is_sglang_mha_formatFunction · 0.85
_per_layer_paged_shapeFunction · 0.85

Tested by

no test coverage detected