Class DetokenizeState

lmdeploy/tokenizer.py:16–36

A state collection for incremental detokenization. Args: ids_offset: offset to all input ids. In LMDeploy, the output ids length is not one by one. It could be random by random. prev_tokens: for incrementally decoding. Default to None, which means the fir

@dataclass
class DetokenizeState:
    """A state collection for incremental detokenization.

    Args:
        ids_offset: offset to all input ids. In LMDeploy, the output
            ids length is not one by one. It could be random by random.
        prev_tokens: for incrementally decoding.
            Default to None, which means the first round.
        prefix_offset: the start index of tokens to be converted to
            string (prev + new tokens). Default to 0 for the first round.
        read_offset: the end index of tokens to be converted to
            string (prev token). Default to 0 for the first round.
    """
    ids_offset: int = 0
    prev_tokens: list[str] | None = None
    prefix_offset: int = 0
    read_offset: int = 0

    def as_tuple(self) -> tuple:
        """Return a tuple of states."""
        return (self.ids_offset, self.prev_tokens, self.prefix_offset, self.read_offset)


class HuggingFaceTokenizer: