Crop the past key values up to a new `maximum_length` in terms of tokens. `maximum_length` can also be negative to remove `maximum_length` tokens. This is used in assisted decoding and contrastive search.
(self, maximum_length: int)
| 1480 | |
| 1481 | # TODO(gante, sanchit-gandhi): move following functionality into `.generate` |
| 1482 | def crop(self, maximum_length: int): |
| 1483 | """Crop the past key values up to a new `maximum_length` in terms of tokens. `maximum_length` can also be |
| 1484 | negative to remove `maximum_length` tokens. This is used in assisted decoding and contrastive search.""" |
| 1485 | self.check_dynamic_cache(self.crop.__name__) |
| 1486 | self.self_attention_cache.crop(maximum_length) |
| 1487 | |
| 1488 | def batch_split(self, full_batch_size: int, split_size: int) -> "List[EncoderDecoderCache]": |
| 1489 | """Split the current instance into a list of `DynamicCache` by the batch size. This will be used by |
nothing calls this directly
no test coverage detected