Generates from an iterable of tokenized prompts. This method is built on top of :meth:`ctranslate2.Generator.generate_batch` to efficiently run generation on an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully
(
generator: Generator,
start_tokens: Iterable[List[str]],
max_batch_size: int = 32,
batch_type: str = "examples",
**kwargs,
)
| 128 | |
| 129 | |
| 130 | def generator_generate_iterable( |
| 131 | generator: Generator, |
| 132 | start_tokens: Iterable[List[str]], |
| 133 | max_batch_size: int = 32, |
| 134 | batch_type: str = "examples", |
| 135 | **kwargs, |
| 136 | ) -> Iterable[GenerationResult]: |
| 137 | """Generates from an iterable of tokenized prompts. |
| 138 | |
| 139 | This method is built on top of :meth:`ctranslate2.Generator.generate_batch` |
| 140 | to efficiently run generation on an arbitrarily large stream of data. It enables |
| 141 | the following optimizations: |
| 142 | |
| 143 | * stream processing (the iterable is not fully materialized in memory) |
| 144 | * parallel generations (if the generator has multiple workers) |
| 145 | * asynchronous batch prefetching |
| 146 | * local sorting by length |
| 147 | |
| 148 | Arguments: |
| 149 | start_tokens: An iterable of tokenized prompts. |
| 150 | max_batch_size: The maximum batch size. |
| 151 | batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". |
| 152 | **kwargs: Any generation options accepted by |
| 153 | :meth:`ctranslate2.Generator.generate_batch`. |
| 154 | |
| 155 | Returns: |
| 156 | A generator iterator over :class:`ctranslate2.GenerationResult` instances. |
| 157 | """ |
| 158 | yield from _process_iterable( |
| 159 | generator.generate_batch, |
| 160 | [start_tokens], |
| 161 | max_batch_size, |
| 162 | batch_type, |
| 163 | **kwargs, |
| 164 | ) |
| 165 | |
| 166 | |
| 167 | def generator_score_iterable( |
nothing calls this directly
no test coverage detected