Transform an encoded sequence of byte pair codeword IDs back into human-readable text. Parameters ---------- codes : list of `N` lists A list of `N` lists. Each sublist is a collection of integer byte-pair token IDs representing a par
(self, codes)
| 337 | return encoded |
| 338 | |
| 339 | def inverse_transform(self, codes): |
| 340 | """ |
| 341 | Transform an encoded sequence of byte pair codeword IDs back into |
| 342 | human-readable text. |
| 343 | |
| 344 | Parameters |
| 345 | ---------- |
| 346 | codes : list of `N` lists |
| 347 | A list of `N` lists. Each sublist is a collection of integer |
| 348 | byte-pair token IDs representing a particular text string. |
| 349 | |
| 350 | Returns |
| 351 | ------- |
| 352 | text: list of `N` strings |
| 353 | The decoded strings corresponding to the `N` sublists in `codes`. |
| 354 | |
| 355 | Examples |
| 356 | -------- |
| 357 | >>> B = BytePairEncoder(max_merges=100).fit("./example.txt") |
| 358 | >>> encoded_tokens = B.transform("Hello! How are you 😁 ?") |
| 359 | >>> encoded_tokens |
| 360 | [[72, 879, 474, ...]] |
| 361 | >>> B.inverse_transform(encoded_tokens) |
| 362 | ["Hello! How are you 😁 ?"] |
| 363 | """ |
| 364 | if isinstance(codes[0], int): |
| 365 | codes = [codes] |
| 366 | |
| 367 | decoded = [] |
| 368 | P = self.parameters |
| 369 | |
| 370 | for code in codes: |
| 371 | _bytes = [self.token2byte[t] if t > 255 else [t] for t in code] |
| 372 | _bytes = [b for blist in _bytes for b in blist] |
| 373 | decoded.append(bytes_to_chars(_bytes, encoding=P["encoding"])) |
| 374 | return decoded |
| 375 | |
| 376 | @property |
| 377 | def codebook(self): |
no test coverage detected