Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing
()
| 52 | |
| 53 | @lru_cache() |
| 54 | def bytes_to_unicode(): |
| 55 | """ |
| 56 | Returns list of utf-8 byte and a corresponding list of unicode strings. |
| 57 | The reversible bpe codes work on unicode strings. |
| 58 | This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. |
| 59 | When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. |
| 60 | This is a signficant percentage of your normal, say, 32K bpe vocab. |
| 61 | To avoid that, we want lookup tables between utf-8 bytes and unicode strings. |
| 62 | And avoids mapping to whitespace/control characters the bpe code barfs on. |
| 63 | """ |
| 64 | _chr = unichr if sys.version_info[0] == 2 else chr |
| 65 | bs = ( |
| 66 | list(range(ord("!"), ord("~") + 1)) |
| 67 | + list(range(ord("¡"), ord("¬") + 1)) |
| 68 | + list(range(ord("®"), ord("ÿ") + 1)) |
| 69 | ) |
| 70 | cs = bs[:] |
| 71 | n = 0 |
| 72 | for b in range(2 ** 8): |
| 73 | if b not in bs: |
| 74 | bs.append(b) |
| 75 | cs.append(2 ** 8 + n) |
| 76 | n += 1 |
| 77 | cs = [_chr(n) for n in cs] |
| 78 | return dict(zip(bs, cs)) |
| 79 | |
| 80 | |
| 81 | def get_pairs(word): |