Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avo
()
| 60 | |
| 61 | @lru_cache() |
| 62 | def bytes_to_unicode(): |
| 63 | """ |
| 64 | Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control |
| 65 | characters the bpe code barfs on. |
| 66 | |
| 67 | The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab |
| 68 | if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for |
| 69 | decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup |
| 70 | tables between utf-8 bytes and unicode strings. |
| 71 | """ |
| 72 | bs = ( |
| 73 | list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + list(range(ord("®"), ord("ÿ") + 1)) |
| 74 | ) |
| 75 | cs = bs[:] |
| 76 | n = 0 |
| 77 | for b in range(2**8): |
| 78 | if b not in bs: |
| 79 | bs.append(b) |
| 80 | cs.append(2**8 + n) |
| 81 | n += 1 |
| 82 | cs = [chr(n) for n in cs] |
| 83 | return dict(zip(bs, cs)) |
| 84 | |
| 85 | |
| 86 | def get_pairs(word): |