Generates a nearly unique ID for a document. Uses the SHA1 hash. Deterministically produces the same result for a given input, and almost never collides. Args: title (bytes): document title body (bytes): document body text Returns: uid (int)
(title, body)
| 36 | |
| 37 | |
| 38 | def get_document_uid(title, body): |
| 39 | """Generates a nearly unique ID for a document. |
| 40 | |
| 41 | Uses the SHA1 hash. Deterministically produces the same result for a given |
| 42 | input, and almost never collides. |
| 43 | |
| 44 | Args: |
| 45 | title (bytes): document title |
| 46 | body (bytes): document body text |
| 47 | |
| 48 | Returns: |
| 49 | uid (int) |
| 50 | """ |
| 51 | b = '{} <sep> {}'.format(title.decode('utf-8'), body.decode('utf-8')) |
| 52 | return int(hashlib.sha1(b.encode('utf-8')).hexdigest(), 16) |
| 53 | |
| 54 | |
| 55 | QUERY_ATTRS = ['text', 'tokens', 'mask_spans', 'orig_doc_uid'] |