MCPcopy Create free account
hub / github.com/google-research/language / get_document_uid

Function get_document_uid

language/realm/featurization.py:38–52  ·  view source on GitHub ↗

Generates a nearly unique ID for a document. Uses the SHA1 hash. Deterministically produces the same result for a given input, and almost never collides. Args: title (bytes): document title body (bytes): document body text Returns: uid (int)

(title, body)

Source from the content-addressed store, hash-verified

36
37
38def get_document_uid(title, body):
39 """Generates a nearly unique ID for a document.
40
41 Uses the SHA1 hash. Deterministically produces the same result for a given
42 input, and almost never collides.
43
44 Args:
45 title (bytes): document title
46 body (bytes): document body text
47
48 Returns:
49 uid (int)
50 """
51 b = '{} <sep> {}'.format(title.decode('utf-8'), body.decode('utf-8'))
52 return int(hashlib.sha1(b.encode('utf-8')).hexdigest(), 16)
53
54
55QUERY_ATTRS = ['text', 'tokens', 'mask_spans', 'orig_doc_uid']

Callers

nothing calls this directly

Calls 2

decodeMethod · 0.45
encodeMethod · 0.45

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…