MCPcopy Create free account
hub / github.com/google-research/language / parse_examples

Function parse_examples

language/multivec/predict/encode_blocks.py:111–125  ·  view source on GitHub ↗

Make retrieval examples.

(serialized_example)

Source from the content-addressed store, hash-verified

109
110
111def parse_examples(serialized_example):
112 """Make retrieval examples."""
113 feature_spec = dict(
114 input_ids=tf.FixedLenSequenceFeature([], tf.int64, True),
115 key=tf.FixedLenSequenceFeature([], tf.int64, True))
116 features = tf.parse_single_example(serialized_example, feature_spec)
117 features = {k: tf.cast(v, tf.int32) for k, v in features.items()}
118 block_ids, block_mask, block_segment_ids = pad_or_truncate_pair(
119 token_ids=features["input_ids"], sequence_length=FLAGS.block_seq_len)
120 key = tf.ensure_shape(features["key"], [1])
121 return dict(
122 block_ids=block_ids,
123 block_mask=block_mask,
124 block_segment_ids=block_segment_ids,
125 key=key)
126
127
128def input_fn(params):

Callers

nothing calls this directly

Calls 1

pad_or_truncate_pairFunction · 0.70

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…