MCPcopy Create free account
hub / github.com/google-research/language / read

Function read

language/orqa/models/orqa_model.py:152–259  ·  view source on GitHub ↗

Do reading.

(features, retriever_logits, blocks, mode, params, labels)

Source from the content-addressed store, hash-verified

150
151
152def read(features, retriever_logits, blocks, mode, params, labels):
153 """Do reading."""
154 tokenizer, vocab_lookup_table = bert_utils.get_tf_tokenizer(
155 params["reader_module_path"])
156
157 orig_blocks = blocks
158
159 (orig_tokens, block_token_map, block_token_ids, blocks) = (
160 bert_utils.tokenize_with_original_mapping(blocks, tokenizer))
161
162 question_token_ids = tokenizer.tokenize(
163 tf.expand_dims(features["question"], 0))
164 question_token_ids = tf.cast(question_token_ids.flat_values, tf.int32)
165
166 orig_tokens = orig_tokens.to_tensor(default_value="")
167 block_lengths = tf.cast(block_token_ids.row_lengths(), tf.int32)
168 block_token_ids = tf.cast(block_token_ids.to_tensor(), tf.int32)
169 block_token_map = tf.cast(block_token_map.to_tensor(), tf.int32)
170
171 answer_token_ids = tokenizer.tokenize(labels).merge_dims(1, 2)
172 answer_lengths = tf.cast(answer_token_ids.row_lengths(), tf.int32)
173 answer_token_ids = tf.cast(answer_token_ids.to_tensor(), tf.int32)
174
175 cls_token_id = vocab_lookup_table.lookup(tf.constant("[CLS]"))
176 sep_token_id = vocab_lookup_table.lookup(tf.constant("[SEP]"))
177 concat_inputs = orqa_ops.reader_inputs(
178 question_token_ids=question_token_ids,
179 block_token_ids=block_token_ids,
180 block_lengths=block_lengths,
181 block_token_map=block_token_map,
182 answer_token_ids=answer_token_ids,
183 answer_lengths=answer_lengths,
184 cls_token_id=tf.cast(cls_token_id, tf.int32),
185 sep_token_id=tf.cast(sep_token_id, tf.int32),
186 max_sequence_len=params["reader_seq_len"])
187
188 tf.summary.scalar("reader_nonpad_ratio",
189 tf.reduce_mean(tf.cast(concat_inputs.mask, tf.float32)))
190
191 reader_module = hub.Module(
192 params["reader_module_path"],
193 tags={"train"} if mode == tf_estimator.ModeKeys.TRAIN else {},
194 trainable=True)
195
196 concat_outputs = reader_module(
197 dict(
198 input_ids=concat_inputs.token_ids,
199 input_mask=concat_inputs.mask,
200 segment_ids=concat_inputs.segment_ids),
201 signature="tokens",
202 as_dict=True)
203
204 concat_token_emb = concat_outputs["sequence_output"]
205
206 # [num_spans], [num_spans], [reader_beam_size, num_spans]
207 candidate_starts, candidate_ends, candidate_mask = span_candidates(
208 concat_inputs.block_mask, params["max_span_width"])
209

Callers 1

model_fnFunction · 0.70

Calls 6

span_candidatesFunction · 0.85
mask_to_scoreFunction · 0.85
to_tensorMethod · 0.80
constantMethod · 0.80
splitMethod · 0.80
tokenizeMethod · 0.45

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…