MCPcopy Create free account
hub / github.com/google-research/language / read

Function read

language/orqa/models/text_classifier_model.py:28–95  ·  view source on GitHub ↗

Do reading.

(features, retriever_logits, blocks, mode, params)

Source from the content-addressed store, hash-verified

26
27
28def read(features, retriever_logits, blocks, mode, params):
29 """Do reading."""
30 tokenizer, vocab_lookup_table = bert_utils.get_tf_tokenizer(
31 params["reader_module_path"])
32
33 (orig_tokens, block_token_map, block_token_ids, blocks) = (
34 bert_utils.tokenize_with_original_mapping(blocks, tokenizer))
35
36 # NOTE: we assume that the batch size is 1.
37 question_token_ids = tokenizer.tokenize(
38 tf.expand_dims(features["question"], 0))
39 question_token_ids = tf.cast(question_token_ids.flat_values, tf.int32)
40
41 orig_tokens = orig_tokens.to_tensor(default_value="")
42 block_lengths = tf.cast(block_token_ids.row_lengths(), tf.int32)
43 block_token_ids = tf.cast(block_token_ids.to_tensor(), tf.int32)
44 block_token_map = tf.cast(block_token_map.to_tensor(), tf.int32)
45
46 fake_answer = tf.constant([""])
47 answer_token_ids = tokenizer.tokenize(fake_answer).merge_dims(1, 2)
48 answer_lengths = tf.cast(answer_token_ids.row_lengths(), tf.int32)
49 answer_token_ids = tf.cast(answer_token_ids.to_tensor(), tf.int32)
50
51 cls_token_id = vocab_lookup_table.lookup(tf.constant("[CLS]"))
52 sep_token_id = vocab_lookup_table.lookup(tf.constant("[SEP]"))
53
54 concat_inputs = orqa_ops.reader_inputs(
55 question_token_ids=question_token_ids,
56 block_token_ids=block_token_ids,
57 block_lengths=block_lengths,
58 block_token_map=block_token_map,
59 answer_token_ids=answer_token_ids,
60 answer_lengths=answer_lengths,
61 cls_token_id=tf.cast(cls_token_id, tf.int32),
62 sep_token_id=tf.cast(sep_token_id, tf.int32),
63 max_sequence_len=params["reader_seq_len"])
64
65 tf.summary.scalar("reader_nonpad_ratio",
66 tf.reduce_mean(tf.cast(concat_inputs.mask, tf.float32)))
67
68 reader_module = hub.Module(
69 params["reader_module_path"],
70 tags={"train"} if mode == tf_estimator.ModeKeys.TRAIN else {},
71 trainable=True)
72
73 concat_outputs = reader_module(
74 dict(
75 input_ids=concat_inputs.token_ids,
76 input_mask=concat_inputs.mask,
77 segment_ids=concat_inputs.segment_ids),
78 signature="tokens",
79 as_dict=True)
80
81 with tf.variable_scope("pooler"):
82 # [reader_beam_size, hidden_size]
83 first_token_tensor = concat_outputs["pooled_output"]
84
85 kernel_initializer = tf.truncated_normal_initializer(stddev=0.02)

Callers 1

model_fnFunction · 0.70

Calls 3

to_tensorMethod · 0.80
constantMethod · 0.80
tokenizeMethod · 0.45

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…