MCPcopy Create free account
hub / github.com/google-research/language / convert_single_example

Function convert_single_example

language/boolq/run_bert_boolq.py:197–286  ·  view source on GitHub ↗

Converts a single `InputExample` into a single `InputFeatures`.

(ex_index, example, label_list, max_seq_length,
                           tokenizer)

Source from the content-addressed store, hash-verified

195
196
197def convert_single_example(ex_index, example, label_list, max_seq_length,
198 tokenizer):
199 """Converts a single `InputExample` into a single `InputFeatures`."""
200 label_map = {}
201 for (i, label) in enumerate(label_list):
202 label_map[label] = i
203
204 tokens_a = tokenizer.tokenize(example.text_a)
205 tokens_b = None
206 if example.text_b:
207 tokens_b = tokenizer.tokenize(example.text_b)
208
209 if tokens_b:
210 # Modifies `tokens_a` and `tokens_b` in place so that the total
211 # length is less than the specified length.
212 # Account for [CLS], [SEP], [SEP] with "- 3"
213 _truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
214 else:
215 # Account for [CLS] and [SEP] with "- 2"
216 if len(tokens_a) > max_seq_length - 2:
217 tokens_a = tokens_a[0:(max_seq_length - 2)]
218
219 # The convention in BERT is:
220 # (a) For sequence pairs:
221 # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
222 # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1
223 # (b) For single sequences:
224 # tokens: [CLS] the dog is hairy . [SEP]
225 # type_ids: 0 0 0 0 0 0 0
226 #
227 # Where "type_ids" are used to indicate whether this is the first
228 # sequence or the second sequence. The embedding vectors for `type=0` and
229 # `type=1` were learned during pre-training and are added to the wordpiece
230 # embedding vector (and position vector). This is not *strictly* necessary
231 # since the [SEP] token unambiguously separates the sequences, but it makes
232 # it easier for the model to learn the concept of sequences.
233 #
234 # For classification tasks, the first vector (corresponding to [CLS]) is
235 # used as as the "sentence vector". Note that this only makes sense because
236 # the entire model is fine-tuned.
237 tokens = []
238 segment_ids = []
239 tokens.append("[CLS]")
240 segment_ids.append(0)
241 for token in tokens_a:
242 tokens.append(token)
243 segment_ids.append(0)
244 tokens.append("[SEP]")
245 segment_ids.append(0)
246
247 if tokens_b:
248 for token in tokens_b:
249 tokens.append(token)
250 segment_ids.append(1)
251 tokens.append("[SEP]")
252 segment_ids.append(1)
253
254 input_ids = tokenizer.convert_tokens_to_ids(tokens)

Calls 4

_truncate_seq_pairFunction · 0.70
InputFeaturesClass · 0.70
tokenizeMethod · 0.45
convert_tokens_to_idsMethod · 0.45

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…