Collate function for encoding. :param features: list of (id, text) tuples
(self, features: List[Tuple[str, str]])
| 95 | tokenizer: PreTrainedTokenizer |
| 96 | |
| 97 | def __call__(self, features: List[Tuple[str, str]]): |
| 98 | """ |
| 99 | Collate function for encoding. |
| 100 | :param features: list of (id, text) tuples |
| 101 | """ |
| 102 | text_ids = [x[0] for x in features] |
| 103 | texts = [x[1] for x in features] |
| 104 | max_length = self.data_args.query_max_len if self.data_args.encode_is_query else self.data_args.passage_max_len |
| 105 | collated_texts = self.tokenizer( |
| 106 | texts, |
| 107 | padding=True, |
| 108 | truncation=True, |
| 109 | max_length=max_length, |
| 110 | return_attention_mask=True, |
| 111 | return_token_type_ids=False, |
| 112 | add_special_tokens=True, |
| 113 | return_tensors='pt', |
| 114 | ) |
| 115 | return text_ids, collated_texts |
nothing calls this directly
no outgoing calls
no test coverage detected