Preapre all necessary files that are required for the training. Args: data_dir: directory in which the data sets will be stored. input_train_path: path to the file that includes "from" training samples. output_train_path: path to the file that includes "to" training samples.
(data_dir, input_train_path, output_train_path, input_dev_path, output_dev_path, input_vocabulary_size,
output_vocabulary_size, tokenizer=None)
| 240 | |
| 241 | |
| 242 | def prepare_data(data_dir, input_train_path, output_train_path, input_dev_path, output_dev_path, input_vocabulary_size, |
| 243 | output_vocabulary_size, tokenizer=None): |
| 244 | """Preapre all necessary files that are required for the training. |
| 245 | |
| 246 | Args: |
| 247 | data_dir: directory in which the data sets will be stored. |
| 248 | input_train_path: path to the file that includes "from" training samples. |
| 249 | output_train_path: path to the file that includes "to" training samples. |
| 250 | input_dev_path: path to the file that includes "from" dev samples. |
| 251 | output_dev_path: path to the file that includes "to" dev samples. |
| 252 | input_vocabulary_size: size of the "from language" vocabulary to create and use. |
| 253 | output_vocabulary_size: size of the "to language" vocabulary to create and use. |
| 254 | tokenizer: a function to use to tokenize each data sentence; |
| 255 | if None, basic_tokenizer will be used. |
| 256 | |
| 257 | Returns: |
| 258 | A tuple of 6 elements: |
| 259 | (1) path to the token-ids for "from language" training data-set, |
| 260 | (2) path to the token-ids for "to language" training data-set, |
| 261 | (3) path to the token-ids for "from language" development data-set, |
| 262 | (4) path to the token-ids for "to language" development data-set, |
| 263 | (5) path to the "from language" vocabulary file, |
| 264 | (6) path to the "to language" vocabulary file. |
| 265 | """ |
| 266 | # Create vocabularies of the appropriate sizes. |
| 267 | output_vocab_path = os.path.join(data_dir, "vocab%d.output" % output_vocabulary_size) |
| 268 | input_vocab_path = os.path.join(data_dir, "vocab%d.input" % input_vocabulary_size) |
| 269 | create_vocabulary(output_vocab_path, output_train_path , output_vocabulary_size, tokenizer) |
| 270 | create_vocabulary(input_vocab_path, input_train_path , input_vocabulary_size, tokenizer) |
| 271 | |
| 272 | # Create token ids for the training data. |
| 273 | output_train_ids_path = output_train_path + (".ids%d" % output_vocabulary_size) |
| 274 | input_train_ids_path = input_train_path + (".ids%d" % input_vocabulary_size) |
| 275 | data_to_token_ids(output_train_path, output_train_ids_path, output_vocab_path, tokenizer) |
| 276 | data_to_token_ids(input_train_path, input_train_ids_path, input_vocab_path, tokenizer) |
| 277 | |
| 278 | # Create token ids for the development data. |
| 279 | output_dev_ids_path = output_dev_path + (".ids%d" % output_vocabulary_size) |
| 280 | input_dev_ids_path = input_dev_path + (".ids%d" % input_vocabulary_size) |
| 281 | data_to_token_ids(output_dev_path, output_dev_ids_path, output_vocab_path, tokenizer) |
| 282 | data_to_token_ids(input_dev_path, input_dev_ids_path, input_vocab_path, tokenizer) |
| 283 | |
| 284 | return (input_train_ids_path, output_train_ids_path, |
| 285 | input_dev_ids_path, output_dev_ids_path, |
| 286 | input_vocab_path, output_vocab_path) |
no test coverage detected