Performs various post-processing on a word embedding tensor. Args: input_tensor: float Tensor of shape [batch_size, seq_length, embedding_size]. use_token_type: bool. Whether to add embeddings for `token_type_ids`. token_type_ids: (optional) int32 Tensor of shape [batch_size, se
(input_tensor,
use_token_type=False,
token_type_ids=None,
token_type_vocab_size=16,
token_type_embedding_name="token_type_embeddings",
use_position_embeddings=True,
position_embedding_name="position_embeddings",
initializer_range=0.02,
max_position_embeddings=512,
dropout_prob=0.1)
| 446 | |
| 447 | |
| 448 | def embedding_postprocessor(input_tensor, |
| 449 | use_token_type=False, |
| 450 | token_type_ids=None, |
| 451 | token_type_vocab_size=16, |
| 452 | token_type_embedding_name="token_type_embeddings", |
| 453 | use_position_embeddings=True, |
| 454 | position_embedding_name="position_embeddings", |
| 455 | initializer_range=0.02, |
| 456 | max_position_embeddings=512, |
| 457 | dropout_prob=0.1): |
| 458 | """Performs various post-processing on a word embedding tensor. |
| 459 | |
| 460 | Args: |
| 461 | input_tensor: float Tensor of shape [batch_size, seq_length, |
| 462 | embedding_size]. |
| 463 | use_token_type: bool. Whether to add embeddings for `token_type_ids`. |
| 464 | token_type_ids: (optional) int32 Tensor of shape [batch_size, seq_length]. |
| 465 | Must be specified if `use_token_type` is True. |
| 466 | token_type_vocab_size: int. The vocabulary size of `token_type_ids`. |
| 467 | token_type_embedding_name: string. The name of the embedding table variable |
| 468 | for token type ids. |
| 469 | use_position_embeddings: bool. Whether to add position embeddings for the |
| 470 | position of each token in the sequence. |
| 471 | position_embedding_name: string. The name of the embedding table variable |
| 472 | for positional embeddings. |
| 473 | initializer_range: float. Range of the weight initialization. |
| 474 | max_position_embeddings: int. Maximum sequence length that might ever be |
| 475 | used with this model. This can be longer than the sequence length of |
| 476 | input_tensor, but cannot be shorter. |
| 477 | dropout_prob: float. Dropout probability applied to the final output tensor. |
| 478 | |
| 479 | Returns: |
| 480 | float tensor with same shape as `input_tensor`. |
| 481 | |
| 482 | Raises: |
| 483 | ValueError: One of the tensor shapes or input values is invalid. |
| 484 | """ |
| 485 | input_shape = get_shape_list(input_tensor, expected_rank=3) |
| 486 | batch_size = input_shape[0] |
| 487 | seq_length = input_shape[1] |
| 488 | width = input_shape[2] |
| 489 | |
| 490 | output = input_tensor |
| 491 | |
| 492 | if use_token_type: |
| 493 | if token_type_ids is None: |
| 494 | raise ValueError("`token_type_ids` must be specified if" |
| 495 | "`use_token_type` is True.") |
| 496 | token_type_table = tf.get_variable( |
| 497 | name=token_type_embedding_name, |
| 498 | shape=[token_type_vocab_size, width], |
| 499 | initializer=create_initializer(initializer_range)) |
| 500 | # This vocab will be small so we always do one-hot here, since it is always |
| 501 | # faster for a small vocabulary. |
| 502 | flat_token_type_ids = tf.reshape(token_type_ids, [-1]) |
| 503 | one_hot_ids = tf.one_hot(flat_token_type_ids, depth=token_type_vocab_size) |
| 504 | token_type_embeddings = tf.matmul(one_hot_ids, token_type_table) |
| 505 | token_type_embeddings = tf.reshape(token_type_embeddings, |
no test coverage detected
searching dependent graphs…