MCPcopy Create free account
hub / github.com/google-research/language / embedding_postprocessor

Function embedding_postprocessor

language/canine/bert_modeling.py:448–544  ·  view source on GitHub ↗

Performs various post-processing on a word embedding tensor. Args: input_tensor: float Tensor of shape [batch_size, seq_length, embedding_size]. use_token_type: bool. Whether to add embeddings for `token_type_ids`. token_type_ids: (optional) int32 Tensor of shape [batch_size, se

(input_tensor,
                            use_token_type=False,
                            token_type_ids=None,
                            token_type_vocab_size=16,
                            token_type_embedding_name="token_type_embeddings",
                            use_position_embeddings=True,
                            position_embedding_name="position_embeddings",
                            initializer_range=0.02,
                            max_position_embeddings=512,
                            dropout_prob=0.1)

Source from the content-addressed store, hash-verified

446
447
448def embedding_postprocessor(input_tensor,
449 use_token_type=False,
450 token_type_ids=None,
451 token_type_vocab_size=16,
452 token_type_embedding_name="token_type_embeddings",
453 use_position_embeddings=True,
454 position_embedding_name="position_embeddings",
455 initializer_range=0.02,
456 max_position_embeddings=512,
457 dropout_prob=0.1):
458 """Performs various post-processing on a word embedding tensor.
459
460 Args:
461 input_tensor: float Tensor of shape [batch_size, seq_length,
462 embedding_size].
463 use_token_type: bool. Whether to add embeddings for `token_type_ids`.
464 token_type_ids: (optional) int32 Tensor of shape [batch_size, seq_length].
465 Must be specified if `use_token_type` is True.
466 token_type_vocab_size: int. The vocabulary size of `token_type_ids`.
467 token_type_embedding_name: string. The name of the embedding table variable
468 for token type ids.
469 use_position_embeddings: bool. Whether to add position embeddings for the
470 position of each token in the sequence.
471 position_embedding_name: string. The name of the embedding table variable
472 for positional embeddings.
473 initializer_range: float. Range of the weight initialization.
474 max_position_embeddings: int. Maximum sequence length that might ever be
475 used with this model. This can be longer than the sequence length of
476 input_tensor, but cannot be shorter.
477 dropout_prob: float. Dropout probability applied to the final output tensor.
478
479 Returns:
480 float tensor with same shape as `input_tensor`.
481
482 Raises:
483 ValueError: One of the tensor shapes or input values is invalid.
484 """
485 input_shape = get_shape_list(input_tensor, expected_rank=3)
486 batch_size = input_shape[0]
487 seq_length = input_shape[1]
488 width = input_shape[2]
489
490 output = input_tensor
491
492 if use_token_type:
493 if token_type_ids is None:
494 raise ValueError("`token_type_ids` must be specified if"
495 "`use_token_type` is True.")
496 token_type_table = tf.get_variable(
497 name=token_type_embedding_name,
498 shape=[token_type_vocab_size, width],
499 initializer=create_initializer(initializer_range))
500 # This vocab will be small so we always do one-hot here, since it is always
501 # faster for a small vocabulary.
502 flat_token_type_ids = tf.reshape(token_type_ids, [-1])
503 one_hot_ids = tf.one_hot(flat_token_type_ids, depth=token_type_vocab_size)
504 token_type_embeddings = tf.matmul(one_hot_ids, token_type_table)
505 token_type_embeddings = tf.reshape(token_type_embeddings,

Callers 1

__init__Method · 0.85

Calls 4

create_initializerFunction · 0.85
layer_norm_and_dropoutFunction · 0.85
as_listMethod · 0.80
get_shape_listFunction · 0.70

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…