Embedding sequence-to-sequence model with attention. This model first embeds encoder_inputs by a newly created embedding (of shape [num_encoder_symbols x input_size]). Then it runs an RNN to encode embedded encoder_inputs into a state vector. It keeps the outputs of this RNN at every step t
(encoder_inputs,
decoder_inputs,
enc_cell,
dec_cell,
num_encoder_symbols,
num_decoder_symbols,
embedding_size,
num_heads=1,
output_projection=None,
feed_previous=False,
dtype=None,
scope=None,
initial_state_attention=False)
| 12 | |
| 13 | |
| 14 | def embedding_attention_seq2seq(encoder_inputs, |
| 15 | decoder_inputs, |
| 16 | enc_cell, |
| 17 | dec_cell, |
| 18 | num_encoder_symbols, |
| 19 | num_decoder_symbols, |
| 20 | embedding_size, |
| 21 | num_heads=1, |
| 22 | output_projection=None, |
| 23 | feed_previous=False, |
| 24 | dtype=None, |
| 25 | scope=None, |
| 26 | initial_state_attention=False): |
| 27 | """Embedding sequence-to-sequence model with attention. |
| 28 | |
| 29 | This model first embeds encoder_inputs by a newly created embedding (of shape |
| 30 | [num_encoder_symbols x input_size]). Then it runs an RNN to encode |
| 31 | embedded encoder_inputs into a state vector. It keeps the outputs of this |
| 32 | RNN at every step to use for attention later. Next, it embeds decoder_inputs |
| 33 | by another newly created embedding (of shape [num_decoder_symbols x |
| 34 | input_size]). Then it runs attention decoder, initialized with the last |
| 35 | encoder state, on embedded decoder_inputs and attending to encoder outputs. |
| 36 | |
| 37 | Warning: when output_projection is None, the size of the attention vectors |
| 38 | and variables will be made proportional to num_decoder_symbols, can be large. |
| 39 | |
| 40 | Args: |
| 41 | encoder_inputs: A list of 1D int32 Tensors of shape [batch_size]. |
| 42 | decoder_inputs: A list of 1D int32 Tensors of shape [batch_size]. |
| 43 | cell: tf.nn.rnn_cell.RNNCell defining the cell function and size. |
| 44 | num_encoder_symbols: Integer; number of symbols on the encoder side. |
| 45 | num_decoder_symbols: Integer; number of symbols on the decoder side. |
| 46 | embedding_size: Integer, the length of the embedding vector for each symbol. |
| 47 | num_heads: Number of attention heads that read from attention_states. |
| 48 | output_projection: None or a pair (W, B) of output projection weights and |
| 49 | biases; W has shape [output_size x num_decoder_symbols] and B has |
| 50 | shape [num_decoder_symbols]; if provided and feed_previous=True, each |
| 51 | fed previous output will first be multiplied by W and added B. |
| 52 | feed_previous: Boolean or scalar Boolean Tensor; if True, only the first |
| 53 | of decoder_inputs will be used (the "GO" symbol), and all other decoder |
| 54 | inputs will be taken from previous outputs (as in embedding_rnn_decoder). |
| 55 | If False, decoder_inputs are used as given (the standard decoder case). |
| 56 | dtype: The dtype of the initial RNN state (default: tf.float32). |
| 57 | scope: VariableScope for the created subgraph; defaults to |
| 58 | "embedding_attention_seq2seq". |
| 59 | initial_state_attention: If False (default), initial attentions are zero. |
| 60 | If True, initialize the attentions from the initial state and attention |
| 61 | states. |
| 62 | |
| 63 | Returns: |
| 64 | A tuple of the form (outputs, state), where: |
| 65 | outputs: A list of the same length as decoder_inputs of 2D Tensors with |
| 66 | shape [batch_size x num_decoder_symbols] containing the generated |
| 67 | outputs. |
| 68 | state: The state of each decoder cell at the final time-step. |
| 69 | It is a 2D Tensor of shape [batch_size x cell.state_size]. |
| 70 | """ |
| 71 | with variable_scope.variable_scope( |