Creates a mapping function to augment a tf.data.Dataset with lengths. Suppose we have a `dataset` with outputs `str` with shape [sequence_length]. Here is an example usage of this function: dataset = dataset.map(length_mapper(['str'])) Now the dataset will also include outputs `str_len` t
(keys_to_map, dims_to_map=None, suffix="_len")
| 24 | |
| 25 | |
| 26 | def length_mapper(keys_to_map, dims_to_map=None, suffix="_len"): |
| 27 | """Creates a mapping function to augment a tf.data.Dataset with lengths. |
| 28 | |
| 29 | Suppose we have a `dataset` with outputs `str` with shape [sequence_length]. |
| 30 | Here is an example usage of this function: |
| 31 | |
| 32 | dataset = dataset.map(length_mapper(['str'])) |
| 33 | |
| 34 | Now the dataset will also include outputs `str_len` that can be used for |
| 35 | masking out padding in a model. The 'str_len' feature will be a scalar int32 |
| 36 | Tensor. |
| 37 | |
| 38 | Args: |
| 39 | keys_to_map: List of strings that are keys for tf.string Tensors to map to |
| 40 | lengths. |
| 41 | dims_to_map: List of dimensions that representing the desired length to |
| 42 | retrieve. Defaults to the zero'th dimension if None. |
| 43 | suffix: String to append to the given keys to indicate the mapped Tensors. |
| 44 | |
| 45 | Returns: |
| 46 | _mapper: A mapping function that can be used with the tf.data.Dataset API. |
| 47 | """ |
| 48 | if dims_to_map is None: |
| 49 | dims_to_map = [0 for _ in keys_to_map] |
| 50 | def _mapper(dataset): |
| 51 | for k, d in zip(keys_to_map, dims_to_map): |
| 52 | dataset[k + suffix] = tf.shape(dataset[k])[d] |
| 53 | return dataset |
| 54 | return _mapper |
| 55 | |
| 56 | |
| 57 | def string_to_int_mapper(keys_to_map, mapping, num_oov_buckets=1, suffix="_id"): |
nothing calls this directly
no outgoing calls
no test coverage detected
searching dependent graphs…