MCPcopy Create free account
hub / github.com/google-research/language / length_mapper

Function length_mapper

language/common/inputs/dataset_utils.py:26–54  ·  view source on GitHub ↗

Creates a mapping function to augment a tf.data.Dataset with lengths. Suppose we have a `dataset` with outputs `str` with shape [sequence_length]. Here is an example usage of this function: dataset = dataset.map(length_mapper(['str'])) Now the dataset will also include outputs `str_len` t

(keys_to_map, dims_to_map=None, suffix="_len")

Source from the content-addressed store, hash-verified

24
25
26def length_mapper(keys_to_map, dims_to_map=None, suffix="_len"):
27 """Creates a mapping function to augment a tf.data.Dataset with lengths.
28
29 Suppose we have a `dataset` with outputs `str` with shape [sequence_length].
30 Here is an example usage of this function:
31
32 dataset = dataset.map(length_mapper(['str']))
33
34 Now the dataset will also include outputs `str_len` that can be used for
35 masking out padding in a model. The 'str_len' feature will be a scalar int32
36 Tensor.
37
38 Args:
39 keys_to_map: List of strings that are keys for tf.string Tensors to map to
40 lengths.
41 dims_to_map: List of dimensions that representing the desired length to
42 retrieve. Defaults to the zero'th dimension if None.
43 suffix: String to append to the given keys to indicate the mapped Tensors.
44
45 Returns:
46 _mapper: A mapping function that can be used with the tf.data.Dataset API.
47 """
48 if dims_to_map is None:
49 dims_to_map = [0 for _ in keys_to_map]
50 def _mapper(dataset):
51 for k, d in zip(keys_to_map, dims_to_map):
52 dataset[k + suffix] = tf.shape(dataset[k])[d]
53 return dataset
54 return _mapper
55
56
57def string_to_int_mapper(keys_to_map, mapping, num_oov_buckets=1, suffix="_id"):

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…