MCPcopy Create free account
hub / github.com/google-research/language / _text_reader

Function _text_reader

language/table_text_eval/table_text_eval.py:78–93  ·  view source on GitHub ↗

Yields lines from the text file. Performs lowercasing and white-space tokenization on each line before returning. Args: text_file: String filename. multiple: Whether multiple references / generations are expected in a line.

(text_file, multiple=False)

Source from the content-addressed store, hash-verified

76
77
78def _text_reader(text_file, multiple=False):
79 """Yields lines from the text file.
80
81 Performs lowercasing and white-space tokenization on each line before
82 returning.
83
84 Args:
85 text_file: String filename.
86 multiple: Whether multiple references / generations are expected in a line.
87 """
88 with io.open(text_file) as f:
89 for line in f:
90 if multiple:
91 yield [item.lower().split() for item in line.strip().split("\t")]
92 else:
93 yield line.strip().lower().split()
94
95
96def _table_reader(table_file):

Callers 1

mainFunction · 0.70

Calls 1

splitMethod · 0.80

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…