Constructs an initializer for an string-to-id table from a text file. It populates a table that its key and value types are string and int64, respectively. It generates one key-value pair per line. The content of the key and value are specified by the key_index and value_index.
(self,
filename,
key_column_index=TextFileIndex.WHOLE_LINE,
value_column_index=TextFileIndex.LINE_NUMBER,
vocab_size=None,
delimiter="\t",
name="text_file_id_table_init",
key_dtype=dtypes.string)
| 730 | """Table initializer for string to `int64` IDs tables from a text file.""" |
| 731 | |
| 732 | def __init__(self, |
| 733 | filename, |
| 734 | key_column_index=TextFileIndex.WHOLE_LINE, |
| 735 | value_column_index=TextFileIndex.LINE_NUMBER, |
| 736 | vocab_size=None, |
| 737 | delimiter="\t", |
| 738 | name="text_file_id_table_init", |
| 739 | key_dtype=dtypes.string): |
| 740 | """Constructs an initializer for an string-to-id table from a text file. |
| 741 | |
| 742 | It populates a table that its key and value types are string and int64, |
| 743 | respectively. It generates one key-value pair per line. |
| 744 | The content of the key and value are specified by the key_index |
| 745 | and value_index. |
| 746 | |
| 747 | - TextFileIndex.LINE_NUMBER means use the line number starting from zero, |
| 748 | expects data type int64. |
| 749 | - TextFileIndex.WHOLE_LINE means use the whole line content, expects data |
| 750 | type string. |
| 751 | - A value >=0 means use the index (starting at zero) of the split line based |
| 752 | on `delimiter`. |
| 753 | |
| 754 | Args: |
| 755 | filename: The filename of the text file to be used for initialization. The |
| 756 | path must be accessible from wherever the graph is initialized (eg. |
| 757 | trainer or eval workers). The filename may be a scalar `Tensor`. |
| 758 | key_column_index: The column index from the text file to get the `key` |
| 759 | values from. The default is to use the whole line content. |
| 760 | value_column_index: The column index from the text file to get the `value` |
| 761 | values from. The default is to use the line number, starting from zero. |
| 762 | vocab_size: The number of elements in the file, if known. |
| 763 | delimiter: The delimiter to separate fields in a line. |
| 764 | name: Optional name for the op. |
| 765 | key_dtype: The `key` data type. |
| 766 | |
| 767 | Raises: |
| 768 | TypeError: when the filename is empty, or when the table key and value |
| 769 | data types do not match the expected data types. |
| 770 | """ |
| 771 | super(TextFileIdTableInitializer, self).__init__( |
| 772 | filename, |
| 773 | key_dtype, |
| 774 | key_column_index, |
| 775 | dtypes.int64, |
| 776 | value_column_index, |
| 777 | vocab_size=vocab_size, |
| 778 | delimiter=delimiter, |
| 779 | name=name) |
| 780 | |
| 781 | |
| 782 | class HasherSpec(collections.namedtuple("HasherSpec", ["hasher", "key"])): |