Generic Unicode table text file reader. The reader takes care of stripping out comments and also parsing the two different ways that the Unicode tables specify code ranges (using the .. notation and splitting the range across multiple lines). Each non-comment line in the table is expecte
(filename, nfields, doline)
| 122 | |
| 123 | |
| 124 | def ReadUnicodeTable(filename, nfields, doline): |
| 125 | """Generic Unicode table text file reader. |
| 126 | |
| 127 | The reader takes care of stripping out comments and also |
| 128 | parsing the two different ways that the Unicode tables specify |
| 129 | code ranges (using the .. notation and splitting the range across |
| 130 | multiple lines). |
| 131 | |
| 132 | Each non-comment line in the table is expected to have the given |
| 133 | number of fields. The first field is known to be the Unicode value |
| 134 | and the second field its description. |
| 135 | |
| 136 | The reader calls doline(codes, fields) for each entry in the table. |
| 137 | If fn raises an exception, the reader prints that exception, |
| 138 | prefixed with the file name and line number, and continues |
| 139 | processing the file. When done with the file, the reader re-raises |
| 140 | the first exception encountered during the file. |
| 141 | |
| 142 | Arguments: |
| 143 | filename: the Unicode data file to read, or a file-like object. |
| 144 | nfields: the number of expected fields per line in that file. |
| 145 | doline: the function to call for each table entry. |
| 146 | |
| 147 | Raises: |
| 148 | InputError: nfields is invalid (must be >= 2). |
| 149 | """ |
| 150 | |
| 151 | if nfields < 2: |
| 152 | raise InputError("invalid number of fields %d" % (nfields,)) |
| 153 | |
| 154 | if type(filename) == str: |
| 155 | if filename.startswith("https://"): |
| 156 | fil = urllib.request.urlopen(filename) |
| 157 | else: |
| 158 | fil = open(filename, "rb") |
| 159 | else: |
| 160 | fil = filename |
| 161 | |
| 162 | first = None # first code in multiline range |
| 163 | expect_last = None # tag expected for "Last" line in multiline range |
| 164 | lineno = 0 # current line number |
| 165 | for line in fil: |
| 166 | lineno += 1 |
| 167 | try: |
| 168 | line = line.decode('latin1') |
| 169 | |
| 170 | # Chop # comments and white space; ignore empty lines. |
| 171 | sharp = line.find("#") |
| 172 | if sharp >= 0: |
| 173 | line = line[:sharp] |
| 174 | line = line.strip() |
| 175 | if not line: |
| 176 | continue |
| 177 | |
| 178 | # Split fields on ";", chop more white space. |
| 179 | # Must have the expected number of fields. |
| 180 | fields = [s.strip() for s in line.split(";")] |
| 181 | if len(fields) != nfields: |
no test coverage detected