Split text on paragraph boundaries into chunks under max_bytes each. The local LanguageTool server crashes ('Connection reset by peer') when fed multi-megabyte inputs in a single request, so we batch by paragraph. Yields (offset, chunk_text) pairs so callers can translate per-chunk
(text, max_bytes=CHUNK_BYTES)
| 214 | |
| 215 | |
| 216 | def chunk_text(text, max_bytes=CHUNK_BYTES): |
| 217 | """Split text on paragraph boundaries into chunks under max_bytes each. |
| 218 | |
| 219 | The local LanguageTool server crashes ('Connection reset by peer') when |
| 220 | fed multi-megabyte inputs in a single request, so we batch by paragraph. |
| 221 | Yields (offset, chunk_text) pairs so callers can translate per-chunk |
| 222 | offsets back to a global offset. |
| 223 | """ |
| 224 | paragraphs = text.split("\n\n") |
| 225 | buf = [] |
| 226 | buf_len = 0 |
| 227 | offset = 0 |
| 228 | chunk_start = 0 |
| 229 | for para in paragraphs: |
| 230 | segment = para + "\n\n" |
| 231 | if buf and buf_len + len(segment) > max_bytes: |
| 232 | yield chunk_start, "".join(buf) |
| 233 | chunk_start = offset |
| 234 | buf = [segment] |
| 235 | buf_len = len(segment) |
| 236 | else: |
| 237 | buf.append(segment) |
| 238 | buf_len += len(segment) |
| 239 | offset += len(segment) |
| 240 | if buf: |
| 241 | yield chunk_start, "".join(buf) |
| 242 | |
| 243 | |
| 244 | # Rules whose findings are dominated by stylistic preferences or |
no test coverage detected