Function _fetch_cloud_pages

openkb/indexer.py:262–287

Fetch all OCR pages of a cloud doc, windowing around the 1000-page cap. ``get_page_content`` returns the whole document and uses its ``pages`` arg only as a client-side filter that ``parse_pages`` caps at 1000 pages — so a single ``"1- "`` request fails for any doc over 1000 pages. Re

(col, doc_id: str)

def _fetch_cloud_pages(col, doc_id: str) -> list[dict[str, Any]]:
    """Fetch all OCR pages of a cloud doc, windowing around the 1000-page cap.

    ``get_page_content`` returns the whole document and uses its ``pages`` arg
    only as a client-side filter that ``parse_pages`` caps at 1000 pages — so a
    single ``"1-<N>"`` request fails for any doc over 1000 pages. Request fixed
    ``1000``-page windows and stop as soon as a window comes back SHORT (fewer
    than a full window): PageIndex page numbers are sequential, so a short window
    means we&#x27;ve passed the last page. This is what makes the common (≤1000-page)
    doc a single request, while still fetching every page of a larger one — and,
    unlike bounding the loop by the tree&#x27;s max page index, it never truncates a
    doc whose tree under-reports its page count (a real case: a paper whose tree
    stops a couple pages short of the references). A wide safety bound guards
    against a backend that never narrows the window.
    """
    pages: list[dict[str, Any]] = []
    start = 1
    while start <= _CLOUD_PAGE_MAX:
        window = _normalize_page_content(
            col.get_page_content(doc_id, f"{start}-{start + _CLOUD_PAGE_WINDOW - 1}")
        )
        pages.extend(window)
        if len(window) < _CLOUD_PAGE_WINDOW:
            break
        start += _CLOUD_PAGE_WINDOW
    return pages


def prepare_cloud_import(doc_id: str, kb_dir: Path, path_key: str) -> CloudImportData: