---

OCRFlux is a multimodal large language model based toolkit for converting PDFs and images into clean, readable, plain Markdown text. It aims to push the current state-of-the-art to a significantly higher level.

Functions: Whole file parsing - On each page - Convert into text with a natural reading order, even in the presence of multi-column layouts, figures, and insets - Support for complicated tables and equations - Automatically removes headers and footers

• Cross-page table/paragraph merging
  • Cross-page table merging
  • Cross-page paragraph merging

Key features: - Superior parsing quality on each page

It respectively achieves 0.095 higher (from 0.872 to 0.967), 0.109 higher (from 0.858 to 0.967) and 0.187 higher (from 0.780 to 0.967) Edit Distance Similarity (EDS) on our released benchmark [OCRFlux-bench-single](https://huggingface.co/datasets/ChatDOC/OCRFlux-bench-single) than the baseline model [olmOCR-7B-0225-preview](https://huggingface.co/allenai/olmOCR-7B-0225-preview), [Nanonets-OCR-s](https://huggingface.co/nanonets/Nanonets-OCR-s) and [MonkeyOCR](https://huggingface.co/echo840/MonkeyOCR).

• Native support for cross-page table/paragraph merging (to our best this is the first to support this feature in all the open sourced project).

• Based on a 3B parameter VLM, so it can run even on GTX 3090 GPU.

Release: - OCRFlux-3B - 3B parameter VLM - Benchmark for evaluation - OCRFlux-bench-single - OCRFlux-pubtabnet-single - OCRFlux-bench-cross - OCRFlux-pubtabnet-cross

News

• Jun 17, 2025 - v0.1.0 - Initial public launch and demo.

Benchmark for single-page parsing

We ship two comprehensive benchmarks to help measure the performance of our OCR system in single-page parsing:

• OCRFlux-bench-single: Containing 2000 pdf pages (1000 English pages and 1000 Chinese pages) and their ground-truth Markdowns (manually labeled with multi-round check).

• OCRFlux-pubtabnet-single: Derived from the public PubTabNet benchmark with some format transformation. It contains 9064 HTML table samples, which are split into simple tables and complex tables according to whether they have rowspan and colspan cells.

We emphasize that the released benchmarks are NOT included in our training and evaluation data. The following is the main result:

1. In OCRFlux-bench-single, we calculated the Edit Distance Similarity (EDS) between the generated Markdowns and the ground-truth Markdowns as the metric.

   Language	Model	Avg EDS ↑
   English	olmOCR-7B-0225-preview	0.885
   Nanonets-OCR-s	0.870
   MonkeyOCR	0.828
   OCRFlux-3B	0.971
   Chinese	olmOCR-7B-0225-preview	0.859
   Nanonets-OCR-s	0.846
   MonkeyOCR	0.731
   OCRFlux-3B	0.962
   Total	olmOCR-7B-0225-preview	0.872
   Nanonets-OCR-s	0.858
   MonkeyOCR	0.780
   OCRFlux-3B	0.967

2. In OCRFlux-pubtabnet-single, we calculated the Tree Edit Distance-based Similarity (TEDS) between the generated HTML tables and the ground-truth HTML tables as the metric.

   Type	Model	Avg TEDS ↑
   Simple	olmOCR-7B-0225-preview	0.810
   Nanonets-OCR-s	0.882
   MonkeyOCR	0.880
   OCRFlux-3B	0.912
   Complex	olmOCR-7B-0225-preview	0.676
   Nanonets-OCR-s	0.772
   MonkeyOCR	0.826
   OCRFlux-3B	0.807
   Total	olmOCR-7B-0225-preview	0.744
   Nanonets-OCR-s	0.828
   MonkeyOCR	0.853
   OCRFlux-3B	0.861

We also conduct some case studies to show the superiority of our model in the blog article.

Benchmark for cross-page table/paragraph merging

PDF documents are typically paginated, which often results in tables or paragraphs being split across consecutive pages. Accurately detecting and merging such cross-page structures is crucial to avoid generating incomplete or fragmented content.

The detection task can be formulated as follows: given the Markdowns of two consecutive pages—each structured as a list of Markdown elements (e.g., paragraphs and tables)—the goal is to identify the indexes of elements that should be merged across the pages.

Then for the merging task, if the elements to be merged are paragraphs, we can just concate them. However, for two table fragments, their merging is much more challenging. For example, the table spanning multiple pages will repeat the header of the first page on the second page. Another difficult scenario is that the table cell contains long content that spans multiple lines within the cell, with the first few lines appearing on the previous page and the remaining lines continuing on the next page. We also observe some cases where tables with a large number of columns are split vertically and placed on two consecutive pages. More examples of cross-page tables can be found in our blog article. To address these issues, we develop the LLM model for cross-page table merging. Specifically, this model takes two split table fragments as input and generates a complete, well-structured table as output.

We ship two comprehensive benchmarks to help measure the performance of our OCR system in cross-page table/paragraph detection and merging tasks respectively:

• OCRFlux-bench-cross: Containing 1000 samples (500 English samples and 500 Chinese samples), each sample contains the Markdown element lists of two consecutive pages, along with the indexes of elements that need to be merged (manually labeled through multiple rounds of review). If no tables or paragraphs require merging, the indexes in the annotation data are left empty.

• OCRFlux-pubtabnet-cross: Containing 9064 pairs of split table fragments, along with their corresponding ground-truth merged versions.

The released benchmarks are NOT included in our training and evaluation data neither. The following is the main result:

1. In OCRFlux-bench-cross, we caculated the Accuracy, Precision, Recall and F1 score as the metric. Notice that the detection results are right only when it accurately judges whether there are elements that need to be merged across the two pages and output the right indexes of them.

   Language	Precision ↑	Recall ↑	F1 ↑	Accuracy ↑
   English	0.992	0.964	0.978	0.978
   Chinese	1.000	0.988	0.994	0.994
   Total	0.996	0.976	0.986	0.986

2. In OCRFlux-pubtabnet-cross, we calculate the Tree Edit Distance-based Similarity (TEDS) between the generated merged table and the ground-truth merged table as the metric.

   Table type	Avg TEDS ↑
   Simple	0.965
   Complex	0.935
   Total	0.950

Installation

Requirements: - Recent NVIDIA GPU (tested on RTX 3090, 4090, L40S, A100, H100) with at least 12 GB of GPU RAM - 20GB of free disk space

You will need to install poppler-utils and additional fonts for rendering PDF images.

Install dependencies (Ubuntu/Debian)

sudo apt-get update
sudo apt-get install poppler-utils poppler-data ttf-mscorefonts-installer msttcorefonts fonts-crosextra-caladea fonts-crosextra-carlito gsfonts lcdf-typetools

Set up a conda environment and install OCRFlux. The requirements for running OCRFlux are difficult to install in an existing python environment, so please do make a clean python environment to install into.

conda create -n ocrflux python=3.11
conda activate ocrflux

git clone https://github.com/chatdoc-com/OCRFlux.git
cd OCRFlux

pip install -e . --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer/

Usage

For quick testing, try the web demo. To run locally, a GPU is required, as inference is powered by vllm under the hood.

Pipeline

• For a pdf document: bash python -m ocrflux.pipeline ./localworkspace --data test.pdf --model /model_dir/OCRFlux-3B

• For an image: bash python -m ocrflux.pipeline ./localworkspace --data test_page.png --model /model_dir/OCRFlux-3B

• For a directory of pdf or images: bash python -m ocrflux.pipeline ./localworkspace --data test_pdf_dir/* --model /model_dir/OCRFlux-3B

Notices: - You can set --skip_cross_page_merge to skip the cross-page merging in the parsing process to accelerate, it would simply concatenate the parsing results of each page to generate final Markdown of the document.

• You can set --gpu_memory_utilization to set GPU memory utiliziation, e.g. --gpu_memory_utilization 0.9, default is 0.8.

• OCRFlux is recommended to run on a GPU with more than 24GB of VRAM. However, if you have multiple smaller GPUs (e.g., 12GB), you can set --tensor_parallel_size N to run it on N GPUs.

• When using OCRFlux on GPUs which do not support bf16 like V100, you can set --dtpye float32 instead.

Results will be stored as JSONL files in the ./localworkspace/results directory.

Each line in JSONL files is a json object with the following fields:

{
    "orig_path": str,  # the path to the raw pdf or image file
    "num_pages": int,  # the number of pages in the pdf file
    "document_text": str, # the Markdown text of the converted pdf or image file
    "page_texts": dict, # the Markdown texts of each page in the pdf file, the key is the page index and the value is the Markdown text of the page
    "fallback_pages": [int], # the page indexes that are not converted successfully
}

Generate the final Markdown files by running the following command. Generated Markdown files will be in ./localworkspace/markdowns/DOCUMENT_NAME directory.

python -m ocrflux.jsonl_to_markdown ./localworkspace

Offline Inference

You can use the inference API to directly call OCRFlux in your codes without using an online vllm server like following:

``` from vllm import LLM from ocrflux.inference import parse

file_path = 'test.pdf'

file_path = 'test.png'

llm = LLM(model="model_dir/OCRFlux-3B",g