Restore original file

Shutdown service

Single task recognition for specific content type Args: input_file: Input file path output_dir: Output directory

Split projection profile: ``` ┌──┐ arr_values │ │ ┌─┐─── ┌──┐ │

s3 reader client. Args: default_prefix_without_bucket: prefix that not contains bucket bucket (str): bucket name

Check if a line ends with one or more letters followed by a hyphen. Args: line (str): The line of text to check. Returns: bool: True

Replace \ufffd, as these characters become garbled when extracted using pymupdf.

Auto configure dtype based on GPU capability

Collect a batch of requests, supports dynamic batch size

Collect a batch of requests, supports dynamic batch size

Create backup file

Find the installation path of LMDeploy library

Get the indentation of a line

Process batch requests using LMDeploy pipeline

Process batch requests using synchronous engine

Apply callable method which. Args: proc (Callable): invoke proc as follows: proc(inference_result, *args, **kwarg

Apply callable method which. Args: proc (Callable): invoke proc as follows: proc(dataset, *args, **kwargs)

Apply callable method which. Args: proc (Callable): invoke proc as follows: proc(dataset, *args, **kwargs)

Apply callable method which. Args: proc (Callable): invoke proc as follows: proc(dataset, *args, **kwargs)

Optimized async version of parse_file that breaks down processing into async chunks

Optimized async version of single_task_recognition

Calculate the proportion of the x-axis covered by the vertical projection of two blocks. Args: block1 (tuple): Coordinates of the fi

Chat with the uploaded image

Check current status of the file

Determine whether it's text PDF or OCR PDF based on PDF metadata.

Image and page length units here are pts

Returns False if 80% of pages have the same maximum image size and area exceeds 0.6 of page area, otherwise returns True

Supplementary rule: if average words per page < AVG_TEXT_LEN_THRESHOLD, not text PDF Mainly for various image collections

Judge if a page consists of narrow strips, two conditions: 1. Image width or height reaches 90% of page width or height, and long side needs

Supplementary rule: there's a type of scanned PDF that puts all scanned pages on each page, which gets deduplicated during metascan, characte

Judge if text layout is mainly vertical. Args: text_layout_per_page (list): Text layout list, each element represents text layout of

Randomly sample 10% of pages, if less than 5 pages, take all pages. Check text length on pages, if any page has text length > TEXT_LEN_THRESH

Clean backup file

Compress a json object and encode it with base64

Create file groups based on total page count limit Args: file_paths: List of file paths max_pages_per_group: Maximum tot

Create ZIP file asynchronously

Dump the file Args: file_path (str): the file path

encode raw data to base64 format.

Export a specific file as an appropriate Dataset. Args: file_index (int): Index of the file to export

Extract text from image or PDF

Automatically find the lmdeploy calibrate.py file in current environment

Convert fitz.Document to image, Then convert the image to numpy array. Args: doc (_type_): pymudoc page dpi (int, optional): rese