Log error to console and potentially to a database in the future

Report progress if callback is provided

Create a MinHash object from text

Load QA pairs from a JSON file

Validate API key configuration

Clean LaTeX text, preserving titles and bold content

Clean result text, extract LaTeX content. Args: text (str): Original text. Returns: str: Cleaned La

Convert LaTeX file to QA pairs and save

Convert parsed file content to LaTeX format

Run the deduplication process

Filter invalid text, rules to be added

Call the model based on prompt and API keys :param text: Input text :param API_KEY: API key :param SECRET_KEY: Secret key :param

Get quality file content based on filename

Get text between titles, i.e., text between two LaTeX section headings

File parsing function, supports progress callback :param hparams: Hyperparameters :param progress_callback: Progress callback function

PDF file parsing, supports progress callback

Process QA file to deduplicate QA pairs. Args: hparams: Deduplication parameters Returns: Tuple con

Run both main processing and deduplication

Initialize the QA deduplication class. Args: hparams: Deduplication parameters progress_callback: Optional c

Additional setup after initialization

Generate output file paths

Get deduplicated file content based on file ID

Get deduplicated file content based on filename

Get the most recent deduplication history record

Get quality file content based on file ID

Process a single text chunk and update progress

Process a single text chunk and update progress

Process LaTeX chunk and generate QA pairs :param latex: LaTeX text :return: List of QA pairs

Text document parsing, which is essentially making a copy of the file :param file_path: Original file path :param save_path: Path for the

Update progress (with throttling control)