Check if block text ends with colon for relevant block types.

Routes the send_message call to the appropriate processor. The underlying processor's send_message method is responsible for handling

Calculate statistics about processed questions.

Create a reference ID for answer details and store the details

Helper method to get unique filepath

Helper method to process text references and create content items. Args: ref_num (int): Reference number for the text ite

Append a json payload to the end of a jsonl file.

Process a single report, returning a list of processed pages and printing a message if corrections were made.

Transform blocks according to formatting rules.

Return prompts tuple for the given schema.

Clean text using regex substitutions and count corrections.

Checks if subset.json exists in root dir and subset.csv is absent. If so, converts the JSON to CSV format.

Creates and returns a DocumentConverter with default pipeline options.

Extract company names from a question by matching against companies in the subset file.

Remove blocks of ignored types.

Format vector retrieval results into RAG context string

Wrapper for generate_content with retry logic

Returns the next available filename by adding a numbered suffix if the file exists. Example: If answers.json exists, returns answers_

Returns page dict for given page number, or None if not found.

Convert serialized table format to text string. Args: table: Table object containing serialized data

Group serialized tables by page number

Get table representation by ID from report data. Returns markdown or serialized text based on configuration.

Handle errors during question processing. Log error details and return a dictionary containing error information.

Initialize paths configuration based on run config settings

Ensure that page numbers in content are sequential by filling gaps with empty pages.

Parse CSV file and create a lookup dictionary with sha1 as key.

Post-process answers for submission format: 1. Convert page indices from one-based to zero-based 2. Clear references for N/A

Render list group with optional header, text and footnotes.

Render table group with optional header, text and footnotes.

Reparse invalid JSON responses using the model itself.

Split page text into chunks. The original text includes markdown tables.

Split report into chunks, preserving markdown tables in content and optionally including serialized tables.

Validate that all page numbers mentioned in the LLM's answer are actually from the retrieval results. If fewer than min_pages valid r

Extract the API endpoint from the request URL.

Assemble the report using either normalized data or raw conversion result.

Process all tables in the report asynchronously

Calls the OpenAI API and saves results.

Split processed reports into smaller chunks for better processing.

Pipeline command line interface for processing PDF reports and questions.

Create a BM25 index from a list of text chunks.

Create vector databases from chunked reports.

Export processed reports to markdown format for review.

Export processed reports to markdown files. Args: reports_dir: Directory containing JSON report files output_

Use LLM to break down a comparative question into individual questions.

Merge complex JSON reports into a simpler structure with a list of pages, where all text blocks are combined into a single string.

Count the number of tokens in the request. Only supports completion and embedding requests.

Parse PDF files in parallel using multiple processes. Args: input_doc_paths: List of paths to PDF files to process

Parse PDF reports in parallel using multiple processes. Args: chunk_size: Number of PDFs to process in each worker

Main method to process page blocks and return assembled string.

Processes API requests in parallel, throttling to stay under rate limits.

Process a question involving multiple companies in parallel: 1. Rephrase the comparative question into individual questions 2

Process JSON files in parallel using thread pool. Args: input_dir: Path to directory containing JSON files ma

Process already parsed PDF reports through the pipeline: 1. Merge to simpler JSON structure 2. Export to markdown 3. Chunk the

Process reports from a directory or list of paths, returning a list of processed reports and saving them to an output directory if s

Process all reports and save individual BM25 indices. Args: all_reports_dir (Path): Directory containing the JSON report

Rerank multiple documents using parallel processing with threading. Combines vector similarity and LLM relevance scores using weighte