LLM-Kit

LLM-Kit is a powerful toolkit for text processing, QA pair generation, quality control, and deduplication. It provides a complete set of API interfaces accessible through the FastAPI framework.

Features

• Support for text parsing in multiple file formats
• Automatic text to LaTeX conversion
• High-quality QA pair generation based on large language models
• Intelligent quality control and optimization system
• Efficient QA pair deduplication algorithm
• Complete REST API interface
• Detailed error logging system
• Support for asynchronous processing and parallel computing

Installation

1. Clone the repository:

git clone https://github.com/Saberlve/LLM-Kit.git
cd LLM-Kit

1. Install dependencies:

pip install -r requirements.txt

1. Configure MongoDB database:
2. Ensure MongoDB service is running
3. Default connection address: mongodb://localhost:27017
4. Default database name: llm_kit

Module Description

1. Text Parsing Module (text_parse)

• Supports parsing of multiple file formats (txt, pdf, etc.)
• Provides text to LaTeX conversion functionality
• Supports OCR functionality
• Automatically saves parsing records

2. QA Pair Generation Module (generate_qas)

• Generates high-quality QA pairs based on parsed text
• Supports multiple large language models (such as Ernie)
• Supports parallel processing for improved efficiency
• Supports custom domain QA generation

3. Quality Control Module (quality_control)

3. Quality Control Module (quality_control)

• Evaluates QA pair quality
• Optimizes low-quality QA pairs
• Supports customizable similarity and coverage thresholds
• Provides detailed quality assessment records

4. Deduplication Module (deduplication)

4. Deduplication Module (deduplication)

• Detects and removes duplicate QA pairs
• Supports deduplication strategies based on questions or answers
• Configurable deduplication thresholds
• Retains highest quality QA pairs

Configuration Guide

Configuration files are located in the hparams directory:

config.yaml

file_path: Input file path
save_path: Results save path
AK: [API key list]
SK: [Corresponding secret keys]
model_name: Model name to use
parallel_num: Number of parallel processes
convert_to_tex: Whether to convert to LaTeX
similarity_rate: Similarity threshold
coverage_rate: Coverage threshold
max_attempts: Maximum retry attempts
domain: Domain setting

dedup.yaml

• Contains deduplication-related configuration parameters
• Configurable deduplication strategy and thresholds

API Interface

Main Endpoints

1. Text Parsing Related
2. POST /parse/upload: Upload file
3. POST /parse: Parse file

4. GET /parse/files/all: Get all uploaded files

5. LaTeX Conversion Related

6. POST /to_tex: Convert to LaTeX format

7. QA Generation Related

8. POST /qa/generate_qa: Generate QA pairs

9. GET /qa/generate_qa/history: Get generation history

10. Quality Control Related

11. POST /quality: Evaluate and optimize QA pairs

12. GET /quality/history: Get quality control history

13. Deduplication Related

14. POST /dedup/deduplicate_qa: Execute QA pair deduplication
15. GET /dedup/deduplicate_qa/history: Get deduplication history

Error Handling

• All interfaces have a unified error handling mechanism
• Error logs can be viewed via GET /error-logs endpoint
• Supports detailed error tracking and recording

Notes

1. Before using, please ensure:
2. MongoDB service is properly started
3. API keys are correctly set in the configuration file

4. Sufficient disk space is available for storing generated files

5. Performance Optimization Suggestions:

6. Set an appropriate number of parallel processes
7. Adjust similarity and coverage thresholds according to actual needs
8. Regularly clean up unnecessary historical records