ZH | EN

DISC-LawLLM

Demo | Technical Report

DISC-LawLLM is a large language model specialized in Chinese legal domain, developed and open-sourced by Fudan University Data Intelligence and Social Computing Lab (Fudan-DISC), to provide comprehensive intelligent legal services.

We will open-source the following resources in this project: * DISC-Law-SFT dataset * DISC-LawLLM model weights * DISC-Law-Eval Benchmark

You can experience our DISC-LawLLM online.

News

[2024/10/15] 🎉 we released DISC-Law-SFT's legal Q&A part（DISC-Law-SFT-Pair-QA-released.jsonl and DISC-Law-SFT-Triplet-QA-released.jsonl）

[2024/03/15] 🎉🥳✨ Our paper "LawLLM: Intelligent Legal System with Legal Reasoning and Verifiable Retrieval" is accepted as a LONG PAPER for the Research Track at DASFAA 2024 (CCF-B). ✨

[2023/12/20] 🎉 We have evaluated DISC-LawLLM on the latest Benchmark Lawbench ，Our performance is only worse than GPT-4, surpassing GPT3.5 and all other existing LLMs in law domain.

[2023/11/20] 🎉 We have open sourced the evaluation scripts of our DISC-Law-Eval Benchmark. You can view more details here.

[2023/10/19] We have open sourced the evaluation datasets (including reference outputs) of our DISC-Law-Eval Benchmark.

[2023/09/25] DISC-LawLLM v1.0 has been officially released, with the DISC-LawLLM-13B model weights and the DISC-Law-SFT dataset made open source.

Table of Contents

• Overview
• Inference and Deployment
• Model Fine-tuning
• DISC-Law-Eval Benchmark
• Acknowledgements
• Disclaimer
• Citation
• License

Overview

DISC-LawLLM is a large language model designed to provide professional, intelligent, and comprehensive legal services. It caters to different user groups and offers assistance in various scenarios, with the following main features:

• Legal text processing abilities: DISC-LawLLM is capable of comprehending legal knowledge and generating based on legal text. Its main functionalities include information extraction, text summarization, etc. which have been fine-tuned using publicly available NLP datasets related to Chinese legal tasks and real-world legal texts.
• Legal reasoning abilities: To meet the requirements of tasks in smart legal services, DISC-LawLLM possesses specialized legal reasoning techniques. It leverages the concept of legal syllogism, a theory of legal reasoning, to effectively enhance its reasoning capabilities in Chinese legal domain.
• Compliance with Chinese legal domain knowledge: DISC-LawLLM is augmented with a retrieval module, strengthening the its ability to retrieve, comprehend, and adhere to background Chinese legal knowledge.

In addition to these features, we have made the following contributions during our research behind DISC-LawLLM:

• High-Quality training datasets and universally effective training paradigms
• Comprehensive Chinese legal model evaluation framework and evaluation datasets

Model Performance on Lawbench

DISC-LawLLM's performance on Lawbench is only worse than GPT-4, surpassing all other existing LLMs in law domain. Below is the average performance (zero-shot and one-shot) of DISC-LawLLM and other LLMs evaluated on LawBench.

Zero-shot Performance

One-shot Performance

Demonstration

Legal consultation

Agreement writing

Professional judicial tools

Examination Assistant

Law retrieval

Legal consultation with retrieval module

DISC-Law-SFT Dataset

Intelligent applications in Chinese legal domain under different scenarios often require a combination of various abilities, including legal text understanding and generation. To achieve this, we have constructed a high-quality supervised fine-tuning dataset called DISC-Law-SFT. This dataset covers different judicial application scenarios and includes a wide variety of tasks such as legal information extraction, legal judgment prediction, legal document summarization, and legal question answering. DISC-Law-SFT comprises two subsets, DISC-Law-SFT-Pair and DISC-Law-SFT-Triplet. The former aims to introduce legal reasoning abilities to the LLM, while the latter helps enhance the model's capability to utilize external legal knowledge. For more detailed information, please refer to our technical report. The distribution of the dataset is as follows:

Dataset	Task/Source	Size	Scenario
DISC-Law-SFT-Pair	Legal information extraction	32K	Legal professional assistant
Legal event detection	27K
Legal case classification	20K
Legal judgement prediction	11K
Legal case matching	8K
Legal text summarization	9K
Judicial public opinion summarization	6K
Legal question answering	93K	Legal consultation services
Legal reading comprehension	38K	Judicial examination assistant
Judicial examination	12K
DISC-Law-SFT-Triplet	Legal judgement prediction	16K	Legal professional assistant
Legal question answering	23K	Legal consultation services
General	Alpaca-GPT4	48K	General scenarios
Firefly	60K
Total	403K

We have released a total of nearly 300K training data, including both DISC-Law-SFT-Pair and DISC-Law-SFT-Triplet datasets. They are currently available from this link.

Retrieval Module

On the basis of DISC-LawLLM, we have augmented it with a retrieval module based on the open-source retrieval framework Langchain-Chatchat. Our knowledge base currently includes repositories of legal provisions, judicial documents, judicial examinations.

• The repository of legal provisions and judicial documents includes over 800 national and local Chinese laws, regulations, and provisions. It covers Constitution, Criminal Law, Administrative Procedure, Labor Law, Copyright Law, Civil Code, Patent Law, Law on the Exclusive Economic Zone and the Continental Shelf, Measures for the Election of Deputies from the Chinese People's Liberation Army to the National People's Congress and Local People's Congresses at or above the County Level, Anti-Secession Law, Regulation on the Administration of the Entry and Exit of Foreign Nationals, Provisions of State Council for Encouraging Taiwan Compatriots to Invest, Provisions on the Administration of Religious Activities of Aliens Within the Territory, and more.
• The repository for judicial examinations includes 24K problems related to Chinese legal knowledge.

In the future, we will continuously expand the knowledge base of our retrieval module. Furthermore, we will continue to explore and enhance the retrieval system of DISC-LawLLM. This may include, but is not limited to, mechanisms for joint training of the retrieval module and the LLM. If you are interested, we welcome further discussions and collaboration in this regard.

Inference and Deployment

The open-source DISC-LawLLM is fine-tuned based on Baichuan-13B-Base. Our model weights can be downloaded directly from Hugging Face, or obtained automatically from the example code below. First of all, please install the dependencies required for this project:

pip install -r requirements.txt

Python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation.utils import GenerationConfig

model_path = "ShengbinYue/DISC-LawLLM"
model = AutoModelForCausalLM.from_pretrained(
    model_path, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True
)
model.generation_config = GenerationConfig.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(
    model_path, use_fast=False, trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "生产销售假冒伪劣商品罪如何判刑？"},
]
response = model.chat(tokenizer, messages)

Command Line Tool

python cli_demo.py

Web Demo

Based on streamlit, the following command will start a web server. The console will output an address, which can be visited by entering in the browser:

streamlit run web_demo.py --server.port 8888

Deployment

The current version of DISC-LawLLM is fine-tuned based on the Baichuan-13B model, so you can refer to Baichuan-13B documentation for information on deploying int8 or int4 quantized inference and CPU deployment.

Model Fine-Tuning

Developers can fine-tune DISC-LawLLM for specialized use. To do this, you can refer to LLaMA Efficient Tuning or our DISC-MedLLM medical model, which are compatible with DISC-LawLLM for fine-tuning. Here we will take LLaMA Efficient Tuning as an example, and provide reference scripts for both full and LoRA fine-tuning.

First, download LLaMA Efficient Tuning and follow its instructions to install the required dependencies. Note that the training data should be processed in the format specified by the project. The example scripts will be given as follows.

Full Fine-Tuning

We have tested full fine-tuning under the setting of 8 * Nvidia A800 80 GB + deepspeed. The script is as follows:

deepspeed --num_gpus=8 src/train_bash.py \
    --stage sft \
    --model_name_or_path S heng bin \
    --do_train \
    --dataset alpaca_gpt4_zh \
    --template baichuan \
    --finetuning_type full \
    --output_dir path_to_your_sft_checkpoint \
    --overwrite_cache \
    --per_device_train_batch_size 4 \ 
    --per_device_eval_batch_size 4 \ 
    --gradient_accumulation_steps 8 \ 
    --preprocessing_num_workers 8 \
    --lr_scheduler_type cosine \
    --logging_steps 10 \
    --save_steps 100 \
    --eval_steps 100 \
    --learning_rate 5e-5 \
    --max_grad_norm 0.5 \
    --num_train_epochs 2.0 \
    --dev_ratio 0.01 \
    --evaluation_strategy steps \
    --load_best_model_at_end \
    --plot_loss \
    --fp16 \
    --deepspeed deepspeed.json

deep_speed.json configuration is as follows:

{
    "train_micro_batch_size_per_gpu": "auto",
    "zero_allow_untested_optimizer": true,
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "initial_scale_power": 16, 
        "loss_scale_window": 1000,
        "hysteresis": 2,
        "min_loss_scale": 1
    },  
    "zero_optimization": {
        "stage": 2,
        "allgather_partitions": true,
        "allgather_bucket_size": 5e8,
        "overlap_comm": false,
        "reduce_scatter": true,
        "reduce_bucket_size": 5e8,
        "contiguous_gradients" : true
    }
}

LoRA Fine-Tuning

We tested LoRA fine-tuning under the setting of 4 * Nvidia A800 80G GPUs. The scripts is as follows:

``` torchrun --nproc_per_node 4 src/train_bash.py \ --stage sft \ --model_name_or_path ShengbinYue/DISC-LawLLM \ --do_train \ --dataset alpaca_gpt4_zh \ --template baichuan \ --finetuning_type lora \ --lora_rank 8 \ --lora_target W_pack \ --output_dir path_to_your_sft_checkpoint \ --overwrite_cache \ --per_device_train_batch_size 4 \ --per_device_eval_batch_size 4 \ --gradient_accumulation_steps 8 \ --preprocessing_num_workers 16 \ --lr