Socratic LLM

Using Large Language Models (LLMs) in education presents unique challenges. Typically, LLMs are designed to provide direct answers to questions, which can hinder students' critical thinking and self-discovery skills. To address this, we focus on fine-tuning LLMs to facilitate Socratic interactions. Instead of giving straightforward answers, these models guide students to explore and find the answers themselves. We achieve this through Direct Preference Optimization (DPO). We test our approach with diverse datasets, including various educational materials and Socratic dialogues. Using advanced models like GPT-3.5 for evaluation, our results show that DPO successfully fine-tunes LLMs for Socratic dialogue, enhancing their educational value.

This repository contains the source material for the paper "Fine Tuning a Large Language Model with DPO for the Socratic method."

Model inference

You can access the model directly from HuggingFace socratic-llm.

from peft import AutoPeftModelForCausalLM, PeftConfig
import torch
from transformers import AutoTokenizer

base_model = PeftConfig.from_pretrained("giovanni-<user>-pinheiro/socratic-llm").base_model_name_or_path
model = AutoPeftModelForCausalLM.from_pretrained(
    "giovanni-<user>-pinheiro/socratic-llm",
    torch_dtype=torch.bfloat16,
    load_in_4bit=True,
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)

Check HuggingFace's transformers library for more details.

Scripts

We also make available evaluation scripts.

• self_eval.py: Perform evaluation of the LLM and prompt engineering (e.g., GPT-4o or Llama3:70b)
• eval_model.py: Perform evaluation of the finetuned model or the base model and prompt engineering only
• gen_train_dataset.py: Generates the dataset for DPO finetuning using another LLM as a judge (i.e., GPT-4o)
• train.py: Runs DPO on the base model
• human_vs_gpt.py: Use Judge model to perform evaluation of the human scored examples (validation of judge LLM)
• pipeline.py: Executes the training pipeline end-to-end (DPO dataset generation + finetuning + evaluation)

For each script, check --help for more details.

Pipeline artifacts

If you choose to run the complete pipeline, the script will generate a set of training and evaluation artifacts following the given structure:

├── training_root                                  # name to be specified by the user
│   ├── dpo                                        # DPO related files
│   │   ├── {dataset}                              # seed dataset {mathdial,tutorchat}
│   │       ├── train_dataset.json                 # Examples generated by the base model + prompt engineering then classified in choosen/rejected by the judge model
│   │       ├── weights                            # Finetuned model weights
│   │       ├── checkpoints                        # Training checkpoints
│   ├── evaluation                                 # Performance assements related files
│   │   ├── {dataset}                              # seed dataset {mathdial,tutorchat,debugging}
│   │   │   ├── from_finetuned_with_tutorchat.json # GPT-4o evaluation using model finetuned with tutorchat data 
│   │   │   ├── from_finetuned_with_mathdial.json  #    "        "       "     "       "      "   mathdial data
│   │   │   ├── base.json                          #    "        "       "     base model + prompt-engineering
│   │   │   ├── gpt4o.json                         #    "        "       "     GPT-4o + prompt-engineering
│   │   ├── human_vs_gpt.json                      # Comparison between human asssessment and judge LLM
│   ├── figures                                    # report evaluation figures

Running in Docker container

You can run any project's script from the Docker container. To do so, first build the image with

$  docker build -t socratic-llm .

Then run it with (tip: don't forget to mount the GPU and script's input/output directories). For example,

$ docker run --rm --gpus '"device=6"' -v socratic-llm-phi3/:/socractic-llm -v /home/<user>/huggingface:/huggingface -e HF_HOME=/huggingface -it socratic-llm -m pipeline --judge-llm openai <open-ai-key> gpt4o gpt-4o-2024-05-13 --output-dir /socractic-llm --instruct-model microsoft/Phi-3-mini-4k-instruct