Online-LoRA

[WACV 2025] Official implementation of "Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation" by Xiwen Wei, Guihong Li and Radu Marculescu.

Abstract

Catastrophic forgetting is a significant challenge in online continual learning (OCL), especially for non-stationary data streams that do not have well-defined task boundaries. This challenge is exacerbated by the memory constraints and privacy concerns inherent in rehearsal buffers. To tackle catastrophic forgetting, in this paper, we introduce Online-LoRA, a novel framework for task-free OCL. Online-LoRA allows to finetune pre-trained Vision Transformer (ViT) models in real-time to address the limitations of rehearsal buffers and leverage pre-trained models' performance benefits. As the main contribution, our approach features a novel online weight regularization strategy to identify and consolidate important model parameters. Moreover, Online-LoRA leverages the training dynamics of loss values to enable the automatic recognition of the data distribution shifts. Extensive experiments across many task-free OCL scenarios and benchmark datasets (including CIFAR-100, ImageNet-R, ImageNet-S, CUB-200 and CORe50) demonstrate that Online-LoRA can be robustly adapted to various ViT architectures, while achieving better performance compared to SOTA methods.

Environment

• Python 3.9
• CUDA 11.8

conda create -n online_lora python=3.9
conda activate online_lora
pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu118
pip install timm scipy matplotlib pandas randaugment torch_optimizer

Data Preparation

1. Create a folder named local_datasets in the main directory.
2. If you are using the CoRE50 dataset, download the 128x128 version from http://bias.csr.unibo.it/maltoni/download/core50/core50_128x128.zip. Otherwise, the data will automatically download to the local_datasets folder when you run the code.

Experiment

Disjoint Class-Incremental Setting

Go to sub-directory Disjoint.

python -m torch.distributed.launch \
        --nproc_per_node=1 \
        --use_env main.py \
        $config_name \
        --model $model_name \
        --batch-size $batch_size \
        --MAS-weight $mas \
        --lr $lr \
        --loss-window-variance-threshold $var \
        --loss-window-mean-threshold $mean \
        --data-path ./local_datasets \
        --num_tasks $num_task \
        --output_dir $output_dir

Example: Split-ImageNet-Sketch with 20 tasks

python -m torch.distributed.launch \
        --nproc_per_node=1 \
        --use_env main.py \
        sketch_online_lora \
        --model vit_base_patch16_224 \
        --batch-size 64 \
        --nb-batch 1 \
        --MAS-weight 2000 \
        --lr 0.0002 \
        --num_tasks 20 \
        --loss-window-variance-threshold 0.08 \
        --loss-window-mean-threshold 5.6 \
        --data-path ./local_datasets \
        --output_dir ./output \

SI-blurry Class-Incremental Setting

(This section uses the settings from Si-Blurry repo. Please refer to their documentation for details.)

Go to sub-directory Si-blurry.

python main.py --mode $MODE \
    --dataset $DATASET \
    --n_tasks $N_TASKS --m $M --n $N \
    --rnd_seed $seed \
    --model_name $MODEL_NAME --opt_name $OPT_NAME --sched_name $SCHED_NAME \
    --lr $LR --batchsize $BATCHSIZE \
    --loss-window-mean-threshold $MEAN --loss-window-variance-threshold $VARIANCE \
    --memory_size $MEM_SIZE $GPU_TRANSFORM --transforms --online_iter $ONLINE_ITER --data_dir $DATA_DIR \
    --note $NOTE --eval_period $EVAL_PERIOD --n_worker 4 --rnd_NM

Domain-incremental Setting

Go to sub-directory Disjoint.

python -m torch.distributed.launch \
        --nproc_per_node=1 \
        --use_env main.py \
        core50_online_lora \
        --model $model_name \
        --batch-size $batch_size \
        --lr $lr \
        --data-path ./local_datasets \
        --output_dir ./output 

Reference

If our code and paper help you, please kindly cite:

@inproceedings{wei2025online,
  title={Online-lora: Task-free online continual learning via low rank adaptation},
  author={Wei, Xiwen and Li, Guihong and Marculescu, Radu},
  booktitle={2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  pages={6634--6645},
  year={2025},
  organization={IEEE}
}

Acknowledgement

Our implementation is based on Si-Blurry , MeLo, and l2p-pytorch. We highly appreciate their contributions to this community.