github.com/ali-vilab/ACE_plus @main

++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling

<a href="https://arxiv.org/abs/2501.02487"><img src='https://img.shields.io/badge/arXiv-ACE++-red' alt='Paper PDF'></a>
<a href='https://ali-vilab.github.io/ACE_plus_page/'><img src='https://img.shields.io/badge/Project_Page-ACE++-blue' alt='Project Page'></a>
<a href='https://github.com/modelscope/scepter'><img src='https://img.shields.io/badge/Scepter-ACE++-green'></a>
<a href='https://huggingface.co/spaces/scepter-studio/ACE-Plus'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Space-orange'></a>
<a href='https://huggingface.co/ali-vilab/ACE_Plus/tree/main'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-orange'></a>
<a href='https://modelscope.cn/models/iic/ACE_Plus/summary'><img src='https://img.shields.io/badge/ModelScope-Model-purple'></a>



<strong>Chaojie Mao</strong>
·
<strong>Jingfeng Zhang</strong>
·
<strong>Yulin Pan</strong>
·
<strong>Zeyinzi Jiang</strong>
·
<strong>Zhen Han</strong>



·
<strong>Yu Liu</strong>
·
<strong>Jingren Zhou</strong>



Tongyi Lab, Alibaba Group

📚 Introduction

The original intention behind the design of ACE++ was to unify reference image generation, local editing, and controllable generation into a single framework, and to enable one model to adapt to a wider range of tasks. A more versatile model is often capable of handling more complex tasks. We have released three LoRA models for specific vertical domains and a more versatile FFT model (the performance of the FFT model declines compared to the LoRA model across various tasks). Users can flexibly utilize these models and their combinations for their own scenarios.

📢 News

• [x] [2025.01.06] Release the code and models of ACE++.
• [x] [2025.01.07] Release the demo on HuggingFace.
• [x] [2025.01.16] Release the training code for lora.
• [x] [2025.02.15] Collection of workflows in Comfyui.
• [x] [2025.02.15] Release the config for fully fine-tuning.
• [x] [2025.03.03] Release the fft model for ACE++, support more image to image tasks.

• [x] [2025.03.11] Release some comfyui workflow examples for ACE++ model.

• We sincerely apologize for the delayed responses and updates regarding ACE++ issues. Further development of the ACE model through post-training on the FLUX model must be suspended. We have identified several significant challenges in post-training on the FLUX foundation. The primary issue is the high degree of heterogeneity between the training dataset and the FLUX model, which results in highly unstable training. Moreover, FLUX-Dev is a distilled model, and the influence of its original negative prompts on its final performance is uncertain. As a result, subsequent efforts will be focused on post-training the ACE model using the Wan series of foundational models. Due to the reasons mentioned earlier, the performance of the FFT model may decline compared to the LoRA model across various tasks. Therefore, we recommend continuing to use the LoRA model to achieve better results. We provide the FFT model with the hope that it may facilitate academic exploration and research in this area.

Models

ACE++ Portrait LoRA

Portrait-consistent generation to maintain the consistency of the portrait.

Tuning Method	Input	Output	Instruction	Models
LoRA + ACE Data			"Maintain the facial features. A girl is wearing a neat police uniform and sporting a badge. She is smiling with a friendly and confident demeanor. The background is blurred, featuring a cartoon logo."

Models' scepter_path:

• ModelScope: ms://iic/ACE_Plus@portrait/xxxx.safetensors

• HuggingFace: hf://ali-vilab/ACE_Plus@portrait/xxxx.safetensors

ACE++ Subject LoRA

Subject-driven image generation task to maintain the consistency of a specific subject in different scenes.

Tuning Method	Input	Output	Instruction	Models
LoRA + ACE Data			"Display the logo in a minimalist style printed in white on a matte black ceramic coffee mug, alongside a steaming cup of coffee on a cozy cafe table."

Models' scepter_path:

• ModelScope: ms://iic/ACE_Plus@subject/xxxx.safetensors

• HuggingFace: hf://ali-vilab/ACE_Plus@subject/xxxx.safetensors

ACE++ LocalEditing LoRA

Redrawing the mask area of images while maintaining the original structural information of the edited area.

Tuning Method	Input	Output	Instruction	Models
LoRA + ACE Data			"By referencing the mask, restore a partial image from the doodle {image} that aligns with the textual explanation: "1 white old owl"."

Models' scepter_path:

• ModelScope: ms://iic/ACE_Plus@local_editing/xxxx.safetensors

• HuggingFace: hf://ali-vilab/ACE_Plus@local_editing/xxxx.safetensors

ACE++ FFT model

Fully finetuning a composite model with ACE’s data to support various editing and reference generation tasks through an instructive approach.

We introduced 64 additional channels in the channel dimension to differentiate between the repainting task and the editing task. In these channels, we place the latent representation of the pixel space from the edited image, while keeping other channels consistent with the repainting task. One issue with this approach is that it changes the input channel number of the FLUX-Fill-Dev model from 384 to 448. The specific configuration can be referenced in the configuration file.

Examples

The ACE++ model supports a wide range of downstream tasks through simple adaptations. Here are some examples.

ACE++ Model	Input Reference Image	Input Edit Image	Input Edit Mask	Output	Instruction	Function
Portrait LoRA(recommended) / FFT model					"Maintain the facial features, A girl is wearing a neat police uniform and sporting a badge. She is smiling with a friendly and confident demeanor. The background is blurred, featuring a cartoon logo."	"Character ID Consistency Generation"
Subject LoRA(recommended) / FFT model					"Display the logo in a minimalist style printed in white on a matte black ceramic coffee mug, alongside a steaming cup of coffee on a cozy cafe table."	"Subject Consistency Generation"
Subject LoRA(recommended) / FFT model					"The item is put on the table."	"Subject Consistency Editing"
Subject LoRA(recommended) / FFT model					"The logo is printed on the headphones."	"Subject Consistency Editing"
Subject LoRA(recommended) / FFT model					"The woman dresses this skirt."	"Try On"
Portrait LoRA(recommended) / FFT model					"{image}, the man faces the camera."	"Face swap"
FFT model					"{image} features a close-up of a young, furry tiger cub on a rock. The tiger, which appears to be quite young, has distinctive orange, black, and white striped fur, typical of tigers. The cub's eyes have a bright and curious expression, and its ears are perked up, indicating alertness. The cub seems to be in the act of climbing or resting on the rock. The background is a blurred grassland with trees, but the focus is on the cub, which is vividly colored while the rest of the image is in grayscale, drawing attention to the tiger's details. The photo captures a moment in the wild, depicting the charming and tenacious nature of this young tiger, as well as its typical interaction with the environment."	"Super-resolution"
FFT model