PaperSummary12 : Orthogonal Adaptation for modular customization of diffusion models

The paper introduces “Orthogonal Adaptation” a new approach for modular customization of text-to-image diffusion models. Traditional methods for customizing diffusion models struggle with scalability and maintaining fidelity when merging multiple concepts. This work addresses these challenges, enabling independent fine-tuning of models for individual concepts and efficient merging during inference.

The key concepts are:

1. Fine-Tuning: It implements a constrained low-rank (LoRA) where only specific parameters are optimized while ensuring orthogonality.
2. Orthogonality Constraint: It ensures the weight residuals of fine-tuned models remain orthogonal reducing “crosstalk” (loss of concept identity).
3. Merging: Independently fine-tuned models are combined via simple summation, requiring no additional computation for joint synthesis.

By disentangling customized concepts into orthogonal directions, the method ensures efficiency , fidelity during merging and synthesis, advancing the usability of text-to-image diffusion models.

References:

Orthogonal Adaptation for Modular Customization of Diffusion Models

Ortha - a Hugging Face Space by ujin-song

Orthogonal Adaptation for Modular Customization of Diffusion Models