Accelerating Style Transfer: Enhancing Efficiency of Diffusion-Based Models with Advanced Sampling Methods

Abstract

Traditionally, generating images with diffusion-based style transfer models has been time-consuming. This study explores how recent advancements in accelerated sampling methods can expedite the image generation process for top-performing style transfer model. Style Injection in Diffusion (StyleID) renowned for its exceptional style transfer results, serves as the foundation for this research. The study integrates various accelerated sampling techniques—Fast Sampling of Diffusion Probabilistic Models (FastDPM), Pseudo Numerical Methods for Diffusion Models (PNDMs), DPM-Solver, and Unified Predictor-Corrector (UniPC) —into the StyleID framework. While the core techniques of StyleID remain unchanged, these accelerated methods are applied to both the inverse and reverse diffusion processes. The findings reveal that UniPC can effectively reduce the Number of Function Evaluations (NFE) from 50, as used by Denoising Diffusion Implicit Models (DDIM) in StyleID, to just 10, while maintaining the quality of the stylized images. Additionally, within the 10-50 NFE range, DPM-Solver and PNDMs yield superior results compared to DDIM. Conversely, FastDPM does not enhance the efficiency or quality of image generation at any NFE. Consequently, the use of UniPC in the sampling method results in a fivefold increase in efficiency for producing high-quality stylized images.