Fine-tuning Flow and Diffusion Generative Models | Carles Domingo-Enrich
Portal is the home of the AI for drug discovery community. Join for more details on this talk and to connect with the speakers: https://portal.valencelabs.com/logg
Dynamical generative models that produce samples through an iterative process, such as Flow Matching and denoising diffusion models, have seen widespread use, but there has not been many theoretically-sound methods for improving these models with reward fine-tuning. In this work, we cast reward fine-tuning as stochastic optimal control (SOC). Critically, we prove that a very specific memoryless noise schedule must be enforced during fine-tuning, in order to account for the dependency between the noise variable and the generated samples. We also propose a new algorithm named Adjoint Matching which outperforms existing SOC algorithms, by casting SOC problems as a regression problem. We find that our approach significantly improves over existing methods for reward fine-tuning, achieving better consistency, realism, and generalization to unseen human preference reward models, while retaining sample diversity.
Paper: https://arxiv.org/abs/2409.08861
Speaker: Carles Domingo-Enrich
Twitter Hannes: @hannesstaerk
Twitter Dominique: @dom_beaini