Researchers have proposed a new method called fixed-point flow matching (FFM) for reinforcement learning (RL) in a preprint posted to arXiv on October 12.
The method aims to address challenges in generating samples from complex, multimodal Boltzmann distributions in maximum entropy and behavior-constrained RL. Traditional flow policies rely on access to training data, which is often unavailable in online RL. FFM offers a data-free procedure for training and fine-tuning flow policies by leveraging the target flow identity to formulate the actor update as a fixed-point iteration. The authors report that the optimal policy is a fixed point of this iteration.
According to the preprint, extensive experiments demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. The authors claim that FFM establishes a principled approach for learning expressive flow-based policies without requiring training data. The preprint has not been peer-reviewed, and the authors' potential commercial interests are not assessed.