FAR.AI proposes VLM-RM: using natural language specifications as rewards to train RL agents

FAR.AI proposes the VLM-RM method, which uses a pretrained vision-language model (specifically CLIP) as the reward model for reinforcement learning agents. A task can be specified with just a text prompt (e.g., a humanoid robot kneeling), without manually designing a reward function. CLIP provides rewards by computing the cosine similarity between the image representation and the task text description; the higher the match, the higher the reward. The authors also propose an optional goal-baseline regularization that projects the observation's CLIP embedding onto the line connecting the baseline and goal prompts, using a baseline prompt that describes the environment (e.g., a humanoid), with the projection degree controlled by the regularization strength α. Experiments used DQN and SAC to train MuJoCo humanoid robots on tasks such as kneeling, sitting cross-legged, doing the splits, and standing with raised arms. CartPole works without regularization, while MountainCar requires regularization, and more realistic textures improve the reward shape.