Skip to content
LessWrong· fitandfend·Indexed by this site · Originally published· Alignment →· Its place in the scan record →

Correlation between behavior and reward is not sufficient for RL to reinforce that behavior

Correlating a behavior with reward doesn't reinforce it

AI summary

LessWrong author fitandfend argues that correlation between behavior and the reward signal is by itself not sufficient for RL to reinforce that behavior: the gradient pressure on an action logit is proportional to the reward times 1−p(action|context), and only within-group correlation after controlling for context produces gradient, i.e., in expectation RL only reinforces factors that are causally upstream of the reward. The author uses a fair coin-guessing game as an example: if the model speaks Shakespearean language only after rollouts in which it guessed correctly, under an unbiased policy gradient estimate the expected gradient is 0, and training will not push toward that behavior. On this basis, the idea that a model selectively acts in rounds where it knows success or failure and uses correlation to get RL to install that behavior does not hold according to the author's theory and experiments.

Read originallesswrong.com