Skip to content
LessWrong· Nischal Mainali·Indexed by this site · Originally published· Alignment →· Its place in the scan record →

A solvable model of feature learning in nonlinear networks

A solvable model of feature learning in a nonlinear network

AI summary

The PIRAMID learning theory team uses a simplified cavity derivation to explain the solvable model of feature learning in nonlinear networks proposed by Rubin et al. The model has a two-layer trainable network learn, under Bayesian equilibrium, a target that depends on only one unknown direction and contains linear and cubic components. The theory predicts a phase transition in the hidden weight distribution, from no preferred direction to alignment with the target direction, whereby the network can represent the cubic component more strongly and the number of samples required drops substantially.

Read originallesswrong.com