A solvable model of feature learning in nonlinear networks
A solvable model of feature learning in a nonlinear network
AI summary
The PIRAMID learning theory team uses a simplified cavity derivation to explain the solvable model of feature learning in nonlinear networks proposed by Rubin et al. The model has a two-layer trainable network learn, under Bayesian equilibrium, a target that depends on only one unknown direction and contains linear and cubic components. The theory predicts a phase transition in the hidden weight distribution, from no preferred direction to alignment with the target direction, whereby the network can represent the cubic component more strongly and the number of samples required drops substantially.