Estimating the Expected Output of Wide Random MLPs More Efficiently Than Sampling

The authors propose a method to estimate the expected output of wide random MLPs at initialization under Gaussian inputs without sampling, approximating the activation distribution layer by layer using cumulants and Hermite expansions. Theory and experiments show that for sufficiently wide networks, it requires significantly fewer FLOPs than Monte Carlo sampling to reach a target mean squared error. The method is especially good at estimating rare event probabilities and can also be used in model training; the authors state that this provides a path to substantially reducing the probability of catastrophic tail risks in models.