Neural networks are not yet clean

LessWrong user leogao wrote a rebuttal to Nick Cammarata's judgment that neuron polysemanticity is overestimated, arguing that activation examples matching a pattern does not mean the neuron has been explained. Using a neuron associated with both dogs and bicycles as an example, the article points out that top activation examples may only correspond to some subset that is difficult to characterize compactly, and the reverse relationship does not necessarily hold. The author advocates instead using downstream KL and computing simulation scores on large datasets: if replacing the neuron with a simulated value significantly degrades model performance, the explanation does not hold, and this criterion can also flag bad explanations previously treated as polysemantic explanations. The author speculates that ablating weak activations would also significantly degrade model performance, so weak activations should not be completely ignored. The author also believes that explaining neurons is only part of understanding network computation; concepts are stored in a superimposed manner, making circuit research still unable to explain most of the variance in model behavior.