Decoding dataset bias in chemistry neural network models with attribution methods

Using attribution methods, researchers dissected a deep neural network that predicts molecule-protein target binding and found that even when the model achieves perfect accuracy on the held-out test set of a synthetic dataset, it still learns spurious associations due to dataset bias. The authors exploited this brittleness to construct adversarial examples that fool the model, and noted that dataset bias prevents the model from reliably revealing protein-ligand binding mechanisms. Based on this, the authors propose a test for dataset bias; if the model fails it, the model needs to be simplified or regularized and the training data expanded.