Is it possible to train a NN to avoid the features that a different neural network finds? For example, let's train a simple 1 layer CNN with 1x1 kernels on a supervised binary classification problem. Then train a CNN with 2x2 kernel so that it doesn't learn any of the features from the 1x1 network?
Phrased another way, whatever the dominant feature of the 1x1 network is, I want my 2x2 network to avoid.
For example, say we were learning MNIST with just 0, 1 digits. However, all the zeros are in black, and the ones in white. So the 1x1 CNN just learns the color "if the picture is mostly black -> 1, else 0". It doesn't actually learn the shapes, just the average color. Now if I train a 2x2 CNN, it might learn both the color and something about the shape. Can I set this color dependance to zero? 'Unlearn' this feature?