# What will a neural network respond to a never-seen feature?

If I want to "shut off" one of the input neurons of an ANN at training time and then turn it back on at inference time, what will happen?

Let's say you want to do ablation study by setting every single feature X value to 0 in the dataset at training time. If at inference time I give (back) this feature, what would be the response from the neural network?

Will it perform as well as if I also set all feature X to 0 at inference time, or performing worse due to confusion on this added feature?

Does the answer change if different training method (specifically optimizer, like Adam optimizer or SGD) is used?

• Neural networks learn about how the inputs are related to the outputs. Setting all of the inputs to 0 at test time tells you what the predicted class is for all 0 inputs. Why would you do this? What would you expect to learn?
– Sycorax
Jun 15 '18 at 17:30
• @Sycorax It is definitely not intended; I was wondering, whether giving a feature at inference time that is not available to the model at training time on invalidates the claim that "The model is not taking that feature at inference time". Jun 15 '18 at 18:08
• Neural networks can only learn from data that's available to them. If you don't give the NN information during training, it will not be useful at test time. What specifically happens depends on the particulars of the experiment.
– Sycorax
Jun 15 '18 at 18:10

If I understand your question correctly, you want to "shut off" one of the input neurons of a ANN at training time and then turn it back on at inference time correct?

Well, this question can be answered by simply understanding how a ANN works. Let's just use a simple dense feed forward network as an example. This neural network will take each input feature vector multiply it with a set of weights, add some biases, and then compute the 2nd layer activations. It then takes those 2nd layer activations and does the same thing to get the 3rd layer activations and so on and so forth until the final output activations are calculated. All neurons in the 2nd layer are connected to every neuron in the first layer with a specific weight $w_{ij}$ where $i$ denotes the first layer's neuron and $j$ denotes the second layer's neuron (let's ignore biases for now). This weight matrix is generally initialized independently of the inputs.

So what happens if you set one of the features to 0? Well you'll have a set of weights $w_{\hat{i}j}$ (where $\hat{i}$ denotes the specific feature you've "turned off") that will never be updated. Why will these weights never be updated? Well because the feature is always 0 so backprop will always find 0 gradient with respect to these weights. I will not show a mathematical proof here, but it's very simple to understand why this is the case. No matter what all the $w_{\hat{i}j}$ are the loss function will never be affected because they will always be multiplied by a 0 and as such there is never a gradient with respect to this set of weights.

So at inference time, what you have is a set of weights $w_{\hat{i}j}$ which are untrained. Now, if you initialized these weights to 0 when you began training, nothing would change. They would stay 0 throughout training, and it would be like if that feature never existed. Even if you put that feature back in during inference time, it will always be multiplied by a set of 0's and so it won't matter. However, if you randomly initialized $w_{\hat{i}j}$ as would be generally the case (you don't want to initialize all weight in a neural network to 0), what adding the feature back in at inference time will do is add a bit of random statistical noise to your neural network. The size of this effect depends on details of how the weights were initialized and how strongly your neural network's outputs depend on those weights. Since generally you initialize the weights randomly, there shouldn't be any bias to this statistical noise, it should just be like adding a little bit of white noise to your network. However, one could conceive of a situation where one maliciously initializes weights so that bias is introduced...

It's a bit hard to understand the reasoning behind why anyone would want to do this.

If you set a feature to 0 or to any constant at training time, the network will not learn anything for that feature. A constant feature has no relationship to the class label.

The optimizer should not matter, because the loss would not be affected by changing the weight associated with a constant feature.

So if you "unmask" for inference, or allow feature X to be what it was before, which might have a relationship with the class, it would just effectively ignore that feature, or worse.

• (+1) In other words, neural networks aren't magic.
– Sycorax
Jun 15 '18 at 17:33
• I don't believe it is the case that the network will learn a weight of 0 for that feature. Why would it? It should just not update the weights for that feature at all. Jun 15 '18 at 17:36
• Also, if you set the feature X to a constant other than 0, then effectively that feature turns into a bias unit (indeed, that's how the name "bias unit" came about as some neural networks had a neuron that acted like a bias). Which could actually be something the neural network learns. Changing a bias unit's behavior doing inference time might have some unintended consequences. Jun 15 '18 at 17:50
• @enumaris you're right -- and a constant feature might then be incorporated into the bias, so during inference exposing the network to the real feature could mess up the contribution of the bias Jun 15 '18 at 18:14