Is neural network able to learn from zero inputs?

Question

In his "Deep learning with Python" book, Francois Chollet says that "With neural networks, it's safe to input missing values as 0, with the condition that 0 isn't already a meaningful value. The network will learn from exposure to the data that the value 0 means missing data and will start ignoring the value."

As I know, the input value 0 has not any effect in the neural network, since it cancels the corresponding weight after multiplication. So,

Q1. Why Chollet says that the network will learn to ignore the value? I think it is not necessary to learn; the network will ignore values 0 without learning!

Q2. Why he says that "with the condition that 0 isn't already a meaningful value. As I said, the value 0 cannot carry any meaning, because it has not any effect in the network.

Answer 1

You should not confuse a zero value in a node in the input layer (can very well affect the connected nodes to it) with a connection with weight zero (does not affect the node at the end of the connection). He is talking about the former and you seem to have misunderstood it as the second thing.

Since with enough layers and nodes a neural network can approximate arbitrarily complex transformations, it can eventually figure out that zeros should be treated totally differently than any other values. In that sense the statement is right, if no observed value can ever be zero. Of course you could use any other value that cannot ever occur in the data.

Of course, there may be the question of whether this is the most efficient approach and if you could do some quite good imputation first, that may be a good idea if your training set is quite small. E.g. the neural net may need a lot of training data to learn how to deal with such missing data and it could be problematic if there is no or hardly any missing data in the training data, but then the neural net encounters this in practice. In that case things could go severly wrong - e.g. if the neural net has learnt a somewhat linear relationship between e.g. property size (input) and property value (what we are trying to predict) and in the training data property size was always at least 200 square meters, then I would not want to know what a neural net would predict if you code an unknown property size as 0 (perhaps close to zero value for the property, perhaps a negative value perhaps some quite low positive number, who knows...). On the other hand, if you training data is massively large and any possible missingness occurs in a good number of cases, then this may well no longer be an issue.

Answer 2

In fact the OP intuition is right, an input value of exactly zero prevents the network from learning something from it. The solution is to be found in the algorithm of gradient retropropagation. The delta between a previous weight and the new weight is a multiplication of several terms that depends on the learning algorithm you use. One of its terms is always the value of the previous neuron. Hence, if an input value is zero, all connected weight will not change at this iteration.

Lets be a little more formal here. Let is an arbitrary weigth at the iteration t, we are looking to get , the new weight, after retropropagating the error calculated at the end of the forward pass is defined by:

With , the output value of a neuron from the previous layer, the partial error of the current neuron (that depends of the learning algorithm) and a learning rate. If is zero, there is no weight modification.

This is the classic formula of error retropropagation, obviously nowadays things get more complex because we train samples in batch and learning algorithms evolved but for sake of simplicity we can jump to the conclusion that it is perfectly safe to replace missing values by zero.