8
$\begingroup$

In his "Deep learning with Python" book, Francois Chollet says that "With neural networks, it's safe to input missing values as 0, with the condition that 0 isn't already a meaningful value. The network will learn from exposure to the data that the value 0 means missing data and will start ignoring the value."

As I know, the input value 0 has not any effect in the neural network, since it cancels the corresponding weight after multiplication. So,

Q1. Why Chollet says that the network will learn to ignore the value? I think it is not necessary to learn; the network will ignore values 0 without learning!

Q2. Why he says that "with the condition that 0 isn't already a meaningful value. As I said, the value 0 cannot carry any meaning, because it has not any effect in the network.

$\endgroup$
2
  • $\begingroup$ A neural network is just a function $f: R \rightarrow R$, and it does not have to be the case that $f(0) =0$. Nodes in the net can have biasing weights to ensure this. $\endgroup$
    – Olivier
    Jan 24, 2018 at 14:08
  • $\begingroup$ @Olivier But input nodes don't have biases - each of their output values can only be the input value times a weight. So those specific "synapse" functions are such that $f(0) = 0$ $\endgroup$
    – DHW
    Sep 10, 2018 at 15:16

2 Answers 2

6
$\begingroup$

You should not confuse a zero value in a node in the input layer (can very well affect the connected nodes to it) with a connection with weight zero (does not affect the node at the end of the connection). He is talking about the former and you seem to have misunderstood it as the second thing.

Since with enough layers and nodes a neural network can approximate arbitrarily complex transformations, it can eventually figure out that zeros should be treated totally differently than any other values. In that sense the statement is right, if no observed value can ever be zero. Of course you could use any other value that cannot ever occur in the data.

Of course, there may be the question of whether this is the most efficient approach and if you could do some quite good imputation first, that may be a good idea if your training set is quite small. E.g. the neural net may need a lot of training data to learn how to deal with such missing data and it could be problematic if there is no or hardly any missing data in the training data, but then the neural net encounters this in practice. In that case things could go severly wrong - e.g. if the neural net has learnt a somewhat linear relationship between e.g. property size (input) and property value (what we are trying to predict) and in the training data property size was always at least 200 square meters, then I would not want to know what a neural net would predict if you code an unknown property size as 0 (perhaps close to zero value for the property, perhaps a negative value perhaps some quite low positive number, who knows...). On the other hand, if you training data is massively large and any possible missingness occurs in a good number of cases, then this may well no longer be an issue.

$\endgroup$
3
  • 1
    $\begingroup$ Thanks. I know that he is talking about a zero in the input layer, but I mean this input has not any effect to the next layer. That is, if $x_i=0$, it cancels whatever weight which is multiplied to it. $\endgroup$
    – Hossein
    Jan 24, 2018 at 13:38
  • $\begingroup$ Does that not depend on your activiation function? But, even if it is multiply input times weight of connection, then it's not a problem in a sense. In that case the network with that node not activated at all is the one for when the value is missing. That would be a bit like a logistic regression, where all covariates are zero so that the only thing that remains is the intercept, which should then be correctly calibrated for that situation. $\endgroup$
    – Björn
    Jan 24, 2018 at 13:55
  • $\begingroup$ It's important to note that zero values may not carry any weight, but including observations with some zero values on some variables in the training set will inevitably affect the weights that the net determines are optimal for the nonzero values. Think of it like a dummy variable in regression: zero values always result in a zero no matter the coefficient, but adding new observations with zero values may change the computed coefficient just like including new observations with nonzero values for that variable. This, ultimately, is why it's important that zero values have no other meaning. $\endgroup$
    – DHW
    Sep 10, 2018 at 15:23
1
$\begingroup$

In fact the OP intuition is right, an input value of exactly zero prevents the network from learning something from it. The solution is to be found in the algorithm of gradient retropropagation. The delta between a previous weight and the new weight is a multiplication of several terms that depends on the learning algorithm you use. One of its terms is always the value of the previous neuron. Hence, if an input value is zero, all connected weight will not change at this iteration.

Lets be a little more formal here. Let w1 is an arbitrary weigth at the iteration t, we are looking to get w2, the new weight, after retropropagating the error calculated at the end of the forward pass is defined by:

f1

With xi, the output value of a neuron from the previous layer, e1 the partial error of the current neuron (that depends of the learning algorithm) and mu a learning rate. If xi is zero, there is no weight modification.

This is the classic formula of error retropropagation, obviously nowadays things get more complex because we train samples in batch and learning algorithms evolved but for sake of simplicity we can jump to the conclusion that it is perfectly safe to replace missing values by zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.