Training a neural network for regression always predicts the mean

I am training a simple convolutional neural network for regression, where the task is to predict the (x,y) location of a box in an image, e.g.:

The output of the network has two nodes, one for x, and one for y. The rest of the network is a standard convolutional neural network. The loss is a standard mean squared error between the predicted position of the box, and the ground truth position. I am training on 10000 of these images, and validating on 2000.

The problem I am having, is that even after significant training, the loss does not really decrease. After observing the output of the network, I notice that the network tends to output values close to zero, for both output nodes. As such, the prediction of the box's location is always the centre of the image. There is some deviation in the predictions, but always around zero. Below shows the loss:

I have run this for many more epochs than shown in this graph, and the loss still never decreases. Interestingly here, the loss actually increases at one point.

So, it seems that the network is just predicting the average of the training data, rather than learning a good fit. Any ideas on why this may be? I am using Adam as the optimizer, with an initial learning rate of 0.01, and relus as activations

If you are interested in some of my code (Keras), it is below:

# Create the model
model = Sequential()
model.add(Convolution2D(32, 5, 5, border_mode='same', subsample=(2, 2), activation='relu', input_shape=(3, image_width, image_height)))
model.add(Convolution2D(64, 5, 5, border_mode='same', subsample=(2, 2), activation='relu'))
model.add(Convolution2D(128, 5, 5, border_mode='same', subsample=(2, 2), activation='relu'))

# Compile the model

# Fit the model
model.fit(images, targets, batch_size=128, nb_epoch=1000, verbose=1, callbacks=[plot_callback], validation_split=0.2, shuffle=True)

• Are the images on top examples of your actual samples? Is that 5 separate samples? There appears to be no information in the images that would help generalize. I mean, you don't need a neural net to find the x,y location of the white square, you can just parse the image and look for a white pixel. Explain a bit more about your vision for this model. Is there some temporal pattern, whereby you are predicting the next location? – photox Feb 14 '17 at 2:06
• Hi, and yes, the images are 5 separate samples. I'm not sure how they are rendered for you, but they should be 5 individual square images (I've changed the layout a little to help...). Yes, I realise that you don't need a neural network for this task, but it is just a test experiment to help me learn how to do regression with a neural network. I don't understand what you mean by there being no information to help generalize.... Each training pair consists of a square image, and a two-dimensional vector of the (x, y) location of the square. Thanks :) – Karnivaurus Feb 14 '17 at 11:57
• 1) Your input shape on the first conv layer is using 3 (rbg) channels, but your data are greyscale (1 channel) 2) You don't need that many conv layers and filters, in fact I think a single layer, and a handful of small kernels will be fine. – photox Feb 14 '17 at 12:04
• Are you sure that images do indeed correspond the targets? – user31264 Feb 14 '17 at 12:18
• Like @photox says, you do not need the conv layers. Adding these make it more difficult for the optimizer to find a good solution. If you remove the 3 conv layers I suspect your "model" will work. – Pieter Feb 14 '17 at 22:02

The optimizer is unable to converge to a (sub-)optimal solution. Why? Your problem is too easy and/or your model is too complex.

Too easy problem

As @photox already said this problem can be solved with only a single hidden layer. I even suspect that it can be done without a hidden layer. That is because this problem is linear seperable.

Let me illustrate this. Imagine a neural network without hidden layers and a linear activation function (you might also call it linear regression). To compute the x-location of the square each pixel is connected to the x-output. The first column of pixels is connected with weight $$1/\text{width}$$. The second column is connected with weight $$2/\text{width}$$. This continues until the last column (e.g. column $$n$$) which is connected with weight $$n/\text{width}$$. Since some pixels are non-zero the activation of the output is linear to the x-coordinate of the square. Hence, a linear function can compute the location of the square.

There are several solutions:

• Choose a harder problem, e.g. image classification
• Add noise, e.g. salt-and-pepper or white-noise
• Make problem harder, e.g. by predicting the location of a red square while there are lots of differently colored circles in the background

Too complex model

Your model has a few parts that add a lot of complexity while not helping the optimizer to find a sweet optimum.

For example, the convolutional layers. The first layer has 32 convolutional filters of size $$5 \times 5$$. What do you expect these filters to learn? In image classification these filters learn to detect edges, corners, gradients and blobs. But in this case there are only few filters that make sense. I can think of edge from left-to-right and vice-versa and from top-to-bottom and vice-versa. In your case there are thus approx 28 filters that just add random noise. Deleting these (or just the whole layer) makes it a lot easier for the optimizer to find an optimum that works.

Another example is the Adam optimizer with a lot of extra parameters. The Adam optimizer might work well with these parameters but why don't you just start with a simple SGD optimizer with default values.

Thus you can make several optimizations:

• use LinearRegression from scikit-learn. OK, this is not what you want but I just wanna illustrate how overly complex this model is.
• Remove the conv layers
• Decrease the size of the hidden Dense layers
• Use the default SGD optimizer
• If you are using a hidden layer, you should try a sigmoid activation. You can think of each of the nodes of the hidden layer as detecting if a square is at a certain location.
• If this all does not work, experiment with the learning rate a bit to find out if it is too high or too low.

Ps

I think you will like this blogpost by Adit Deshpande.

• Let me know if these solutions did change the optimizers behavior. – Pieter Feb 14 '17 at 22:55
• Thanks, that's very useful. I'm working on trying your suggestions. However, I don't understand your first point. I'm confused as to why, if the problem is too simple, then it is harder to optimize than a more complex problem. For a given network, why would a simpler problem be harder to optimize than a more complex problem? In a simple problem, I would have thought that there would be very strong gradients and a strong global optimum. But your first point says that the simplicity of the problem makes optimization hard, suggesting that a more complex problem would help the optimization... – Karnivaurus Feb 15 '17 at 2:41
• I don't buy either of these as a cause for this problem. A "complex" model should be more capable of overfitting and therefore shouldn't struggle to fit the training data. A "easy" problem shouldn't make it harder to solve either..."easy" is a relative term anyway...maybe you mean to say that his neural network architecture isn't able to capture the signal in his data, even though a "simpler" but fundamentally different model could. – JacKeown May 20 '20 at 3:06
• > "I'm confused as to why, if the problem is too simple, then it is harder to optimize than a more complex problem." Because a complex model has a lot of modeling capacity that it does not use. This overcapacity introduces noise and makes it harder to find the real signal. – Pieter May 21 '20 at 7:02
• > "I don't buy either of these as a cause for this problem" - Looking back at the answer, they are written from a practice point-of-view. If your complex model does not converge properly, one of the ways to debug it is to use a simpler model. – Pieter May 21 '20 at 7:06

I am facing the same problem with my data set. It turns out that in my case the predictors are highly concentrated with a very small variance. You should check out the variance of your prediction variables and see how it is distributed.

However, some transformations on the output variable can be performed to modify or change its scale. This might result in a more uniform type distribution. For example, in image recognition tasks histogram equalization or contrast enhancement sometimes works in the favor of correct decision making.

Having the same issue, my data is also "imbalanced" in the sense that outputs are highly concentrated (as pointed out by Ravi Shankar). Not sure, if this is the real problem. In my case, more complex models get the training error down (to 0), but the test error remains about the standard deviation, ie. at test time the prediction is essentially no better than using the mean. Thus, overfitting is a concern. At this point, I am not sure, if regularization or other tricks actually help or whether some more severe model changes are needed.

Remark: Pieter's answer is interesting, but I agree with JacKeown's comment that it might need a few clarifications.

Here is a more information-theoretic reason for why this is happening. Let $$Y$$ be the true and correct output that your network should return (the target), and let $$\hat{Y}$$ be the output that your network actually returns. The problem that you observe is that $$\hat{Y}=K$$, where $$K$$ is some constant, which means that $$Y$$ is independent of $$\hat{Y}$$.

If we compute the mutual information between $$Y$$ and $$\hat{Y}$$, we observe that: $$I(Y;\hat{Y}) = h(Y) - h(Y|\hat{Y}) = h(Y) - h(Y) = 0$$ Where $$h(\cdot)$$ is the differential entropy and $$h(Y|\hat{Y})=h(Y)$$ since $$Y$$ and $$\hat{Y}$$ are independent. In other words, if we assume that $$Y \rightarrow \hat{Y}$$ is a communications channel, then there is no information flowing between the input of the channel $$Y$$ and the output of the channel $$\hat{Y}$$!

A more accurate channel would be the Markov chain $$Y \rightarrow X \rightarrow \hat{Y}$$, where $$X$$ is the input to the network. However, by the data processing inequality: $$I(Y;\hat{Y}) \leq I(Y;X) \\ I(Y;\hat{Y}) \leq I(X;\hat{Y})$$ Which are just upper-bounds on the mutual information $$I(Y;\hat{Y})$$. Our objective is to increase the mutual information $$I(Y;\hat{Y})$$, which can be done by increasing a lower-bound on $$I(Y;\hat{Y})$$. So, we are not interested in $$I(Y;X)$$ and $$I(X;\hat{Y})$$ at this time.

Now let the mean-squared error between $$Y$$ and $$\hat{Y}$$ be: $$\mathbb{E}\left[(Y-\hat{Y})^2\right] = \delta$$ Now, to find a lower-bound on $$I(Y;\hat{Y})$$, note that: \begin{align} I(Y;\hat{Y}) &= h(Y) - h(Y|\hat{Y}) \\ &= h(Y) - h(Y-\hat{Y}|\hat{Y}) \tag{1} \\ &\geq h(Y) - h(Y-\hat{Y}) \tag{2} \\ &\geq h(Y) - \frac{1}{2} \log{(2\pi e \delta)} \tag{3} \end{align} Where:

• $$(1)$$ is because differential entropy is translation invariant.
• $$(2)$$ is because conditioning reduces entropy.
• $$(3)$$ is because random variable $$Z=Y-\hat{Y}$$ has variance $$\mathbb{E}[(Z-\mathbb{E}[Z])^2]=\mathbb{E}\left[(Y-\hat{Y})^2\right] = \delta$$, and so this is the Gaussian bound on differential entropy.

So: $$I(Y;\hat{Y}) \geq h(Y) - \frac{1}{2} \log{(2\pi e \delta)}$$ If we further assume that the target $$Y$$ has mean $$\mathbb{E}[Y]=0$$ and variance $$\text{Var}(Y)=\sigma^2$$, then we get the tighter lower-bound: $$I(Y;\hat{Y}) \geq \frac{1}{2} \log{(2\pi e \sigma^2)} - \frac{1}{2} \log{(2\pi e \delta)} \geq h(Y) - \frac{1}{2} \log{(2\pi e \delta)}$$ And so we can increase the mutual information by either increasing the variance $$\sigma^2$$ of $$Y$$ or decreasing the mean squared-error $$\delta$$. Unfortunately, we do not usually have the ability to adjust the variance of $$Y$$, so we would need to resort to decreasing $$\delta$$.

Interestingly, we have just shown that minimizing the mean squared-error $$\delta$$ corresponds to maximizing the mutual information between $$Y$$ and $$\hat{Y}$$, and vice-versa. At first glance, it seems like there is a contradiction here: you clearly can achieve small mean squared-error values, which means that the mutual information between $$Y$$ and $$\hat{Y}$$ is maximized. However, what you practically observe is that the mutual information between $$Y$$ and $$\hat{Y}$$ is $$0$$.

To resolve this contradiction, it is helpful to take a closer look at the mean squared-error between $$Y$$ and $$\hat{Y}$$. We know that: $$\mathbb{E}\left[(Y-\hat{Y})^2\right] = \text{Var}(\hat{Y}) + \text{Bias}(\hat{Y},Y)^2$$ Since both of these terms are non-negative, then the variance and bias of $$\hat{Y}$$ each individually form lower-bounds for the mean squared-error: $$\mathbb{E}\left[(Y-\hat{Y})^2\right] \geq \text{Var}(\hat{Y}) \\ \mathbb{E}\left[(Y-\hat{Y})^2\right] \geq \text{Bias}(\hat{Y},Y)^2$$ And therefore the true lower-bounds on the mutual information are: $$I(Y;\hat{Y}) \geq h(Y) - \frac{1}{2} \log{(2\pi e \cdot \text{Var}(\hat{Y}))} \geq h(Y) - \frac{1}{2} \log{(2\pi e \delta)} \\ I(Y;\hat{Y}) \geq h(Y) - \frac{1}{2} \log{(2\pi e \cdot \text{Bias}(\hat{Y},Y)^2)} \geq h(Y) - \frac{1}{2} \log{(2\pi e \delta)}$$ These new lower-bounds indicate that although your mean squared-error values were small, they were not small enough, as they are upper-bounded by the variance and bias of $$\hat{Y}$$. Therefore, what we really want to do to maximize the mutual information between $$Y$$ and $$\hat{Y}$$ is to decrease the variance and bias of $$\hat{Y}$$. Unfortunately, we know that there is a trade-off between the variance and bias of $$\hat{Y}$$.

As @Pieter mentioned in their answer, it is highly likely that your network is too complex, leading to a $$\hat{Y}$$ with high variance and low bias, otherwise known as overfitting. In this case, although the low bias offers a high "floor" for the mutual information, the mutual information can possibly be lower than this floor due to the high variance.

First, as discussed in What should I do when my neural network doesn't learn, consider over-fitting and then testing your network on a single example. If the network does not predict the mean on a single example, then you can scale up slowly and start over-fitting and testing the network on more examples.

However, if your network still predicts the mean even when over-fitting and testing on a single example, then, in addition to simplifying your network and using less parameters, consider using a modified version of the mean squared-error loss function. More precisely, if $$y$$ is your target and $$\hat{y}$$ is the output of your network, and if: $$\mathcal{L}(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N (y_i-\hat{y}_i)^2$$ Is the mean squared-error loss function that you are currently using, then consider using the following loss function instead: $$\mathcal{L}(y,\hat{y}) = \left[\frac{1}{N} \sum_{i=1}^N (y_i-\hat{y}_i)^2\right] + \alpha \cdot \left[\frac{1}{N} \sum_{i=1}^N (\log(y_i)-\log(\hat{y}_i))^2\right]$$ Where $$\alpha$$ is a hyperparameter that can be tuned via trial and error. A starting value for $$\alpha$$ could be $$\alpha=5$$. The advantage of this loss function over the plain mean squared-error loss function is that the $$\log(\cdot)$$ function stretches small values in the interval $$[0,1]$$ away from each other, which means that small differences between $$y$$ and $$\hat{y}$$ are amplified, leading to larger gradients. I have personally found this loss function to be very helpful in avoiding predicting the mean in regression problems.

For this to work well, it is recommended (but not necessary) that $$y$$ and $$\hat{y}$$ are each scaled to have values in the interval $$[0,1]$$. Also, since $$\log(0)=\infty$$, and since it is likely that $$y$$ and $$\hat{y}$$ will have values very close to $$0$$, then it is recommended to add a small value $$\epsilon$$, such as $$\epsilon=10^{-9}$$, to $$y$$ and $$\hat{y}$$ in the loss function as follows: $$\mathcal{L}(y,\hat{y}) = \left[\frac{1}{N} \sum_{i=1}^N (y_i-\hat{y}_i)^2\right] + \alpha \cdot \left[\frac{1}{N} \sum_{i=1}^N (\log(y_i + \epsilon)-\log(\hat{y}_i + \epsilon))^2\right]$$ This loss function may be thought of as the Mean Squared Log-scaled Error Loss.

It looks like a typical overfitting problem. Your data does not provide enough information to get the better result. You choose the complex NN with you train to remember all nuances of the train data. Loss can never be a zero, as it is on your graph. BTW It seems your validation has a bug or validation set is not a good for validation because the validation loss is also getting zero.

• The question says the network almost always outputs zero. That would be a case of severe underfitting, not overfitting. There's also no gap between training and validation error on the learning curve, indicating that overfitting isn't the problem (the error isn't zero, the scale is logarithmic) – user20160 Oct 2 '17 at 8:28

I was actually working on a very similar problem. Basically, I had a bunch of dots on a white background and I was training a NN to recognize the dot that was placed on the background first. The way I found to work was to just use one fully-connected layer of neurons (so a 1-layer NN). For example, for a 100x100 image, I would have 10,000 input neurons (the pixels) directly connected to 2 output neurons (the coordinates). In PyTorch, when I converted the pixel values to a tensor, it was normalizing my data automatically, by subtracting the mean and dividing by the standard deviation. In normal machine learning problems, this is fine, but not for an image where there might be a disparity in the number of colored pixels in an image (i.e. yours where there are only a few white pixels). So, I manually normalized by dividing all pixel intensity values by 255 (so they're now in the range of 0-1 without the typical normalization technique that tries to fit all the intensity values to a normal distribution). Then, I still had issues because it was predicting the average coordinate of the pixels in the training set. So, my solution was to set the learning rate very high, which goes against almost all ML instructors and tutorials. Instead of using 1e-3, 1e-4, 1e-5, like most people say, I was using a learning rate of 1 or 0.1 with stochastic gradient descent. This fixed my issues and my network finally learned to memorize my training set. It doesn't generalize to a testing set too well, but at least it somewhat works, which is a better solution than most everybody else suggested on your question.