Training a neural network for regression always predicts the mean

I am training a simple convolutional neural network for regression, where the task is to predict the (x,y) location of a box in an image, e.g.:     The output of the network has two nodes, one for x, and one for y. The rest of the network is a standard convolutional neural network. The loss is a standard mean squared error between the predicted position of the box, and the ground truth position. I am training on 10000 of these images, and validating on 2000.

The problem I am having, is that even after significant training, the loss does not really decrease. After observing the output of the network, I notice that the network tends to output values close to zero, for both output nodes. As such, the prediction of the box's location is always the centre of the image. There is some deviation in the predictions, but always around zero. Below shows the loss: I have run this for many more epochs than shown in this graph, and the loss still never decreases. Interestingly here, the loss actually increases at one point.

So, it seems that the network is just predicting the average of the training data, rather than learning a good fit. Any ideas on why this may be? I am using Adam as the optimizer, with an initial learning rate of 0.01, and relus as activations

If you are interested in some of my code (Keras), it is below:

# Create the model
model = Sequential()
model.add(Convolution2D(32, 5, 5, border_mode='same', subsample=(2, 2), activation='relu', input_shape=(3, image_width, image_height)))
model.add(Convolution2D(64, 5, 5, border_mode='same', subsample=(2, 2), activation='relu'))
model.add(Convolution2D(128, 5, 5, border_mode='same', subsample=(2, 2), activation='relu'))

# Compile the model
adam = Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)

# Fit the model
model.fit(images, targets, batch_size=128, nb_epoch=1000, verbose=1, callbacks=[plot_callback], validation_split=0.2, shuffle=True)
• Are the images on top examples of your actual samples? Is that 5 separate samples? There appears to be no information in the images that would help generalize. I mean, you don't need a neural net to find the x,y location of the white square, you can just parse the image and look for a white pixel. Explain a bit more about your vision for this model. Is there some temporal pattern, whereby you are predicting the next location? – photox Feb 14 '17 at 2:06
• Hi, and yes, the images are 5 separate samples. I'm not sure how they are rendered for you, but they should be 5 individual square images (I've changed the layout a little to help...). Yes, I realise that you don't need a neural network for this task, but it is just a test experiment to help me learn how to do regression with a neural network. I don't understand what you mean by there being no information to help generalize.... Each training pair consists of a square image, and a two-dimensional vector of the (x, y) location of the square. Thanks :) – Karnivaurus Feb 14 '17 at 11:57
• 1) Your input shape on the first conv layer is using 3 (rbg) channels, but your data are greyscale (1 channel) 2) You don't need that many conv layers and filters, in fact I think a single layer, and a handful of small kernels will be fine. – photox Feb 14 '17 at 12:04
• Are you sure that images do indeed correspond the targets? – user31264 Feb 14 '17 at 12:18
• Like @photox says, you do not need the conv layers. Adding these make it more difficult for the optimizer to find a good solution. If you remove the 3 conv layers I suspect your "model" will work. – Pieter Feb 14 '17 at 22:02

The optimizer is unable to converge to a (sub-)optimal solution. Why? Your problem is too easy and/or your model is too complex.

Too easy problem

As @photox already said this problem can be solved with only a single hidden layer. I even suspect that it can be done without a hidden layer. That is because this problem is linear seperable.

Let me illustrate this. Imagine a neural network without hidden layers and a linear activation function (you might also call it linear regression). To compute the x-location of the square each pixel is connected to the x-output. The first column of pixels is connected with weight $1/\text{height}/\text{width}$. The second column is connected with weight $2/\text{height}\text{width}$. This continues until the last column (e.g. column $n$) which is connected with weight $n/\text{height}\text{width}$. Since $\text{height}*\text{width}$ pixels are non-zero (e.g. bias is equal to grey color value) the activation of the x-output is equal to the center of the square. Hence, a linear function can compute the location of the square.

There are several solutions:

• Choose a harder problem, e.g. image classification
• Add noise, e.g. salt-and-pepper or white-noise
• Make problem harder, e.g. by predicting the location of a red square while there are lots of differently colored circles in the background

Too complex model

Your model has a few parts that add a lot of complexity while not helping the optimizer to find a sweet optimum.

For example, the convolutional layers. The first layer has 32 convolutional filters of size $5 \times 5$. What do you expect these filters to learn? In image classification these filters learn to detect edges, corners, gradients and blobs. But in this case there are only few filters that make sense. I can think of edge from left-to-right and vice-versa and from top-to-bottom and vice-versa. In your case there are thus approx 28 filters that just add random noise. Deleting these (or just the whole layer) makes it a lot easier for the optimizer to find an optimum that works.

Another example is the Adam optimizer with a lot of extra parameters. The Adam optimizer might work well with these parameters but why don't you just start with a simple SGD optimizer with default values.

Thus you can make several optimizations:

• use LinearRegression from scikit-learn. OK, this is not what you want but I just wanna illustrate how overly complex this model is.
• Remove the conv layers
• Decrease the size of the hidden Dense layers
• Use the default SGD optimizer
• If you are using a hidden layer, you should try a sigmoid activation. You can think of each of the nodes of the hidden layer as detecting if a square is at a certain location.
• If this all does not work, experiment with the learning rate a bit to find out if it is too high or too low.

Ps

I think you will like this blogpost by Adit Deshpande.

• Let me know if these solutions did change the optimizers behavior. – Pieter Feb 14 '17 at 22:55
• Thanks, that's very useful. I'm working on trying your suggestions. However, I don't understand your first point. I'm confused as to why, if the problem is too simple, then it is harder to optimize than a more complex problem. For a given network, why would a simpler problem be harder to optimize than a more complex problem? In a simple problem, I would have thought that there would be very strong gradients and a strong global optimum. But your first point says that the simplicity of the problem makes optimization hard, suggesting that a more complex problem would help the optimization... – Karnivaurus Feb 15 '17 at 2:41

It looks like a typical overfitting problem. Your data does not provide enough information to get the better result. You choose the complex NN with you train to remember all nuances of the train data. Loss can never be a zero, as it is on your graph. BTW It seems your validation has a bug or validation set is not a good for validation because the validation loss is also getting zero.

• The question says the network almost always outputs zero. That would be a case of severe underfitting, not overfitting. There's also no gap between training and validation error on the learning curve, indicating that overfitting isn't the problem (the error isn't zero, the scale is logarithmic) – user20160 Oct 2 '17 at 8:28

I am facing the same problem with my data set. It turns out that in my case the predictors are highly concentrated with a very small variance. You should check out the variance of your prediction variables and see how it is distributed. However, some transformations on the output variable can be performed to modify or change its scale. This might result in a more uniform type distribution. For example, in image recognition tasks histogram equalization or contrast enhancement sometimes works in the favor of correct decision making.

I was actually working on a very similar problem. Basically, I had a bunch of dots on a white background and I was training a NN to recognize the dot that was placed on the background first. The way I found to work was to just use one fully-connected layer of neurons (so a 1-layer NN). For example, for a 100x100 image, I would have 10,000 input neurons (the pixels) directly connected to 2 output neurons (the coordinates). In PyTorch, when I converted the pixel values to a tensor, it was normalizing my data automatically, by subtracting the mean and dividing by the standard deviation. In normal machine learning problems, this is fine, but not for an image where there might be a disparity in the number of colored pixels in an image (i.e. yours where there are only a few white pixels). So, I manually normalized by dividing all pixel intensity values by 255 (so they're now in the range of 0-1 without the typical normalization technique that tries to fit all the intensity values to a normal distribution). Then, I still had issues because it was predicting the average coordinate of the pixels in the training set. So, my solution was to set the learning rate very high, which goes against almost all ML instructors and tutorials. Instead of using 1e-3, 1e-4, 1e-5, like most people say, I was using a learning rate of 1 or 0.1 with stochastic gradient descent. This fixed my issues and my network finally learned to memorize my training set. It doesn't generalize to a testing set too well, but at least it somewhat works, which is a better solution than most everybody else suggested on your question.