# Regression with neural network

I tried to create a neural network for regression. In order to test the concept, I created a dataset the following way:

x1 = random('Normal',0,5,500,1);
x2 = random('Normal',0,5,500,1);
y = x1 + 2*x2;
X = [x1 x2];


So x1 and x2 are vectors of random numbers with mean value 0 and standard deviation 5.

The neural network therefore had 2 inputs, 2 or 3 units in hidden layer and one output unit. In hidden layer units I implemented sigmoid activation function and in the output layer linear function:

h(y) = 1*k20 + a1*k21+ a2*k22 + a3*k23


and for hidden layer outputs are calculated the following way:

a1 = sigmoid(1*k10 + x1*k11 + x2*k12)
(and similarly a2 = sigmoid(1*l10 + x1*l11 + x2*l12)


Edit: The Backpropagation algorithm is implemented the following way:

Delta_1=0;
Delta_2=0;

for t=1:m
currentSampleSize = sampleSize(t,:);
for i = 1 : currentSampleSize
% Step 1
a_1 = X(t,:)';
z_2 = Theta1*a_1;
a_2 = sigmoid(z_2);
a_2 = [1; a_2];
z_3 = Theta2*a_2;
a_3 = z_3;

% Step 2
delta_3 = (a_3 - y_new(t,:)');

% Step 3
grad = m^(-1) * X' * (a_3-y);
delta_2 = Theta2'*delta_3.*[1;sigmoidGradient(z_2)];
delta_2 = delta_2(2:end);

%Step 4
Delta_2 = Delta_2 + delta_3*a_2';
Delta_1 = Delta_1 + delta_2*a_1';
end
end

Theta1_grad = Delta_1/m;
Theta2_grad = Delta_2/m;

reg_1 = lambda/m.*Theta1(:,2:end);
reg_2 = lambda/m.*Theta2(:,2:end);

Theta1_grad(:,2:end) = Theta1_grad(:,2:end) + reg_1;
Theta2_grad(:,2:end) = Theta2_grad(:,2:end) + reg_2;


As the predictions are not accurate I wonder what could be done in order to improve the regression performance. I tried to change the combinations of number of neurons in hidden layer and different regularization parameter values but the performance wasn't good enough.

I would be very thankful if anyone could describe the appropriate design of neural network for regression problems - which activation functions are most appropriate and how to diagnose training and test error for choosing the optimal number of hidden units.

• Sorry about my earlier comment, did not notice it was a toy example. – Gala Jul 23 '13 at 7:40
• I evaluate the performance by checking training and test error. I just wanted to try out the concept with neural networks. Actually I wanted to reproduce the time series prediction result from the book (Machine Learning: An Algorithmic Perspective) and couldn't achieve comparable results so I wanted to find the main cause of bad performance. – niko Jul 23 '13 at 7:42

## 2 Answers

Assuming you wrote the implementation yourself, you may simply have a bug in your backpropagation algorithm. Some bugs can be quite subtle and leave the algorithm partially working with poor performance. You might try adding some gradient checking code to your implementation to verify the calculated gradients. Here's an excellent video by Andrew Ng on how to do this:

http://www.youtube.com/watch?v=12a9fsLyFes

Note that I'm assuming you're getting poor performance on the training set, rather than the test set. You should be able to get near perfect results on the training set, at which point your implementation is likely correct and you can start dealing with overfitting by adding regularization, etc. I would disable regularization until you get to that point.

As a side note, since you're dataset has a linear relationships between the inputs and the outputs you'll likely get a better result with NO hidden neurons (i.e. a single layer perceptron), but then an MLP should work too, and you can then test it on a non-linear dataset.

• Great answer - I don't know why people write their own backpropagation code anyway. Numerical algorithms have all kinds of complications - it's easier, and smarter, to use something already built and tested. – Rohit Chatterjee Jul 24 '13 at 5:04
• Thank you. Actually I followed Andrew Ng's lectures and I performed gradient checking. Again, this is just a toy example to test the concept. The actual problem I am trying to solve has more inputs and the relationship between inputs and output is not linear. – niko Jul 24 '13 at 7:28

Unless you restrict the range of your inputs, the sigmoid may be giving you a problem. You won't even be able to learn the function $y=x$ if you have a sigmoid in the middle.

If you have a restricted range, then the input-hidden weights could scale the input values so that they hit the sigmoid at the part of the graph which looks like a line i.e. the part around 0:

Once you have the linear behavior, the hidden-output weights could re-scale the values back. The training process will take care of all this for you - but to test this hypothesis you could just train (and validate) with input values in $[-1,1]$.

Do you know what the network's function looks like? If you plot $(x_1, x_2) \rightarrow y$, does it look anything like a plane? At least in parts?

You could also try training with more data and seeing if the graph gets closer to $x_1+2x_2$.