# Neural network weights

This is going to be a long question :

I have written a code in MATLAB for updating the weights of MLP with one hidden layer . Here is the code :

weights_1 : weight matrix for input to hidden layer weights_2 : weight matrix for hidden to output layer

function [ weights_1,weights_2 ] = changeWeights( X,y,weights_1,weights_2,alpha )
%CHANGEWEIGHTS updates the weight matrices
%   This function changes the weight of the weight matrix
%   for a given value of alpha using the back propogation algortihm

m = size(X,1) ;     % number of samples in the training set

for i = 1:m
% Performing the feed-forward step
X_i  = [1 X(i,1:end)]    ;
z2_i = X_i*weights_1'    ;
a2_i = sigmoid(z2_i)     ;
a2_i = [1 a2_i]          ;
z3_i = a2_i*weights_2'   ;
h_i  = sigmoid(z3_i)     ;

% Calculating the delta_output_layer
delta_output_layer = ( y(i)' - h_i' )...
.*sigmoidGradient(z3_i')  ; % 3-by-1 matrix

% Calculating the delta_hidden_layer
delta_hidden_layer =  (weights_2'*delta_output_layer)...
.*sigmoidGradient([1;z2_i']) ; % 5-by-1 matrix
delta_hidden_layer = delta_hidden_layer(2:end) ;

% Updating the weight matrices
weights_2 = weights_2 + alpha*delta_output_layer*a2_i ;
weights_1 = weights_1 + alpha*delta_hidden_layer*X_i  ;
end

end


Now I wanted to test it on the fisheriris dataset given in MATLAB which can be accesed by load fisheriris command . I renamed meas to X and changed species to a 150-by-3 matrix where each row depicts the name of species (as for example first row is [1 0 0])

I compute error of the output layer using the following function :

function [ g ] = costFunction( X,y,weights_1,weights_2 )
%COST calculates the error
%   This function calculates the error in the
%   output of the neural network

% Performing the feed-forward propogation
m = size(X,1) ;
X_temp  = [ones([m 1]) X]   ;  % 150-by-5 matrix
z2 = X_temp*weights_1'       ; % 150-by-5-by-5-by-4
a2 = sigmoid(z2)       ;
a2 = [ones([m 1]) a2]            ; % 150-by-5
z3 = a2*weights_2'     ; % 150-by-3
h  = sigmoid(z3)       ; % 150-by-3

g = 0.5*sum(sum((y-h).^2)) ;
g = g/m ;
end


Now in the course the prof gave an example of toy network with 3 iterations , I tested this on that network and it gives the right values but when I test it on the fisheriris data the cost keeps on increasing . And I am not able to understand where it is going wrong .

Here is the toy network for which it runs fine :

there is only training example for this set .

PS : Ignore the comments ( they are size of matrix used for checking the validity of matrix multiplication for a sample case )

Finally here is the test-bench execute.m , sigmoid.m and sigmoidGradient.m which I have shared just in case to run the functions and test them

• Decrease your alpha. You are diverging during gradient descent – ignorant Feb 5 '14 at 22:04
• Why do you use the same bias unit for different layers?. That seems odd to me. The bias term compensates for the difference between the mean value of training data and the mean response of neurons. Each layer corresponds to a different mapping of the data, so I would expect a different bias unit per layer. – jpmuc Feb 6 '14 at 9:04