# matlab neural network strange simulation performance

I am not quite sure if this is the right place for a question like this, but asking anyway. Having <14x10 double> input matrix (manually normalized) and <5x10 double> output matrix (manually normalized), after a long session of training and comparing performances (the less the better) for different functions, I have finally created five neural networks with the following sets of MATLAB functions:


1 newcf  trainlm initnw mse  learngd  satlin
2 newcf  trainlm initnw msne learngdm compet
3 newelm trainlm initwb mse  learnhd  purelin
4 newff  trainlm initnw mse  learngd  purelin
5 newff  trainlm initnw mse  learnwh  tansig


The best performance results (perf) for each one of them vary from xxxE-30 to xxxE-32.

But still, after running simulation of those networks for each single column of the input matrix, I got the expected output results in just 60% of the cases, while the other 40% are totally wrong.

I have exactly the same 60%/40% relationship between good and bad simulation results for all the above networks, with different bad columns per net.

Can something like this happen? What do you think could be wrong? Maybe the perf result when training is not enough to judge when a neural network is good enough? Maybe I didn't understand something well in the concepts?

[edit 26/04/2013] The approximate code I am using :

inp = {
0,1300  0,0300  0,0300  0,0300  -0,0100 0,0300  0,0900  0,0100  0,0600  0,0700;
0,0500  -0,0400 -0,0400 -0,0400 -0,0100 0,0100  0,1400  0,0900  0,0600  -0,1700;
0,2400  0,2200  0,2200  0,2200  0,3700  0,3000  0,5300  0,3400  0,1400  -0,6700;
.......
}

outp={
0,0427  -0,1071 0,0605  -0,0637 -0,0410 0,2566  -0,0551 -0,0902 -0,2483 0,1543;
-0,0249 0,0192  -0,1199 -0,3748 0,3212  0,5490  -0,1655 -0,1213 -1,0236 0,4678;
.......
}

[out_r, out_c] = size(cell2mat(outp));
[inp_r, inp_c] = size(cell2mat(inp));

%------------ for 2 layers:

biasConnect = [1;1];
inputConnect = [1; 0];
layerConnect = [0 0; 1 0];
outputConnect = [0 1];

%-------------or for 3 layers:

% biasConnect = [1;1;1];
% inputConnect = [1; 0; 0];
% layerConnect = [0 0 0; 1 0 0; 0 1 0];
% outputConnect = [0 0 1];

my_net = network(1, 2, biasConnect,inputConnect,layerConnect,outputConnect);    %create neural network

my_net.inputs{1}.size = inp_r;      % set the number of elements in an input vector
my_net.outputs{1}.size = out_r;     % set the number of elements in an output vector

my_net = newff(inp, outp, [inp_r out_r]);  % Create feed-forward backpropagation network

% or for 3 layers: my_net = newff(inp, outp, [inp_r round(inp_r/2) out_r]);

my_net.layers{1}.size = inp_r;                % 1st layer size
my_net.layers{1}.transferFcn = 'purelin';     % transfer function
my_net.layers{1}.initFcn = 'initnw';

% ---- Functions ----------------------------------%/
my_net.divideFcn = 'divideblock';
my_net.plotFcns = {'plotperform','plottrainstate'};
my_net.initFcn ='initlay';      % Layer init function
my_net.performFcn = 'mse';      % performing function
my_net.trainFcn = 'trainlm';    % training function
my_net.adaptFcn = 'learngd';    % should be from list: learngdm, learngd

% ---- set a few traininig params and train the net

my_net.trainParam.epochs = 100;
my_net.trainParam.goal = 1.0000e-030;
my_net.trainParam.max_fail = 3;
my_net.trainParam.mu = 1.0000e-03;

my_net.trainParam.mu_inc = 10;
my_net.trainParam.mu_dec = 0.1000;
my_net.trainParam.mu_max = 1e10;

my_net.trainParam.showWindow = false;
my_net.trainParam.showCommandLine = false;
my_net.trainParam.show = 0;

[my_net,tr] = train(my_net,inp,outp); % train the network

% ---- After all when the best my_net is found, I perform simulation for each column:

Y = sim(my_net, inp{:,i}); % for each i-column of the inp matrix and expect y=outp{:,i}


It is not very clear from the question exactly what has been done, but I suspect the problem is that there just is not enough data (10 patterns) to train a useful neural network to do anything meaningfully. The errors are very low, which suggests that the neural network has essentially memorised the training data, which is likely to result in severe over-fitting.

Unless more data is available, I would advise using something like ridge regression, rather than neural networks.

What exactly is meant by "But still, after running simulation of those networks for each single column of the input matrix"? If you could write a MATLAB script that shows what you are trying to do, that would make it much easier to find the problem.

• Thank you very much for you answer! Indeed no more data is available but neural network as a statistical method is preferable in my case for other reasons … Therefore, I tested (automatically) all possible combinations of train/transfer/adapt functions trying to find an appropriate combination with the lowest possible performance value (perf). Should I include a few more criteria for the determination of the best result (except for perf value)? If yes, then what other criteria could be used? If no, then how can I avoid that over-fitting while still using neural networks? Thank you Commented Apr 26, 2013 at 14:12
• Regarding the simulation, please refer to the code I attached in the edited question above - I try to compare the output of the simulation with the output data I trained my network with. Shouldn't I expect to have the simulation output be approximately equal to my output data, for the same inputs as the input data? - even in case neural network just memorized these values. I am asking because I suspect that I do some other folly mistake in my thinking. Thank you again Commented Apr 26, 2013 at 14:17
• can you explain the "other reasons" for which a neural net is preferable. As you have more attributes than training patterns a linear model will be able to learn the data with zero error, so it is hard to see why a neural network should be expected to do better. Commented Apr 26, 2013 at 15:04
• I've not really used the neural network toolbox (NETLAB, which is freely available from Aston University is a much better toolbox), but it looks to me as if you are using the newff incorrectly, the first parameter should be an n by 2 array of minimum and maximum values for each feature, and the second an array of the dimensions for each layer, whereas you have passed in the input and response variables. Commented Apr 26, 2013 at 15:10
• They can also perturb the training images by slight elastic transforms, subsampling/resampling, etcetera. Commented Oct 14, 2013 at 22:02