matlab neural network strange simulation performance

Question

I am not quite sure if this is the right place for a question like this, but asking anyway. Having <14x10 double> input matrix (manually normalized) and <5x10 double> output matrix (manually normalized), after a long session of training and comparing performances (the less the better) for different functions, I have finally created five neural networks with the following sets of MATLAB functions:

1 newcf  trainlm initnw mse  learngd  satlin
2 newcf  trainlm initnw msne learngdm compet
3 newelm trainlm initwb mse  learnhd  purelin
4 newff  trainlm initnw mse  learngd  purelin
5 newff  trainlm initnw mse  learnwh  tansig

The best performance results (perf) for each one of them vary from xxxE-30 to xxxE-32.

But still, after running simulation of those networks for each single column of the input matrix, I got the expected output results in just 60% of the cases, while the other 40% are totally wrong.

I have exactly the same 60%/40% relationship between good and bad simulation results for all the above networks, with different bad columns per net.

Can something like this happen? What do you think could be wrong? Maybe the perf result when training is not enough to judge when a neural network is good enough? Maybe I didn't understand something well in the concepts?

Thank you in advance.

[edit 26/04/2013] The approximate code I am using :

inp = {
0,1300  0,0300  0,0300  0,0300  -0,0100 0,0300  0,0900  0,0100  0,0600  0,0700;
0,0500  -0,0400 -0,0400 -0,0400 -0,0100 0,0100  0,1400  0,0900  0,0600  -0,1700;
0,2400  0,2200  0,2200  0,2200  0,3700  0,3000  0,5300  0,3400  0,1400  -0,6700;
.......
}

outp={
0,0427  -0,1071 0,0605  -0,0637 -0,0410 0,2566  -0,0551 -0,0902 -0,2483 0,1543;
-0,0249 0,0192  -0,1199 -0,3748 0,3212  0,5490  -0,1655 -0,1213 -1,0236 0,4678;
.......
}

[out_r, out_c] = size(cell2mat(outp));   
[inp_r, inp_c] = size(cell2mat(inp));

%------------ for 2 layers:                 

biasConnect = [1;1];                            
inputConnect = [1; 0];
layerConnect = [0 0; 1 0];  
outputConnect = [0 1];      

%-------------or for 3 layers:

% biasConnect = [1;1;1];      
% inputConnect = [1; 0; 0]; 
% layerConnect = [0 0 0; 1 0 0; 0 1 0];
% outputConnect = [0 0 1];

my_net = network(1, 2, biasConnect,inputConnect,layerConnect,outputConnect);    %create neural network 

my_net.inputs{1}.size = inp_r;      % set the number of elements in an input vector 
my_net.outputs{1}.size = out_r;     % set the number of elements in an output vector 

my_net = newff(inp, outp, [inp_r out_r]);  % Create feed-forward backpropagation network

% or for 3 layers: my_net = newff(inp, outp, [inp_r round(inp_r/2) out_r]);  

my_net.layers{1}.size = inp_r;                % 1st layer size   
my_net.layers{1}.transferFcn = 'purelin';     % transfer function 
my_net.layers{1}.initFcn = 'initnw';         

% ---- Functions ----------------------------------%/
my_net.divideFcn = 'divideblock';    
my_net.plotFcns = {'plotperform','plottrainstate'};  
my_net.initFcn ='initlay';      % Layer init function           
my_net.performFcn = 'mse';      % performing function 
my_net.trainFcn = 'trainlm';    % training function 
my_net.adaptFcn = 'learngd';    % should be from list: learngdm, learngd    

% ---- set a few traininig params and train the net 

my_net.trainParam.epochs = 100;
my_net.trainParam.goal = 1.0000e-030; 
my_net.trainParam.max_fail = 3;     
my_net.trainParam.mu = 1.0000e-03;  

my_net.trainParam.mu_inc = 10;      
my_net.trainParam.mu_dec = 0.1000;  
my_net.trainParam.mu_max = 1e10;    

my_net.trainParam.showWindow = false; 
my_net.trainParam.showCommandLine = false; 
my_net.trainParam.show = 0; 

[my_net,tr] = train(my_net,inp,outp); % train the network 

% ---- After all when the best my_net is found, I perform simulation for each column:

Y = sim(my_net, inp{:,i}); % for each i-column of the inp matrix and expect y=outp{:,i} 

Answer 1

It is not very clear from the question exactly what has been done, but I suspect the problem is that there just is not enough data (10 patterns) to train a useful neural network to do anything meaningfully. The errors are very low, which suggests that the neural network has essentially memorised the training data, which is likely to result in severe over-fitting.

Unless more data is available, I would advise using something like ridge regression, rather than neural networks.

What exactly is meant by "But still, after running simulation of those networks for each single column of the input matrix"? If you could write a MATLAB script that shows what you are trying to do, that would make it much easier to find the problem.