I am trying to build a neural network on MATLAB using NNTool. I have a huge data set, with more than 20k samples. It has 3 input parameters and 1 output. Here is a data sample: enter image description here

U varies at 0.5 step until 10 (0.2 ; 0.5 ; 1.0 ... 10). n varies from 0 to 1 and m varies from 0 until it equals n.

I am using the Levenberg-Marquartd algorithm and have been playing with number of neurons and hidden layers. Despise that, I have done nothing in other parameters.

How can I know if I am achieving a good performance without ovetfitting with such a huge data set?

Here's what MATLAB tell me about the performance: enter image description here


The specific answer to your question is that one uses the validation accuracy as a measure of how well your model fits -- if validation accuracy is low but training accuracy is high, you've overfit. If both are low, then your model generalizes well and you can use it on your test set.

Traditionally, when testing neural network training and performance, one splits the data into three parts: Training, validation, and testing. Training data is fed to the algorithm; validation data is used to determine appropriate hyperparameters and see which random start is best; testing data is never shown to the classifier but is used to look at generalization performance (ideally, I think, after all training is over).

20k samples is not large at all for a neural network. I'm a bit confused by the graph you show--it shows training, validation, and testing all going down to essentially zero over many epochs. This is to be expected for training error, but validation should normally do slightly worse than training. Further, at least as I see it, test error should only be computed at the end of training (but I suppose MATLAB does not agree with me). It makes me a bit suspicious, but I would say that based on this graph and knowing nothing else it appears your model has achieved good performance (assuming, at least, that the input labels were normalized).

  • $\begingroup$ Because of this graph I am confused. I didn't implemented the network through command line, but used the nntool that has an UI. NNTOOL does the job of spliting the data. I actually didn't pre or post treated the data. I had more than 400k samples, but I needed to reduce because the variation between them were too small, so it could overfit easily. $\endgroup$ – Caio Custódio Sep 6 '17 at 13:36
  • $\begingroup$ Why would you need to reduce the amount of data? I don't understand what you mean that the variation is too small. Further, I'm confused by your description of your variables -- they're all categorical? And you know the relationships between them already? Because if so, you don't want to be training a neural network $\endgroup$ – bibliolytic Sep 6 '17 at 16:27
  • $\begingroup$ Yes, I know the relationship between them! I work with a functional in physics, but it doesn't return good results when my parameter U is small (U<6), but I do know how to calculate it using a exact method and got the right answer, but the functional don't. Since my group still need to found a functional that do this, we are trying to make a ANN that can fit the data using parameters that follow the same rule. If I already can calculate it, why use a ANN? Because to achieve good results, the exact calculation takes too much time and computer resourses, while the ANN is easier to deal. $\endgroup$ – Caio Custódio Sep 6 '17 at 17:39
  • $\begingroup$ ANNs are powerful but finicky -- I wouldn't really suggest it if you have a tractable model already. However if what you're dealing with is a zero-noise pure function estimation and your error is that low, then my person opinion from the graph would be that it worked. Still you could test it by using your model on a subset of your test points to ensure the ANN predictions are good. $\endgroup$ – bibliolytic Sep 7 '17 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.