0
$\begingroup$

When I trained ANN model, sometime training set has high R-square but testing set has low R-square. How to explain this situation? Is there any over-fitting problem? (Show in figure) Regression

enter image description here

$\endgroup$
1
$\begingroup$

In addition to what written by @enricoanderlini, I would also point out that there don't seem to be many datapoints in your test set. Having a (small) number of points, exposes you, even more, to the impact of leverage / influential points (you can also read about it here). With regards to this, @enricoanderlini is correct when he suggests cross-validation.

Interestingly, while the focus is often on model capacity (degrees of freedom), your set-up highlights that ANN calibration, in a train-validate-test pipeline, is "hungrier" of datapoints than, say, just a OLS with a train-test split.

$\endgroup$
0
$\begingroup$

The results seem to suggest overfitting is a problem at the moment.

Have you tried to select the training, validation and test sets differently? What I mean is, choose a number of different sets, say 5-10, from the whole data set you have. This way if you observe the same behaviour with all of them you know you have to redesign the NN.

If overfitting is a problem, you may want to decrease the number of neurons.

$\endgroup$
  • $\begingroup$ Can you give me a example? what is "choose number of different sets"? I can' get it. And how you recognize the overfitting from figure? Thank you, I appreciate. $\endgroup$ – Jeffrey Dec 29 '16 at 14:18
  • $\begingroup$ Overfitting occurs when the statistical model describes the noise of the data as well as the general relationship. In a simplistic way, this occurs when you fit the training data "too well", whereas the validation data presents a poorer fit. This is more or less what you observe: the quality of the fit for the training data is excellent, while it shrinks for the validation and test data sets. $\endgroup$ – Enrico Anderlini Dec 29 '16 at 15:01
  • $\begingroup$ So, what I suggest you do is: take your data set, order it randomly, select approximately 70-80% of the points as the training set, 20-10% as the validation set and 10% as the test set (you may use other values based on what you find in the literature). Then, you repeat this process another ~4 times. $\endgroup$ – Enrico Anderlini Dec 29 '16 at 15:04
  • $\begingroup$ If you observe the same behaviour (i.e. quality of the fit of the validation and test data sets worse than that of the training set) for all repeats, then you understand there is clearly a problem with your model as is, and you need to change it. $\endgroup$ – Enrico Anderlini Dec 29 '16 at 15:06
  • $\begingroup$ Some people also take the average from all cases to produce the final model. This is a way to fully exploit a possibly limited data set. I hope this helps. $\endgroup$ – Enrico Anderlini Dec 29 '16 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.