I'm trying to predict a parameter with use of neural networks. The parameter is highly linearly correlated with some of its predictors (Pearson's r~0.9)(However, the influence of time may not be linear/trivial). I use a network with one hidden layer and it never overfits. Very often I get two smooth parallel lines of training and test errors that go close to each other.
I expect that may mean that the train and test sets are very similar. They contain over 2500 units of every parameter. However, in what way are these sets similar? Also, does that mean that the influence of time on the prediction is linear as well? ( I cannot beat MLR with the ANN - although I get close to it)
$\begingroup$
$\endgroup$
4
-
1$\begingroup$ Are you sure you're plotting accuracy and not the error rate? A higher accuracy is better; or, equivalently, a lower error rate is better. If this is truly a plot of accuracy, it would seem that your problem is underfitting because the network is correct less than 5% of the time $\endgroup$– Sycorax ♦Commented Aug 20, 2018 at 18:17
-
$\begingroup$ You are probably mixing the validation set - used to decide when to stop the training - and the test set - used to evaluate real world performance of the model. Usually you analyze test sets using selected accuracy measures, not error/performance lines over training rounds. Under the conditions you laid, the test set must be out-of-sample, out-of-time to do accurate predictions. $\endgroup$– Lucas GallindoCommented Aug 20, 2018 at 18:19
-
$\begingroup$ @Sycorax it should be error rate, sorry for that $\endgroup$– DexzMenCommented Aug 20, 2018 at 18:41
-
$\begingroup$ @LucasGallindo I train the model on winter one year and validate it on winter of the next year $\endgroup$– DexzMenCommented Aug 20, 2018 at 18:43
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
One possibility is that you've simply found a good model. There's no reason a learning algorithm must necessarily overfit as training progresses.
One way for things to go wrong is if the validation and training sets are not independent, causing the validation set error to be overoptimistically biased. There are many ways this could happen. For example, preprocessing the entire dataset before splitting it can allow information to improperly leak from the validation set to the training set. Procedures that make use of the target values (e.g. feature selection) are particularly egregious here, but even seemingly benign, unsupervised procedures could have an effect in some cases. Alternatively, the training and validation sets might not be split in a way that respects natural correlations in the data. For example, with time series data, it would be a mistake to independently subsample the validation set points at random. In this case, points in the validation set would not be independent of training points that are nearby in time, due to temporal correlations. Many other dependence structures are possible, and the procedure for splitting the training and validation sets must take this into account.
answered Aug 20, 2018 at 19:08