2
$\begingroup$

Below is a specific case but a general situation I find myself involved within my job. This question is intended at getting ideas on how to pick the best model:

Dataset:

rows: 10,166, features: 1692, Model used: Feed Forward Neural Networks

logs for a particular fold:

epoch: 0, train loss : 0.585782928946,  train score: 0.497655085604, val score: 0.610834066686 
training  with 32 sized batches
best validation score till now: 0.0

epoch: 1, train loss : 0.350900779755,  train score: 0.509627315655, val score: 0.625354453897 
training  with 32 sized batches
best validation score till now: 0.0

epoch: 2, train loss : 0.249989495512,  train score: 0.528054031557, val score: 0.646059450474 
training  with 32 sized batches
best validation score till now: 0.0

epoch: 3, train loss : 0.210879948519,  train score: 0.553915714449, val score: 0.65603304977 
training  with 32 sized batches
best validation score till now: 0.0

epoch: 4, train loss : 0.196106336583,  train score: 0.578792288474, val score: 0.644788305466 
training  with 32 sized batches
best validation score till now: 0.0

epoch: 5, train loss : 0.189342178014,  train score: 0.601882261054, val score: 0.640632639093 
training  with 32 sized batches
best validation score till now: 0.0

epoch: 6, train loss : 0.18660737301,  train score: 0.616972833149, val score: 0.640021511685 
training  with 32 sized batches
new best validation score: 0.629363449692
saving model...

epoch: 7, train loss : 0.183136423458,  train score: 0.630294120457, val score: 0.629363449692 
training  with 32 sized batches
best validation score till now: 0.629363449692

epoch: 8, train loss : 0.180001894893,  train score: 0.641283331582, val score: 0.611542974479 
training  with 32 sized batches
best validation score till now: 0.629363449692

epoch: 9, train loss : 0.179555135817,  train score: 0.646623580034, val score: 0.606971741469 
training  with 32 sized batches
best validation score till now: 0.629363449692

epoch: 10, train loss : 0.17941892064,  train score: 0.650499675163, val score: 0.603036080962

and the trend of increasing training score and decreasing validation score continues. My question is which model (after which epoch) should be selected. Currently, I select by early stopping based on validation score not having increased for more than 15 epochs and under the condition that training score is greater than the validation score at that epoch. Which makes it model after epoch 7 here.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Note that a problem of feedforward neural networks in general is overfitting, getting a better score on the training set but decreasing performance on the validation and test set. To counter this early stopping is a possibility. The approach I usually use comes from this chapter of a book. There are several stopping criteria possible to use. I recommend you to read this chapter for a good overview.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ that is understandable. But my training schedule exact opposite. Validation accuracy is higher than training accuracy $\endgroup$
    – figs_and_nuts
    Commented Oct 18, 2018 at 9:38
  • $\begingroup$ You probably want to investigate how you're splitting. Sometimes there can be structural reasons why the test set fits better than the training set. For example, say you're fitting 1 year worth of time series data and you do a non-shuffled 80:20 split. Now maybe the last 20% was autumn - maybe the model works better in autumn (i.e. in short maybe your test/ train split isn't as random as it could be) $\endgroup$
    – David Waterworth
    Commented Mar 9, 2020 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.