1
$\begingroup$

This question already has an answer here:

I designed my own neural network for solving the problem of text summarization. The number of documents in my training dataset is big (more than 100,000 documents) so it is hard to check it on the whole data. In order to verify that my model is good, I train it on a very small dataset (100 documents) in about 100 epochs to see how it behaves. I split this small dataset into 3 sets (6/2/2): training, validation and test. Here is the chart of the losses (red line is training loss, blue line is validation loss and green line is test loss)

Results on small data

  • Is my evaluation on this small dataset good enough to tell whether my model is performing well?

  • Does the above chart shows that my model is getting overfitting easily?

  • Do you have any recommendation for quickly evaluating a new proposed model and avoid overfitting?

Update:

I trained my network again on a bigger data set (4000 documents) and I got the following chart. It does seem my model is not well-designed. Result on 4000 documents

$\endgroup$

marked as duplicate by Sycorax, kjetil b halvorsen, Peter Flom Jun 19 '18 at 11:26

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 2
    $\begingroup$ Wow, 100 entries seems way too small to get valuable information. From my experience you can push up to 5000-6000 entries and it will not take too much time to train (around 15 minutes). Then you will have meaningful information to play with. $\endgroup$ – LoulouChameau Aug 24 '16 at 9:20
  • 1
    $\begingroup$ You seem to be overfitting to your small training data-set. You should add more data, like suggested by @LoulouChameau. If you insist on keeping the small data set for training, then from your graph, you should only train for around 10 epochs, because after that the validation error starts going up, then model starts overfitting and it looses the power to generalize. $\endgroup$ – Gumeo Aug 24 '16 at 10:57
  • $\begingroup$ Thank you for your comments. I have updated my post with a chart when training on a bigger dataset (4000 docs). It seems to behave similarly though $\endgroup$ – The Lazy Log Aug 24 '16 at 11:47
  • $\begingroup$ Overfitting still happens at 10 epochs, though. So. Don't train it for more than 10 epochs. $\endgroup$ – Sycorax Aug 24 '16 at 14:16
  • $\begingroup$ @GeneralAbrial. Yeah, but in that case the loss is still too big and the network does not learn anything useful in my problem. $\endgroup$ – The Lazy Log Aug 25 '16 at 0:13
1
$\begingroup$

Is my evaluation on this small dataset good enough to tell whether my model is performing well?

It's difficult to tell when using a fraction of the available dataset. Also, it depends on the complexity of the task, among other things.

Does the above chart shows that my model is getting overfitting easily?

Yes since overfitting starts before epoch 5.

Do you have any recommendation for quickly evaluating a new proposed model and avoid overfitting?

Add some patience to avoid computing more epochs that needed, and some regularization technique to avoid overfitting too early

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.