1
$\begingroup$

I have a dataset set with ~40 features onto which I'm applying a multi-layer perceptron for regression purposes. The train, validation, and test sets are made up of 3M, 800K, and 800K examples each, respectively. At the end of 50 epochs, the loss for training, validation, and testing is at 0.1436, 0.1433, and 0.1422 respectively---which makes me think that the model generalizes quite well out of sample.

This said, the learning curves look very volatile, and the downward loss over the epochs is barely visible even with when smoothing the plots. What is this indicative of? According to this blog, it could be because the validation set is too small. However, as mentioned earlier, I already have 800K examples in the validation (i.e., ~1/4 of the training set's size). Also, the validation learning curve does go down, albeit in a very "jagged" way, unlike what is shown on the blog.

Here are the learning curves in question:

enter image description here

$\endgroup$
1
  • $\begingroup$ The apparent "volatility" is actually a trick of how the axes are scaled. The range on the vertical axis for Epoch Loss is more than 0.143 to less than 0.146 -- almost 0.003 units. The loss seems to be wandering at random in a very small range, and that range is very close to the loss achieved by the initialization. You need to figure out how to get the loss to decrease substantially from the initialization loss, which is addressed in the duplicate thread. $\endgroup$
    – Sycorax
    Jun 23 at 13:48

1 Answer 1

1
$\begingroup$

This means that your optimization hasn't really settled, yet. There could be many causes for that. If you are sure that the validation set is from the same population as the training set, then the problem lies in the model and optimization. Measures I would consider are:

  • Just be patient, give it more time, maybe it is sorted out with time.
  • Add more regularization.
  • Maybe your step size is too large. Reduce your learning rate.
  • Ordinary MLPs often don't settle anymore if they contain too many layers. Consider switching to a better architecture.
  • Google one of the DLL tips and tricks documents, and go through all the measures that are suggested there, checking whether they improve the convergence.
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.