1
$\begingroup$

I have built an LSTM model for multivariate time series (6 time series). I am predicting one of the variables given the expected values of other five variables. Goal is to compare the accuracy of the LSTM model to Vector autoregression(VAR) model that I built earlier. LSTM gives me better accuracy than VAR. So, should I stop tuning the LSTM any further or should I tune it further for gain in accuracy?

This is a general question which often comes up when tuning deep learning and machine learning algorithms such as recurrent neural network, multilayer perceptron or SVM etc.

When should one decide to stop tuning a neural network any further. I understand that there could be many kinds of constraints such as a deadline when the tuned model is required, etc. for a particular problem but what I am asking is in general. Or one should simply stop tuning a network anymore when one feels that s/he has achieved the desired prediction accuracy or should one try to achieve even better accuracy.

There are plenty of guidelines available on hyper parameter tuning/optimization but its very difficult to find one which summaries when to stop tuning the network any further and stop.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You need to stop training when the generalisation error, the error your model will make on unseen data, increases. Since you cannot measure this error directly by definition, one typically uses an approximation by evaluating the model on a held out (validation) set. When the error in this set increases, one stops. Cross-validation can also be used, but the variance of the error estimator will be higher (meaning: more noisy values) and you will have to use some sort of smoothing or heuristic to decide when to actually stop.

On the other hand, for certain classes of regression models in reproducing kernel Hilbert spaces there exist bounds related to the number of samples which provide early stopping rules.

Improve this answer
$\endgroup$
2
  • $\begingroup$ When we tune the hyper parameters of a deep learning model every possible combination of a hyper parameter results in a different model. And we select an optimal combination based on the loss curves. My question is exactly this: There is an infinite number of combinations of hyperparams possible. What should the model selection decision be based upon? We know that there are many possible configurations of hyperparams possible that give similar generalization error. And how do I know I have hit the bottom and no other combination of hyperparams will give me better results? $\endgroup$
    – naive
    Commented Jun 10, 2018 at 9:54
  • $\begingroup$ Well, you don't. You are describing the problem of local mínima of a non convex function. You just don't know if there are better choices of hyperparameters. You can use clever techniques to help exploring the space, like bayesian optimization, but the issue remains that one cannot guarantee global optimality. $\endgroup$
    – Miguel
    Commented Jun 12, 2018 at 5:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.