1
$\begingroup$

Does it make sense to perform hyperparameter tunning if the validation set is not big enough?

I know, because that size of the validation set is not big (or maybe representative) enough since the best results on the validation set are not the best results on the test dataset.

$\endgroup$
2
$\begingroup$

If the validation set size is small, so as your training should be. Then you can perform cross-validation, even leave-one-out CV where the validation set is just one sample, assuming the training time is small. K-fold CV (including LOOCV) is typically more robust compared to using one constant validation set.

If you aren't able to do CV, performing searches over broad ranges may still be fine but you need to be cautious with finer resolutions since your results will probably have large variances. Eventually, it makes sense to do HPO but, in general, is not possible to fine tune your algorithm. Another option is to use heuristics for your hyper-parameters completely depending on your problem, if they exist.

$\endgroup$
  • $\begingroup$ That is absolutely reasonable. I'm not excited of CV (and leave-one-out especially) since it's time consuming. $\endgroup$ – Michael D Dec 11 '19 at 13:43
1
$\begingroup$

In many situations: no. However, there are some steps you can take to a) ensure that what you're doing is sensible and b) push the limits a bit further.


The size of the validation set used to steer hyperparameter optimiziation directly influences the random uncertainty of the performance estimate you are trying to optimize.

Such an optimization usually requires that this uncertainty is << than the differences in performance you encounter.

Essentially, I'd recommend:

  • measure (estimate) the random uncertainty due to the small sample size and compare this to the difference in performance you observe during the optimization.
    If this uncertainty tells you that there's no way you can do the data-driven hyperparameter optimization you want to, it's a complete waste to nevertheless try it. Your only chance then is to fix the hyperparameters by external knowledge and/or to switch to a model where you can do that.

  • Not all figures of merit/performance measures are the same in terms of random uncertainty. Some of the worst ones are counting fractions of cases such as classification accuracy, error rate, sensitivity, specificity, recall and the like. Avoid them and instead use more benign ones if possible. For classification, proper scoring rules are not only better as they actually provide the steering that is needed for the optimization, but they are typically also more benign in terms of this random uncertainty. Regression figures of merit often do comparatively better.

  • Evaluate the performance/performance gain with respect to the random uncertainty due to your validation (optimization) set size and use this to restrict model complexity in hyperparameter regions where you cannot be sure of any improvement (compare 1-sd-rule).

  • The risk of overfitting to your validation (optimization) set increases with increasing number of model comparisons (basically you run into a multiple-comparison problem). Thus, avoid optimization schemes that rely on comparing huge numbers of model (e.g. genetic algorithms or hyperparameter grids with many data points)

  • If your validation (optimization) set was obtained by a random split of the existing data set, you can only gain from using cross validation instead of a single such split: you'll have more cases evaluated, thus somewhat larger sample size (plus possibly information on model stability) and the single split doesn't achieve anything in terms of statistical independence that the cross validation splits don't. If your sample size is so small that you face trouble during optimization, running a 5- or 10-fold cross validation on a coarser hyperparameter grid with only a/5 or 1/10 of the hyperparameter sets to be evaluated is sensible and often not computationally prohibitive.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.