As far as I understood, when implementing a learning algorithm that integrates model selection/hyper parameters tuning in itself, nested cross-validation is necessary to lower the bias in the performance estimate.

I’d propose to summarize the algorithm to compute the estimation of that performance as:

performances = [] for training, test in partition(data): model = find_best_model(data_to_choose_best_model=partition(training)) performances.append(model.fit_and_measure_performance(training, test)) return some_method_to_aggregate_for_ex_average(performances)

As we don’t have an infinite amount of time, we’re obliged to restrict the number of model/parameters to browse during find_best_model. Taking aside the fact that we don’t use the models we don’t know, I’d enumerate two ways of selecting that subset of model/parameters:

  1. experience/gut feeling,
  2. exploration/plotting some curves to evaluate how an algorithm reacts to a given data.

My question is the following: Is there is a way to implement 2., for example, in the way to select/explore the data, that would permit lowering the bias it creates ?

Indeed, implementing 2 ourselves, i.e. out of the “find_best_model” method in the algorithm above, seems to be a “seemingly benign short cut” that may induce a non negligible “magnitude of [...] bias” (taking expressions from the very instructive first answer in Use of nested cross-validation). Said otherwise, it seems similar to tuning hyper parameters without going through nested cross-validation.

  • $\begingroup$ The proposal seems to vague to really comment. Generally if you don't need a reliable performance estimate, then you don't need nested cross-validation (provided your models have approximately the same number of hyper-parameters to tune of comparable sensitivity). If you do need an unbiased performance estimate, then you need some data that hasn't been used in any way, even indirectly via the operator, to make any choices about the model. $\endgroup$ – Dikran Marsupial Aug 19 '16 at 17:30
  • $\begingroup$ Let's assume I need an unbiased performance estimate to choose a model. In that case, what you suggest is that for a given use case, to know whether for example, random forest or svm should be applied, either the find_best_model chooses it, or another set of data should be used to make that choice ? If that's so, isn't it better to let 'find_best_model' make that choice ? $\endgroup$ – yetanotherion Aug 19 '16 at 18:41
  • $\begingroup$ You don't need an unbiased performance estimate to choose a model (you can add an arbitrary constant to the estimates and you will still choose the same model). Ideally you want a low variance estimator, so that the bias introduced by using it to tune the hyper-parameters will be small and hopefully approximately the same for all models under consideration (i.e. the same number of hyper-parameters and the number of hyper-parameters being small, similar sensitivity to hyper-parameter settings). $\endgroup$ – Dikran Marsupial Aug 20 '16 at 14:18
  • $\begingroup$ Thanks for the answer. What I still don't get though, is why on one hand we can expect that choosing svm over random forest may generate a negligible bias, and on the other hand, tuning the parameter of one of the models will not. We can't have a low variance estimator to choose hyperparameters ? Moreoever, if we add the 'choice of the number of features' to the issue: How can we know (besides experience/asking questions to experts) whether we can have a low variance estimator to choose the number of features ? $\endgroup$ – yetanotherion Aug 21 '16 at 17:47
  • $\begingroup$ optimising the hyper-parameters of a model via cross-validation will always mean the cross-validation estimate becomes biased (as part of the reduction in CV error will be due to the random sampling, rather than true improvement in generalisation). However, the SVM tends to be more sensitive to its hyper-parameter settings than the random forest, so the bias will tend to be greater for the SVM than RF. We do want a low variance estimator, and can get one (e.g. bootstrapping) but it is usually expensive. Feature selection usually makes these problems much worse. $\endgroup$ – Dikran Marsupial Aug 22 '16 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.