As far as I understood, when implementing a learning algorithm that integrates model selection/hyper parameters tuning in itself, nested cross-validation is necessary to lower the bias in the performance estimate.
I’d propose to summarize the algorithm to compute the estimation of that performance as:
performances = [] for training, test in partition(data): model = find_best_model(data_to_choose_best_model=partition(training)) performances.append(model.fit_and_measure_performance(training, test)) return some_method_to_aggregate_for_ex_average(performances)
As we don’t have an infinite amount of time, we’re obliged to restrict the number of model/parameters to browse during find_best_model. Taking aside the fact that we don’t use the models we don’t know, I’d enumerate two ways of selecting that subset of model/parameters:
- experience/gut feeling,
- exploration/plotting some curves to evaluate how an algorithm reacts to a given data.
My question is the following: Is there is a way to implement 2., for example, in the way to select/explore the data, that would permit lowering the bias it creates ?
Indeed, implementing 2 ourselves, i.e. out of the “find_best_model” method in the algorithm above, seems to be a “seemingly benign short cut” that may induce a non negligible “magnitude of [...] bias” (taking expressions from the very instructive first answer in Use of nested cross-validation). Said otherwise, it seems similar to tuning hyper parameters without going through nested cross-validation.
