Return to Question

As far as I understood, when implementing a learning algorithm that integrates model selection/hyper parameters tuning in itself, nested cross-validation is necessary to lower the bias in the performance estimate.

I’d propose to summarize the algorithm to compute the estimation of that performance as:

performances = [] for training, test in partition(data): model = find_best_model(data_to_choose_best_model=partition(training)) performances.append(model.fit_and_measure_performance(training, test)) return some_method_to_aggregate_for_ex_average(performances)

As we don’t have an infinite amount of time, we’re obliged to restrict the number of model/parameters to browse during find_best_model. Taking aside the fact that we don’t use the models we don’t know, I’d enumerate two ways of selecting that subset of model/parameters:

1. experience/gut feeling,
2. exploration/plotting some curves to evaluate how an algorithm reacts to a given data.

My question is the following: Is there is a way to implement 2., for example, in the way to select/explore the data, that would permit lowering the bias it creates ?

Indeed, implementing 2 ourselves, i.e. out of the “find_best_model” method in the algorithm above, seems to be a “seemingly benign short cut” that may induce a non negligible “magnitude of [...] bias” (taking expressions from the very instructive first answer in Use of nested cross-validation). Said otherwise, it seems similar to tuning hyper parameters without going through nested cross-validation.

As far as I understood, when implementing a learning algorithm that integrates model selection/hyper parameters tuning in itself, nested cross-validation is necessary to lower the bias in the performance estimate.

I’d propose to summarize the algorithm to compute the estimation of that performance as:

performances = [] for training, test in partition(data): model = find_best_model(data_to_choose_best_model=partition(training)) performances.append(model.fit_and_measure_performance(training, test)) return some_method_to_aggregate_for_ex_average(performances)

As we don’t have an infinite amount of time, we’re obliged to restrict the number of model/parameters to browse during find_best_model. Taking aside the fact that we don’t use the models we don’t know, I’d enumerate two ways of selecting that subset of model/parameters:

1. experience/gut feeling,
2. exploration/plotting some curves to evaluate how an algorithm reacts to a given data.

My question is the following: Is there is a way to implement 2., for example, in the way to select/explore the data, that would permit lowering the bias it creates ?

Indeed, implementing 2 ourselves, i.e. out of the “find_best_model” method in the algorithm above, seems to be a “seemingly benign short cut” that may induce a non negligible “magnitude of [...] bias” (taking expressions from the very instructive first answer in Use of nested cross-validation). Said otherwise, it seems similar to tuning hyper parameters without going through nested cross-validation.

As far as I understood, when implementing a learning algorithm that integrates model selection/hyper parameters tuning in itself, cross nested validation is necessary to lower the bias in the estimation of performance.

I’d propose to summarize the algorithm to compute the estimation of that performance as:

performances = [] for training, test in partition(data): model = find_best_model(data_to_choose_best_model=partition(training)) performances.append(model.fit_and_measure_performance(training, test)) return some_method_to_aggregate_for_ex_average(performances)

As we don’t have an infinite amount of time, we’re obliged to restrict the number of model/parameters to browse during find_best_model. Taking aside the fact that we don’t use the models we don’t know, I’d enumerate two ways of selecting that subset of model/parameters:

1. experience/gut feeling,
2. exploration/plotting some curves to evaluate how an algorithm reacts to a given data.

My question is the following: Is there is a way to implement 2, for example, in the way to select/explore the data, that would permit lowering the bias it creates ?

Indeed, implementing 2 ourselves, i.e. out of the “find_best_model” method in the algorithm above, seems to be a “seemingly benign short cut” that may induce a non negligible “magnitude of [...] bias” (taking expressions from the very instructive first answer in Use of nested cross-validation). Said otherwise, it seems similar to tuning hyper parameters without going through nested cross-validation.

As far as I understood, when implementing a learning algorithm that integrates model selection/hyper parameters tuning in itself, nested cross-validation is necessary to lower the bias in the performance estimate.

I’d propose to summarize the algorithm to compute the estimation of that performance as:

performances = [] for training, test in partition(data): model = find_best_model(data_to_choose_best_model=partition(training)) performances.append(model.fit_and_measure_performance(training, test)) return some_method_to_aggregate_for_ex_average(performances)

As we don’t have an infinite amount of time, we’re obliged to restrict the number of model/parameters to browse during find_best_model. Taking aside the fact that we don’t use the models we don’t know, I’d enumerate two ways of selecting that subset of model/parameters:

1. experience/gut feeling,
2. exploration/plotting some curves to evaluate how an algorithm reacts to a given data.

My question is the following: Is there is a way to implement 2., for example, in the way to select/explore the data, that would permit lowering the bias it creates ?

Indeed, implementing 2 ourselves, i.e. out of the “find_best_model” method in the algorithm above, seems to be a “seemingly benign short cut” that may induce a non negligible “magnitude of [...] bias” (taking expressions from the very instructive first answer in Use of nested cross-validation). Said otherwise, it seems similar to tuning hyper parameters without going through nested cross-validation.