As far as I understood,  when implementing a learning algorithm that integrates model selection/hyper parameters tuning in itself, nested cross-validation is necessary to lower the bias in the performance estimate.

I’d propose to summarize the algorithm to compute the estimation of that performance as:

`performances = []
for training, test in partition(data):
    model = find_best_model(data_to_choose_best_model=partition(training))
    performances.append(model.fit_and_measure_performance(training, test))
return some_method_to_aggregate_for_ex_average(performances)`

As we don’t have an infinite amount of time, we’re obliged to restrict the number of model/parameters to browse during `find_best_model`. Taking aside the fact that we don’t use the models we don’t know, I’d enumerate two ways of selecting that subset of model/parameters:

 1. experience/gut feeling,
 2. exploration/plotting some curves to evaluate how an algorithm reacts to a given data.

My question is the following:
Is there is a way to implement 2., for example, in the way to select/explore the data, that would permit lowering the bias it creates ? 

Indeed, implementing 2 ourselves, i.e. out of the “find_best_model” method in the algorithm above, seems to be a “seemingly benign short cut” that may induce a non negligible “magnitude of [...] bias” (taking expressions from the very instructive first answer in http://stats.stackexchange.com/questions/103828/use-of-nested-cross-validation). Said otherwise, it seems similar to tuning hyper parameters without going through nested cross-validation.