I've recently picked up attention for AutoML algorithms; Meta-algorithms that intelligently search the space of machine learning models to find the "pipeline" (preprocessing/feature selection method/prediction technique etc.) that attains the best predictive power on a certain classification/regression task. Examples are:
In my understanding, these techniques evaluate the accuracy of a certain set of pipelines using cross validation, and then perform some kind of directed search across the space of pipelines, based on the obtained results, to find pipelines that achieve a better cross validation error (using the same split in the data). However, using a certain training sample to evaluate a pipeline, then improving the pipeline based on the results, and evaluating on the same sample (although with cross validation, using the same split) feels like upwards biasing the cross validation error as estimator for the true predictive performance, or 'overfitting the selection criterion'. This effect is also described in the paper: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation
However, in a hypothetical situation where we would have infinite computing power, we could perform a grid search across all different pipeline configurations and parameters, evaluate using cross validation, and pick the one with the best performance. (Although we would need to evaluate the actual performance using for example nested cross-validation) something like this is often done in practice, and not commonly seen as 'overfitting the selection criterion' .
Although both methods could eventually lead to the same performance estimates for the same pipelines and this could select the same pipeline, the first feels like 'overfitting the selection criterion', while the latter does not. I am struggling to see the difference or the similarity between the two. Is the grid search overfitting as well or are the methods fundamentally different? Some views/discussion/insight on this would be appreciated.
Note: i am sure both AutoML algorithms have mechanisms in place to avoid such an overfit, and the purpose of this question is not to discuss a certain specific algorithm, but more the general concepts of overfitting during model selection.