I am trying incorporate a formal strategy to find the most optimal set hyper-parameters for a machine learning algorithm. I understand you can either do a grid-search or a k-fold cross validation, among many other possible search options.
I came across
scikit-learn's grid-search and cross-validated estimators in here. It says
by default, the GridSearchCV uses a 3-fold cross-validation. Seems to me they are combining grid-search and cross-validation together. Does it mean the k-fold validation is done on each grid-search combination? If so, what is the advantage of that? Is it for more stable estimator? And is this combined grid-search and k-fold CV almost always better than just grid-search only or just k-fold CV only in finding the optimal hyper-parameters?