I trained set of models using RandomizedSearchCV and picked the best using .best_estimator_ and then tested on my test set. However, I would like to check how any other model from the grid performed on the test set. Is it possible to do so and if yes, how can I do that?
RandomizedSearchCV discards the actual models it trains on each fold after evaluating them, so you won't be able to extract the fitted models from the output.
RandomizedSearchCV will pick the parameters that performed best on the validation sets and re-train the model with these parameters, this time on all observations. This is the model that you extract.
That would be inefficient to do for all parameter settings that were not selected, so
RandomizedSearchCV does not retrain the second best model.
You could look at
_cv_results to find the second best parameter settings and manually train the model with these parameters on all observations. Use method
set_params() to set the parameters to the second or third best parameter set in
I would not recommend this procedure to choose a set of parameters, since it negates the reason to do parameter selection using CV rather than your test set. You are likely overfit your parameter choice to the test data.