When we fit (train) a regression model, we usually pick the best performing model (for example the one which gives the smallest RMSE). By doing this we do not take into account the correlation between model performances. For example, on a given sample, model A and B demonstrate some RMSEs. If we take another sample, model A and B will demonstrate other RMSEs. Could it be a problem? For example, imagine that in one region in parameters space models are less correlated than in another region. Therefore, models from a low correlation region demonstrate very different (uncorrelated) RMSEs and, therefore, this region has better chances to contain the best model. Or, the other way around, the models from the highly correlated region demonstrate more or less the same performance and, therefore this region has a smaller chance to contain the best model.
My question is: Are there methods to take the correlation between models into account and, in this way, make a better model selection?