Potential for basic inference from comparison of cross-validation and testing scores using a Random Forest Regressor

Question

I have run a series of Random Forest Regression models using different feature combinations (F-x). These feature variables are geological features. As some of these features are different ways to represent the same phenomenon or with combined features, I have separated them into a range of different feature combinations so only one feature variable for each phenomenon is represented top avoid double accounting.

For each feature set, I utilised a Random hyper-parameter search combined with K-fold cross-validation (RandomSearchCV). The output of this process results in the best estimator and the resulting CV scores. This best estimator was then tested using a testing set (randomly sampled from the main testing set) with the testing score recorded.

The chart below shows the cross-validation and testing scores the different feature sets, sorted on Testing score.

As one moves from the left to the right of the graph, the testing score reduces, but the cross-validation score appears relatively constant (swinging around a mean of around 0.45).

It seems that as you move from the left to the right of the graph, the model appears to overfit the training data more and more. Therefore, can you infer from this that these feature combinations are just junk and have no real association with the target variable?

Can this information also be used in combination with the apparent differences between the cross-validation and testing scores? For example, feature combination F-27 has a testing score ~ 0.8 and a cross-validation score of ~0.25, yet combination F-17 has around the same testing score but has a much higher cross-validation score of ~0.65. Does this mean that F-27 is just junk, and can be removed even with a higher testing score.

As requested, a cross plot of CV and Testing score:

Future thanks for your help.

Answer 1

Unfortunately I think the only concusion you can draw here is that your optimization/selection approach failed.

For a technically successful optimization (somewhat regardless of the achieved generalization error), cross validation score and test results should be highly and positiviely correlated and that's not the case here.

One reason I can think of is that your sample size(s) may be too small. You then get large error (random uncertainty; variance) on the test results and that would de-correlate CV and test set estimates since they are from mutually independent samples and in general lead to estimates being all over the place so that possible trends in the underlying data (true model performance) may be completely masked.

Here's an illustration, left to right 20 training + 4 test, 100 train + 20 test, 1000 train + 200 test.

The underlying data has 100 features with gradually varying information content, I train linear regression with 10 features, moving those 10 features over my columns to generate the points. Error is root mean squared error, so lower is better.
With 1200 cases we get a very clear picture of the prediction quality. With 120, the picture is a bit blurred but we still see differences between the models. With only 24 cases, we cannot be sure of seeing any real differences.

My simulation doesn't have any overfitting of the cross validation (yet - because we have not yet selected a model based on the CV results), so CV error is an unbiased predictor of test error (and generalization error). With too small sample size it's nevertheless a bad predictor of generalization and of test error.

• For successful optimization/selection, CV error needs to be a sufficiently good predictor of generalization error.
• For testing, test error needs to be a sufficiently good predictor of generalization error.

---

How about other approaches to the feature selection:

• using the random forest's variable importance?
• L1-regularizing the regression (LASSO)?
• If the sample size is too small, the best you can do may be using expert geological knowledge which features you should include.