I have run a series of Random Forest Regression models using different feature combinations (F-x). These feature variables are geological features. As some of these features are different ways to represent the same phenomenon or with combined features, I have separated them into a range of different feature combinations so only one feature variable for each phenomenon is represented top avoid double accounting.
For each feature set, I utilised a Random hyper-parameter search combined with K-fold cross-validation (
RandomSearchCV). The output of this process results in the best estimator and the resulting CV scores. This best estimator was then tested using a testing set (randomly sampled from the main testing set) with the testing score recorded.
The chart below shows the cross-validation and testing scores the different feature sets, sorted on Testing score.
As one moves from the left to the right of the graph, the testing score reduces, but the cross-validation score appears relatively constant (swinging around a mean of around 0.45).
It seems that as you move from the left to the right of the graph, the model appears to overfit the training data more and more. Therefore, can you infer from this that these feature combinations are just junk and have no real association with the target variable?
Can this information also be used in combination with the apparent differences between the cross-validation and testing scores? For example, feature combination F-27 has a testing score ~ 0.8 and a cross-validation score of ~0.25, yet combination F-17 has around the same testing score but has a much higher cross-validation score of ~0.65. Does this mean that F-27 is just junk, and can be removed even with a higher testing score.
As requested, a cross plot of CV and Testing score:
Future thanks for your help.