We have a dataset of 25 observations with ~250 measurements on each. However, these are highly correlated and doing PCA on them shows that we have at most 6 to 8 effective components to work with.
I found that Partial Least Squares is a suitable technique for this type of situation, but after training such a model via
caret, my cross-validated RMSE is over twice of what they got with their forward stepwise p-value based (corrected for multiple comparisons) variable selection method.
Below you may see correlations for the whole dataset as well as those for the variables that my colleague ended up selecting.
It is my understanding that high correlations such as those can make a model unstable, but it remained the top performer after cross validation and the coefficients themselves aren't greatly affected by dropping any single observation.
Given the above, could we say that we have sufficiently tested the model? Are those highly correlated covariates not an issue in light of the model's stability?
(incidentally, while the outcome is non-negative and left skewed, gaussian errors also gave the best results after having tried a bunch of other transformations and error families)
EDIT, adding from comment below:
We care about predictions. Capping negative predictions at zero will work just fine for our purposes. I did try horseshoe and elastic net, neither resulted in as good performance.
Anything reasonable I've tried gives me at best a RMSE that is about twice as large as the stepwise fitted model.