1
$\begingroup$

We have a dataset of 25 observations with ~250 measurements on each. However, these are highly correlated and doing PCA on them shows that we have at most 6 to 8 effective components to work with.

I found that Partial Least Squares is a suitable technique for this type of situation, but after training such a model via caret, my cross-validated RMSE is over twice of what they got with their forward stepwise p-value based (corrected for multiple comparisons) variable selection method.

Below you may see correlations for the whole dataset as well as those for the variables that my colleague ended up selecting.

data correlation

It is my understanding that high correlations such as those can make a model unstable, but it remained the top performer after cross validation and the coefficients themselves aren't greatly affected by dropping any single observation.

Given the above, could we say that we have sufficiently tested the model? Are those highly correlated covariates not an issue in light of the model's stability?

(incidentally, while the outcome is non-negative and left skewed, gaussian errors also gave the best results after having tried a bunch of other transformations and error families)

EDIT, adding from comment below:

We care about predictions. Capping negative predictions at zero will work just fine for our purposes. I did try horseshoe and elastic net, neither resulted in as good performance.

Anything reasonable I've tried gives me at best a RMSE that is about twice as large as the stepwise fitted model.

$\endgroup$
  • 4
    $\begingroup$ Whats the purpose of the model? Is it being used to predict something, estimate causal effects, etc... Just wondering whether impossible predicted values would be a problem. Also have you tried ridge/lasso/elasticnet (latter two if you want dimension reduction), or bayes with shorshoe priors? These seem to outperform stepwise methods in the sims I've seen (in terms of recovering true parameter estimates). $\endgroup$ – Tdisher Apr 30 '18 at 16:44
  • 3
    $\begingroup$ We care about predictions. Capping negative predictions at zero will work just fine for our purposes. I did try horseshoe and elastic net, neither resulted in as good performance. $\endgroup$ – zipzapboing Apr 30 '18 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.