Multicollinearity and Regression models

Question

We're trying to build a statistical model that predicts hand weight based upon measurements derived from photos of hands. There are about 50 individual measurements per hand, but the measurements are obviously highly correlated. For example, tip of ring finger to wrist is obviously highly correlated with tip of middle finger to wrist, etc. Collinearity diagnostics are obviously off the charts.

We tried to simplify the model by doing a PCA to cluster the individual measurements so we didn't have 50 highly correlated predictors. We ended up with 5 PC's but most of the variables loaded onto only one PC (assuming due to the very high correlation between the measurements).

The real concern here is overfitting of our model. The difference between the R-Squared and Adjusted R-Squared is quite significant no matter how we run the model.

Does anyone have any advice for me? Maybe there's some ways of dealing with the data that I'm not aware of (something other than straight forward multiple regression like PLS)? We have a pretty decent sample size (n = 800) and we're mainly concerned with predicting hand weight using the 50 measurements of the hands. Both the predictor and outcome variables are continuous.

Answer 1

The difference between the R-Squared and Adjusted R-Squared values are offer no indication of whether there is over-fitting. R-Squared is a function of the residual sum of squares(RSS) and the total sum of squares(TSS), whereas Adjusted R-Squared is a function of the residual sum of squares, the total sum of squares, and their respective degrees of freedom. With 50 explanatory variables, the RSS has significantly fewer degrees of freedom than the TSS. The difference between R-Squared and Adjusted R-Squared will be large no matter what.