Too good to be true? Ridge prediction

Question

I have a small data set of 18 persons. I have an outcome variable Y, and 200 predictors. These predictors were chosen based on biology and prior data.

I used the caret R package and split the data set in two, a training set of 10 persons and a test set of the remaining 8 persons.

I then trained a RIDGE model using glmnet on the 10 persons with 4-fold cross validation. I then predicted the outcome on the remaining 8 persons using the trained model.

The correlation between actual Y and predicted Y was r=0.933, which is amazingly good. Too good to be true?

Heres my codes:

set.seed(7)
a <- createDataPartition(dfpred$Y, list=FALSE)
training <- dfpred[a,]
test <- dfpred[-a,]

library(mlbench)
library(caret)

control <- trainControl(method = "cv", number = 4)

set.seed(7)
modelRIDGE <- train(Y~., data=training, method="glmnet", trControl=control, tuneGrid = expand.grid(alpha = 0, lambda = 0))

MEA: 0.9280647
RMSE: 1.071609
R squared: 0.9748333

hm=predict(object=modelRIDGE,newdata=test[,-1])
cor.test(hm,test$Y) # r=0.933 p<0.001

This looks very very good. So good that I worry I have overlooked something? Any input would help, thank you!

A note on the data: Its clinical relevant basic research on difficult/rare human species. The data are really unique and tries på predict body glucose uptake based on serum protein concentrations. The methods are really robust and accurate. The sample size is small, but there are no larger data sets of this type available anywhere.

I also tested some other models:

Answer 1

Split sample validation can require up to 20,000 observations to perform well enough. Otherwise the results may depend dramatically on the luck of the split. That’s why 100 repeats of 10-fold cross-validation or several hundred bootstrap resamples provide better estimates of likely model performance in most cases.

The number of observations needed to only estimate a residual variances is 70. The number of observations needed only to estimate the intercept in a binary logistic model is 96. Estimation of a single correlation coefficient with precision requires 300 observations. Those are much easier tasks that what you are doing, so there is no hope for looking at 200 predictors with your sample size.

This lack of hope is cemented by considering that ridge regression like lasso and elastic net depend on your being able to choose the value of penalty parameters, usually done through cross-validation. The minimum sample size needed to reliably choose a penalty parameter is at least in the hundreds.

The only hope is to ask questions of the data that the data are capable of answering. This involves using data reduction (unsupervised learning) on the 200 features to reduce them to one or two numbers that you can then play against Y. Principal components analysis is often used for this purpose. The PCs will be very unstable, but not unstable in a way that makes prediction of Y optimistic.

These issues are dealt with in detail in RMS.

Variable importance is a misleading measure in this context unless you get confidence intervals for the importance measures. If you run the bootstrap exercise at the end of this on your data, you’ll find that the data are consistent with the “winning” feature also being the biggest loser.

Answer 2

A quick-and-dirty demonstration might be instructive to supplement Frank's answer:

n <- 18
control <- caret::trainControl(method = "cv", number = 4)
set.seed(1)

res <- replicate(1000, {
  dfpred <- data.frame(Y = rnorm(n))
  
  for(i in 1:200) dfpred[paste0("X", i)] <- rnorm(n)
  
  a <- caret::createDataPartition(dfpred$Y, list=FALSE)
  training <- dfpred[a,]
  test <- dfpred[-a,]
  
  modelRIDGE <- caret::train(Y~., data=training, method="glmnet",
                             trControl=control,
                             tuneGrid = expand.grid(alpha = 0, lambda = 0))
  
  hm=predict(object=modelRIDGE,newdata=test[,-1])
  cor.test(hm,test$Y)$estimate
})

As you can see even total random noise can pretty easily give values >0.9:

Here we have two results above >0.9 in absolute value (one is even larger than yours, with 0.94). Again, from pure noise with only 1000 simulations - thus, having a value of 0.93 doesn't really mean that much, and it is unfortunately surely not "amazingly good" at least in my view in the light of the above simulation.