This is extension of the suggestion by Frank in the comments. Dr. Harrel please correct if I am wrong (appreciate corrections). **Your data:** #random population of 200 subjects with 1000 variables M <- matrix(rep(0,200*100),200,1000) for (i in 1:200) { set.seed(i) M[i,] <- ifelse(runif(1000)<0.5,-1,1) } rownames(M) <- 1:200 #random yvars set.seed(1234) u <- rnorm(1000) g <- as.vector(crossprod(t(M),u)) h2 <- 0.5 set.seed(234) y <- g + rnorm(200,mean=0,sd=sqrt((1-h2)/h2*var(g))) myd <- data.frame(y=y, M) **Install `rms` package and load it.** require(rms) `ols` function is used for Linear Model Estimation Using Ordinary Least Squares where can specify penalty term. As suggested below in comments I added `petrace` function. This function trace AIC and BIC vs Penalty. # using holdout (50% of the data) cross validation training.id <- sample(1:nrow(myd), round(nrow(myd)/2,0), replace = FALSE) test.id <- setdiff(1:nrow(myd), training.id) myd_train <- myd[training.id,] myd_test <- myd[test.id,] frm <- as.formula(paste("y~",paste(names(myd_train)[2:100],collapse="+"))) Important note I could not use all 1000 of the variables as the program complains if number of variable exceeds 100. Also `y~.` type formula designation did not work. So see above way of doing same creating formula object `frm` f <- ols(frm, data = myd_train, method="qr", x=TRUE, y=TRUE) p <- pentrace(f, seq(.2,1,by=.05)) Error in array(x, c(length(x), 1L), if (!is.null(names(x))) list(names(x), : 'data' must be of a vector type, was 'NULL' plot(p) "For an ordinary unpenalized fit from lrm or ols and for a vector or list of penalties, fits a series of logistic or linear models using penalized maximum likelihood estimation, and saves the effective degrees of freedom, Akaike Information Criterion (AIC), Schwarz Bayesian Information Criterion (BIC), and Hurvich and Tsai's corrected AIC (AIC_c). Optionally pentrace can use the nlminb function to solve for the optimum penalty factor or combination of factors penalizing different kinds of terms in the model." from `rms` package manual. `calibrate` function is for Resampling Model Calibration and Uses bootstrapping or cross-validation to get bias-corrected (overfitting- corrected) estimates of predicted vs. observed values based on subsetting predictions into intervals. The `validate` function does resampling validation of a regression model, with or without backward step-down variable deletion. B = number of repetitions. For method="crossvalidation", is the number of groups of omitted observations cal <- calibrate(f, method = "cross validation", B=20) plot(cal) You can use `Predict` function to compute predicted values and confidence limits. I am not sure I this works in test situation.