# Elastic Net: How to get more sparsity than “lambda.1se” in R package glmnet

Package glmnet provides a cross validation function called cv.glmnet that allows us to choose between two suggested models (from the many), labelled "lambda.min" and "lambda.1se". However even "lambda.1se" does not provide enough sparsity for me, as can be seen in the example below:

Here is a 27x27 variance-covariance matrix, from real life, based on foreign exchange market returns (sample size 5 years), from which I have reconstructed some data using the MASS package's mvrnorm function.

rr <- matrix(c(+3.905427e-04,-1.672944e-04,-3.868682e-05,-2.012400e-05,-5.208216e-05,-1.676518e-05,+3.191132e-05,-8.393085e-05,+2.994233e-05,+4.142394e-05,
+2.788218e-05,+2.100455e-05,+8.915135e-06,+3.237941e-05,-1.370915e-05,-1.869792e-05,-2.818437e-05,+4.867242e-06,-1.484061e-06,-9.871850e-07,
-2.301505e-05,+1.239755e-05,+2.707801e-06,-2.730393e-05,-3.439632e-05,-2.850229e-06,-1.212873e-06,-1.672944e-04,+3.148921e-04,+3.981354e-05,
-2.159599e-05,-7.679438e-06,+7.327542e-07,+8.133157e-06,+1.944368e-05,-3.291582e-05,-2.830392e-05,-9.711673e-06,-1.953129e-05,-8.678863e-06,
-1.468040e-05,-2.999372e-05,-8.443994e-06,-3.243783e-06,+8.457205e-06,-3.331677e-05,-2.585784e-05,-9.963355e-06,-1.006820e-05,-1.510451e-05,
-2.297489e-05,-2.822061e-05,-1.058895e-05,-1.523963e-05,-3.868682e-05,+3.981354e-05,+6.136468e-05,+6.385946e-06,+5.402061e-06,+1.340335e-06,
-2.120207e-05,-1.004486e-05,-2.145629e-06,-2.930022e-06,-2.912909e-06,-1.334397e-06,-1.120393e-06,-3.215213e-06,-3.748124e-06,-5.266702e-07,
+1.385250e-06,+2.594281e-06,-2.793294e-06,-3.918135e-06,+3.701831e-06,-1.357614e-06,+1.050664e-06,+1.330178e-06,-1.893086e-06,-8.355664e-07,
-9.429525e-07,-2.012400e-05,-2.159599e-05,+6.385946e-06,+4.514971e-05,+1.487041e-05,+4.051007e-06,-2.579603e-05,-7.896501e-06,+3.735320e-06,
-8.991534e-07,-1.909784e-06,-2.652713e-07,-3.995973e-07,-1.947112e-06,+1.223827e-05,+2.273053e-06,+5.392409e-06,-4.957146e-07,+6.043096e-07,
+1.373694e-06,+7.145170e-06,+4.236828e-08,+2.855058e-06,+1.196767e-05,+6.420234e-06,+3.212209e-06,+5.828089e-06,-5.208216e-05,-7.679438e-06,
+5.402061e-06,+1.487041e-05,+3.968051e-05,+8.180625e-06,-1.679490e-05,+1.438802e-06,-9.333792e-07,-3.856856e-06,-4.196170e-06,-2.545341e-06,
-1.465675e-06,-4.679705e-06,+5.273817e-06,+4.392979e-06,+6.707402e-06,-1.777882e-06,+3.340004e-06,+3.041857e-06,+1.046557e-05,-1.228042e-06,
+3.006442e-06,+1.042182e-05,+6.513678e-06,+2.545036e-06,+3.286945e-06,-1.676518e-05,+7.327542e-07,+1.340335e-06,+4.051007e-06,+8.180625e-06,
+4.515972e-06,-2.384442e-06,+2.346723e-06,-6.908337e-07,-1.428231e-06,-1.410132e-06,-1.144762e-06,-4.898731e-07,-1.704683e-06,+1.394768e-07,
+7.696074e-07,+1.714821e-06,-4.269990e-07,+3.351277e-07,+5.123849e-07,+2.740856e-06,-4.688568e-07,+3.553067e-07,+2.336701e-06,+5.732269e-07,
+3.052545e-07,+2.869335e-07,+3.191132e-05,+8.133157e-06,-2.120207e-05,-2.579603e-05,-1.679490e-05,-2.384442e-06,+1.496400e-04,+2.803930e-05,
-7.596035e-06,-2.577285e-06,-1.057582e-06,-1.404342e-05,-5.620170e-06,-1.767036e-06,-3.168803e-05,-1.800529e-05,-1.292217e-05,-4.453331e-06,
-3.319978e-05,-1.786611e-05,-1.284912e-05,+4.276601e-07,-1.347161e-05,-3.513421e-05,-2.668011e-05,-1.162202e-05,-1.245654e-05,-8.393085e-05,
+1.944368e-05,-1.004486e-05,-7.896501e-06,+1.438802e-06,+2.346723e-06,+2.803930e-05,+1.250962e-04,-8.983686e-06,-1.080656e-05,-5.574779e-06,
-1.244544e-05,-5.308212e-06,-9.088221e-06,-1.134928e-05,-7.996861e-06,-1.762774e-06,-8.772736e-06,-6.480971e-06,-9.015757e-06,-9.206922e-06,
-5.871784e-06,-5.546947e-06,-9.127687e-06,-1.096436e-05,-5.814707e-06,-7.572505e-06,+2.994233e-05,-3.291582e-05,-2.145629e-06,+3.735320e-06,
-9.333792e-07,-6.908337e-07,-7.596035e-06,-8.983686e-06,+1.254965e-05,+7.247157e-06,+1.709803e-06,+2.861585e-06,+1.100077e-06,+2.904772e-06,
+4.547653e-06,+2.510788e-07,-6.168826e-07,-3.769066e-07,+1.891723e-06,+2.750163e-06,-8.707644e-07,+1.931014e-06,+1.641571e-06,+2.066175e-06,
+7.301858e-07,+1.004776e-06,+2.087421e-06,+4.142394e-05,-2.830392e-05,-2.930022e-06,-8.991534e-07,-3.856856e-06,-1.428231e-06,-2.577285e-06,
-1.080656e-05,+7.247157e-06,+1.374557e-05,+2.806961e-06,+3.332886e-06,+1.514102e-06,+3.783409e-06,-1.389411e-06,-1.112583e-06,-2.430049e-06,
+2.748058e-07,+5.139875e-07,+1.300466e-06,-2.462923e-06,+1.697307e-06,+1.043461e-06,-7.175271e-07,-2.794225e-06,-1.512459e-07,+2.815655e-07,
+2.788218e-05,-9.711673e-06,-2.912909e-06,-1.909784e-06,-4.196170e-06,-1.410132e-06,-1.057582e-06,-5.574779e-06,+1.709803e-06,+2.806961e-06,
+5.153962e-06,+3.137140e-06,+1.442756e-06,+2.461616e-06,-1.514675e-06,-2.032794e-06,-1.900112e-06,+3.598444e-07,+1.077534e-06,-6.946718e-07,
-2.577196e-06,+2.510696e-07,+4.922748e-07,-1.789320e-06,-3.410673e-06,-1.516124e-07,-5.658895e-07,+2.100455e-05,-1.953129e-05,-1.334397e-06,
-2.652713e-07,-2.545341e-06,-1.144762e-06,-1.404342e-05,-1.244544e-05,+2.861585e-06,+3.332886e-06,+3.137140e-06,+1.322267e-05,+3.742704e-06,
+2.897316e-06,+2.190498e-06,+2.205173e-06,+1.143671e-06,+6.378957e-07,+5.430563e-06,+1.286320e-06,+2.526458e-07,+2.177544e-07,+2.828000e-06,
+4.718304e-06,+1.089443e-06,+1.589336e-06,+9.667085e-07,+8.915135e-06,-8.678863e-06,-1.120393e-06,-3.995973e-07,-1.465675e-06,-4.898731e-07,
-5.620170e-06,-5.308212e-06,+1.100077e-06,+1.514102e-06,+1.442756e-06,+3.742704e-06,+3.252501e-06,+1.140401e-06,+9.556437e-07,+1.074267e-06,
+6.025770e-07,+2.229803e-07,+2.229262e-06,+7.350322e-07,-1.697000e-07,-3.558592e-08,+1.125714e-06,+1.675207e-06,+4.637744e-07,+7.091632e-07,
+1.517128e-07,+3.237941e-05,-1.468040e-05,-3.215213e-06,-1.947112e-06,-4.679705e-06,-1.704683e-06,-1.767036e-06,-9.088221e-06,+2.904772e-06,
+3.783409e-06,+2.461616e-06,+2.897316e-06,+1.140401e-06,+4.663585e-06,-6.658251e-07,-6.095897e-07,-2.046872e-06,+6.571694e-07,+7.337406e-07,
+4.691496e-07,-2.068054e-06,+1.187547e-06,+5.255661e-07,-1.547152e-06,-2.647444e-06,+1.106522e-07,+4.997985e-08,-1.370915e-05,-2.999372e-05,
-3.748124e-06,+1.223827e-05,+5.273817e-06,+1.394768e-07,-3.168803e-05,-1.134928e-05,+4.547653e-06,-1.389411e-06,-1.514675e-06,+2.190498e-06,
+9.556437e-07,-6.658251e-07,+5.056387e-05,+5.733135e-06,+4.876198e-06,-7.288427e-07,+8.635461e-06,+6.028157e-06,-1.714284e-06,+1.581347e-08,
+3.218589e-06,+1.109427e-05,+1.044090e-05,+4.586428e-06,+6.845056e-06,-1.869792e-05,-8.443994e-06,-5.266702e-07,+2.273053e-06,+4.392979e-06,
+7.696074e-07,-1.800529e-05,-7.996861e-06,+2.510788e-07,-1.112583e-06,-2.032794e-06,+2.205173e-06,+1.074267e-06,-6.095897e-07,+5.733135e-06,
+2.044822e-05,+5.782109e-06,+3.019946e-07,+6.052292e-06,+4.901258e-06,+3.139930e-06,-7.229161e-08,+2.063215e-06,+9.119608e-06,+5.919809e-06,
+2.508532e-06,+1.952830e-06,-2.818437e-05,-3.243783e-06,+1.385250e-06,+5.392409e-06,+6.707402e-06,+1.714821e-06,-1.292217e-05,-1.762774e-06,
-6.168826e-07,-2.430049e-06,-1.900112e-06,+1.143671e-06,+6.025770e-07,-2.046872e-06,+4.876198e-06,+5.782109e-06,+1.050558e-05,-4.464648e-07,
+3.715678e-06,+2.971496e-06,+4.662022e-06,-2.933217e-07,+1.929764e-06,+8.453756e-06,+6.938162e-06,+2.054123e-06,+2.107770e-06,+4.867242e-06,
+8.457205e-06,+2.594281e-06,-4.957146e-07,-1.777882e-06,-4.269990e-07,-4.453331e-06,-8.772736e-06,-3.769066e-07,+2.748058e-07,+3.598444e-07,
+6.378957e-07,+2.229803e-07,+6.571694e-07,-7.288427e-07,+3.019946e-07,-4.464648e-07,+6.154732e-06,-4.826199e-07,+7.972120e-08,-1.076161e-08,
+3.200496e-07,+5.782623e-08,-2.955595e-07,-2.241002e-06,-1.286211e-07,-1.497812e-08,-1.484061e-06,-3.331677e-05,-2.793294e-06,+6.043096e-07,
+3.340004e-06,+3.351277e-07,-3.319978e-05,-6.480971e-06,+1.891723e-06,+5.139875e-07,+1.077534e-06,+5.430563e-06,+2.229262e-06,+7.337406e-07,
+8.635461e-06,+6.052292e-06,+3.715678e-06,-4.826199e-07,+4.450922e-05,+1.085514e-05,+5.123687e-06,+5.368289e-07,+7.298413e-06,+1.028015e-05,
+4.371002e-06,+3.204427e-06,+2.715323e-06,-9.871850e-07,-2.585784e-05,-3.918135e-06,+1.373694e-06,+3.041857e-06,+5.123849e-07,-1.786611e-05,
-9.015757e-06,+2.750163e-06,+1.300466e-06,-6.946718e-07,+1.286320e-06,+7.350322e-07,+4.691496e-07,+6.028157e-06,+4.901258e-06,+2.971496e-06,
+7.972120e-08,+1.085514e-05,+2.321785e-05,+6.428409e-06,+1.349267e-06,+5.085592e-06,+6.280345e-06,+5.900478e-06,+2.167341e-06,+2.780337e-06,
-2.301505e-05,-9.963355e-06,+3.701831e-06,+7.145170e-06,+1.046557e-05,+2.740856e-06,-1.284912e-05,-9.206922e-06,-8.707644e-07,-2.462923e-06,
-2.577196e-06,+2.526458e-07,-1.697000e-07,-2.068054e-06,-1.714284e-06,+3.139930e-06,+4.662022e-06,-1.076161e-08,+5.123687e-06,+6.428409e-06,
+2.909737e-05,+1.584565e-06,+5.282449e-06,+6.901083e-06,+5.037513e-06,+2.245936e-06,+2.913474e-06,+1.239755e-05,-1.006820e-05,-1.357614e-06,
+4.236828e-08,-1.228042e-06,-4.688568e-07,+4.276601e-07,-5.871784e-06,+1.931014e-06,+1.697307e-06,+2.510696e-07,+2.177544e-07,-3.558592e-08,
+1.187547e-06,+1.581347e-08,-7.229161e-08,-2.933217e-07,+3.200496e-07,+5.368289e-07,+1.349267e-06,+1.584565e-06,+1.202174e-05,+6.348587e-07,
+7.476531e-07,-3.430022e-07,-2.735203e-08,+1.029644e-06,+2.707801e-06,-1.510451e-05,+1.050664e-06,+2.855058e-06,+3.006442e-06,+3.553067e-07,
-1.347161e-05,-5.546947e-06,+1.641571e-06,+1.043461e-06,+4.922748e-07,+2.828000e-06,+1.125714e-06,+5.255661e-07,+3.218589e-06,+2.063215e-06,
+1.929764e-06,+5.782623e-08,+7.298413e-06,+5.085592e-06,+5.282449e-06,+6.348587e-07,+5.694378e-06,+4.881555e-06,+3.197262e-06,+1.468878e-06,
+1.591542e-06,-2.730393e-05,-2.297489e-05,+1.330178e-06,+1.196767e-05,+1.042182e-05,+2.336701e-06,-3.513421e-05,-9.127687e-06,+2.066175e-06,
-7.175271e-07,-1.789320e-06,+4.718304e-06,+1.675207e-06,-1.547152e-06,+1.109427e-05,+9.119608e-06,+8.453756e-06,-2.955595e-07,+1.028015e-05,
+6.280345e-06,+6.901083e-06,+7.476531e-07,+4.881555e-06,+3.021823e-05,+1.387415e-05,+5.421927e-06,+6.312873e-06,-3.439632e-05,-2.822061e-05,
-1.893086e-06,+6.420234e-06,+6.513678e-06,+5.732269e-07,-2.668011e-05,-1.096436e-05,+7.301858e-07,-2.794225e-06,-3.410673e-06,+1.089443e-06,
+4.637744e-07,-2.647444e-06,+1.044090e-05,+5.919809e-06,+6.938162e-06,-2.241002e-06,+4.371002e-06,+5.900478e-06,+5.037513e-06,-3.430022e-07,
+3.197262e-06,+1.387415e-05,+6.129212e-05,+4.943961e-06,+6.180768e-06,-2.850229e-06,-1.058895e-05,-8.355664e-07,+3.212209e-06,+2.545036e-06,
+3.052545e-07,-1.162202e-05,-5.814707e-06,+1.004776e-06,-1.512459e-07,-1.516124e-07,+1.589336e-06,+7.091632e-07,+1.106522e-07,+4.586428e-06,
+2.508532e-06,+2.054123e-06,-1.286211e-07,+3.204427e-06,+2.167341e-06,+2.245936e-06,-2.735203e-08,+1.468878e-06,+5.421927e-06,+4.943961e-06,
+6.807734e-06,+2.987841e-06,-1.212873e-06,-1.523963e-05,-9.429525e-07,+5.828089e-06,+3.286945e-06,+2.869335e-07,-1.245654e-05,-7.572505e-06,
+2.087421e-06,+2.815655e-07,-5.658895e-07,+9.667085e-07,+1.517128e-07,+4.997985e-08,+6.845056e-06,+1.952830e-06,+2.107770e-06,-1.497812e-08,
+2.715323e-06,+2.780337e-06,+2.913474e-06,+1.029644e-06,+1.591542e-06,+6.312873e-06,+6.180768e-06,+2.987841e-06,+9.474784e-06), 27, 27)
colnames(rr) <- rownames(rr) <- c("USD","EUR","GBP","CAD","AUD","NZD","JPY","XAU","NOK","SEK","CZK","PLN","HUF","RON","RUB","TRY","ZAR","ILS","INR","IDR","KRW","TWD","PHP",
"MXN","BRL","CLP","COP")
library("MASS")
set.seed(1234)
data = mvrnorm(260 * 5, rep(0, 27), rr)


Clearly principal component 1 is EUR vs the USD, so EURUSD. This will be easy to replicate in the market with a single transaction, namely, EURUSD:

barplot(eigen(cov(data))$vectors[, 1], names.arg = colnames(data), las = 2) title("PC1")  PC2 however, which I interpret as "emerging markets + commodity currencies", is more interesting, and would need lots of transactions to replicate accurately: barplot(eigen(cov(data))$vectors[, 2], names.arg = colnames(data), las = 2)
title("PC2")


So in an attempt to lower the transaction costs, I want a sparse fit for PC2. Enter glmnet:

library("glmnet")
cvmodel <- cv.glmnet(x = data, y = data %*% eigen(cov(data))$vectors[, 2]) barplot(coef(cvmodel, s = "lambda.1se")[-1], names.arg = colnames(data), las = 2, main = "cv.glmnet lambda.1se coefficients")  As you can see I really haven't obtained much sparsity, and if I plot actual PC2 vs my "sparse" PC2, I get an altogether too good fit. plot(data %*% eigen(cov(data))$vectors[, 2], data %*% coef(cvmodel, s = "lambda.1se")[-1], main = "actual vs cv.glmnet")


So my question is, how do I get a "worse" fit, using much more sparse model, using glmnet, in a rigorous way? I know the fit provides a whole bunch of lambda values but I don't know how to choose a worse one that still suits my purposes. Could I go through each one, create an lm each time, and choose the first with an r-squared for example that's better than 0.8, say? What more efficient strategy can I use here?

• See sparse principal components in R packages "PMA" (newer and recommended) and "elasticnet" (older), that will give you a more direct solution. You will find citations of research papers in the documentation of the main functions of these packages. – Richard Hardy Aug 9 '16 at 20:23

The most rigorous way to get a more sparse model is to just increase the size of $\lambda$. That's all. Larger $\lambda$ will shrink the coefficients more towards zero, which causes more of them to be pinned at 0, which is precisely the sparsity property that you want. So pick the value of $\lambda$ that corresponds to how much sparseness you want; sparisty of each solution is reported by the nzero part of the returned cv.glmnet object.
• It's nzero in the returned cv.glmnet object. This is discussed in the documentation. – Sycorax Aug 9 '16 at 2:27