1
$\begingroup$

Given a selection of features, how can I get some insight about why have those features (and not others) been selected? Is there a standard approach?

$\endgroup$
4
$\begingroup$

The selection of the penalty value in penalized likelihood typically makes use of some form of information criteria such as AIC, BIC etc. Then the optimal value for the penalty term is where the information criteria minimizes. See the example below in R where a linear model is created with 45 zero coefficients and 5 non-zero ones and 500 random samples are generated from this model:

#install.packages('DLASSO')
library(DLASSO)
r = 5      # Total non zero coefficients
zr = 45    # Total zero coefiifients
n = 500    # Total samples
b = c(seq(1, 3, length.out = r), rep(0, zr))
x = matrix(rnorm((r + zr) * n), ncol = r + zr)
y = x %*% b + rnorm(n)
dLasso = dlasso(
    x = x,
    y = y,
    force = FALSE,
    lambda = seq(.05,3,length.out = 100)
)
plot(dLasso,
         label = .1,
         cex = .80,
         all = 1)
cbind(true = b, coef(dLasso))

The information criteria plots are shown below:

enter image description here

Here is the end result:

enter image description here

and the comparison of the coefficients:

> cbind(true = b, coef(dLasso))
     true coef.AICc coef.GIC coef.BIC coef.GCV
X.1  1.00    0.8426   0.9321   0.9090   0.9090
X.2  1.25    1.0588   1.1304   1.1147   1.1147
X.3  1.50    1.3499   1.4432   1.4231   1.4231
X.4  1.75    1.5568   1.6476   1.6301   1.6301
X.5  2.00    1.7547   1.8440   1.8211   1.8211
X.6  0.00    0.0000   0.0524   0.0076   0.0076
X.7  0.00    0.0000   0.0122   0.0001   0.0001
X.8  0.00    0.0000   0.0000   0.0000   0.0000
X.9  0.00    0.0000   0.0005   0.0000   0.0000
X.10 0.00    0.0000   0.0039   0.0000   0.0000
X.11 0.00    0.0000   0.0000   0.0000   0.0000
X.12 0.00    0.0000  -0.0080  -0.0001  -0.0001
X.13 0.00   -0.0003  -0.0759  -0.0474  -0.0474
X.14 0.00    0.0000  -0.0006   0.0000   0.0000
X.15 0.00    0.0000   0.0397   0.0173   0.0173
X.16 0.00    0.0000   0.0001   0.0000   0.0000
X.17 0.00    0.0000  -0.0037   0.0000   0.0000
X.18 0.00    0.0000  -0.0087  -0.0001  -0.0001
X.19 0.00    0.0000   0.0067   0.0000   0.0000
X.20 0.00    0.0000   0.0010   0.0000   0.0000
X.21 0.00    0.0000  -0.0046   0.0000   0.0000
X.22 0.00    0.0000   0.0196   0.0003   0.0003
X.23 0.00    0.0000   0.0079   0.0001   0.0001
X.24 0.00    0.0000   0.0000   0.0000   0.0000
X.25 0.00    0.0000   0.0113   0.0001   0.0001
X.26 0.00    0.0000  -0.0011   0.0000   0.0000
X.27 0.00    0.0000   0.0076   0.0001   0.0001
X.28 0.00    0.0000  -0.0167  -0.0002  -0.0002
X.29 0.00    0.0000   0.0080   0.0001   0.0001
X.30 0.00    0.0000  -0.0258  -0.0004  -0.0004
X.31 0.00    0.0000   0.0027   0.0000   0.0000
X.32 0.00    0.0000  -0.0207  -0.0003  -0.0003
X.33 0.00    0.0000   0.0011   0.0000   0.0000
X.34 0.00    0.0000  -0.0067  -0.0001  -0.0001
X.35 0.00    0.0000  -0.0009   0.0000   0.0000
X.36 0.00    0.0000   0.0183   0.0003   0.0003
X.37 0.00    0.0000   0.0019   0.0000   0.0000
X.38 0.00    0.0000  -0.0185  -0.0002  -0.0002
X.39 0.00    0.0000  -0.0004   0.0000   0.0000
X.40 0.00    0.0000   0.0501   0.0154   0.0154
X.41 0.00    0.0000  -0.0001   0.0000   0.0000
X.42 0.00    0.0000   0.0017   0.0000   0.0000
X.43 0.00    0.0000  -0.0037   0.0000   0.0000
X.44 0.00    0.0000   0.0001   0.0000   0.0000
X.45 0.00    0.0000   0.0000   0.0000   0.0000
X.46 0.00    0.0000   0.0083   0.0001   0.0001
X.47 0.00    0.0000   0.0000   0.0000   0.0000
X.48 0.00    0.0000   0.0001   0.0000   0.0000
X.49 0.00    0.0000  -0.0017   0.0000   0.0000
X.50 0.00    0.0000   0.0095   0.0001   0.0001
$\endgroup$
2
  • 1
    $\begingroup$ I'd recommend adding some explanation to the plots. Your coefficient plot has multiple lines with the same line design. Otherwise a good answer. $\endgroup$ – Forgottenscience May 7 '20 at 8:36
  • $\begingroup$ Thanks! I guess I would now ask: how to get some insight about why those features minimize the information criteria, and not others? $\endgroup$ – Mencia May 7 '20 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.