3
$\begingroup$

I'm trying to use sparse linear model for my data,input x(29*50),output y(29*1). In R, the package of glmnet can be used.

Firstly, cv.glmnet() choose lambda and coefficients(at min error), here with leave-one-out cv method,and then plot it. 

cv.fit = cv.glmnet(x,y,family="gaussian",nfolds=29)

plot(cv.fit)

the plot of mse aganist log(lambda) in cv model

Next, print the coefficients

coef(cv.fit,s="lambda.min")

51 x 1 sparse Matrix of class "dgCMatrix" 

              1

(Intercept) 267.7241

cluster_0 .
cluster_1 .
cluster_2 .
cluster_3 .
cluster_4 .
...

cluster_47 .
cluster_48 .
cluster_49 .

Finally, to measure the model's ability for prediction, accuracy is calculated(defined as 1 minus average absolute error divided by numeric range of y)

py <- predict(cv.fit,newx=x,s="lambda.min")
py

V1 267.7241

V2 267.7241

...

v29 267.7241 

ave_abs_error <- mean(abs(py-y))
n_range <- max(y)-min(y)
acc <- 1-ave_abs_error/n_range
acc

0.918365

Although the acc(0.918365) is very high, there is a serious problem. As seen from the plot above, the lambda.min is very large(73.03439),and all coefficients are zero(only with intercept value 267.7241), all predicted py are the same as intercept. That's really weird!

I searched lots of threads in forum, hereAn example: LASSO regression using glmnet for binary outcome explains that there is no local min for too few observations and all coefficients were shrunk to zero with the shrinkage penalties.

Does anybody has other interpretations?

Thanks in advance!

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I have run models with exactly these kinds of results: an effect that I was hoping would vary from state to state (in the US, including DC, for 51 "states" total) turned out really not to vary by state at all. One of many ways to double-check things is to rerun the model using lm alone: most likely all the estimated coefficients will be small and most will be nonsignificant. $\endgroup$
    – whuber ♦
    Commented Feb 24, 2016 at 21:53
  • $\begingroup$ First, plot us the distribution of y. (Is it in fact normal? If not, you need to transform with some function to make it sufficiently normal, e.g. log, log1p). Second, make sure to standardize your x variables (e.g. using scale()). $\endgroup$
    – smci
    Commented Feb 10, 2017 at 9:00
  • $\begingroup$ Also, nfolds=29 is choosing LOOCV. You could try smaller nfolds (3..10), with the understanding that the coefficients from any individual run will depend on the random-seed and may not be stable. See Variablity in cv.glmnet results $\endgroup$
    – smci
    Commented Feb 10, 2017 at 9:04

1 Answer 1

Reset to default
3
$\begingroup$

The fine-tuning of the penalization factor of Elastic Net during the cross validation has resulted in a penalty that shrinks all coefficients to zero.

Without being mathematically exact this seems to indicates that none of your features is very helpful. In this case Elastic Net will always predict the mean of the data it was trained on.

Your measure for accuracy is very problematic, as just predicting the mean can produce very high results.

For example, given the standard normal distribution the average absolute error is close to 0.8. Given a large sample size the range is easily around 8, giving you an accuracy of 0.9.

See here:

> set.seed(123)
> x <- rnorm(1e5)
> 1-mean(abs(x-mean(x)))/diff(range(x))
0.9056073
Improve this answer
$\endgroup$
3
  • $\begingroup$ Good explanations! Is that mean all my input features are useless and there is no linear model between input x and response y under the lasso condition? I'm also curious about how to measure a model's ability to predict. Is there any indicators except mse(defined as mean(y-predicted_y)^2)? Many thanks! $\endgroup$
    – cying Jack
    Commented Nov 20, 2015 at 2:24
  • $\begingroup$ In terms of the measure for accuracy, it is indeed problematic in this case where all the coefficients are zero. However, in those more proper cases, which means the number of nonzero-coefficients isn't zero,and the output of the prediction is not the mean of the trained data. In this way, the accuracy may be a reasonable indication to measure prediction ability. Is that right? $\endgroup$
    – cying Jack
    Commented Nov 21, 2015 at 6:33
  • $\begingroup$ @cyingJack I suppose you could make it informative, by comparing it with the accuracy of the model that just predicts the mean all the time. This would be a baseline performance. I would look at classical indicators like R squared instead. $\endgroup$
    – Erik
    Commented Nov 21, 2015 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.