3
$\begingroup$

I fit Elastic Net model on 20-50 variables. Elastic Net selects 10 (but actually I can choose any model on the solution path for the next step).

Next, I take these 10 variables and fit standard regression with them. The estimated parameters differ and usually one variable appears to explain most of the variance.

Question is - why? Is it because regression fits the model in one step, getting all parameter estimates at once, whereas Elastic Net does estimation it step by step, variable by variable (I do not know the algorithm)?

Does shrinkage in the Elastic Net influence parameter estimates in such a way that their interpretation becomes - let's try this wording - untrue? If yes, then I would select Elastic Net for best forecasting, but rather choose estimates from regression for interpretation.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Elastic net and OLS optimize different criteria (penalized vs unpenalized sums of squared residuals), so have different solutions. $\endgroup$
    – Andrew M
    Commented Dec 31, 2015 at 1:30

2 Answers 2

Reset to default
6
$\begingroup$

There is no free lunch in statistics. Elastic Net reduces overfitting (lowers variance) at the cost of increasing bias. With OLS, you could fit a model with all 50 variables. This OLS model would have very low bias (under certain assumptions, the coefficient estimates may be unbiased) but suffer from high variance (overfitting).

In your case, you mentioned that the OLS coefficients look very different than the Elastic Net coefficients, even though both models use the same 10 variables. The difference may be due to bias introduced by the fact that Elastic Net does not compute the coefficients by minimizing the residual sum of squares (which is how OLS computes the coefficients). Elastic net computes the coefficients by minimizing the "penalized" residual sum of squares.

Alternatively, the coefficient estimates may be different between OLS and Elastic Net due to sample size. With small sizes, p-values from OLS may not be reliable. With small sample sizes, the bias from elastic net may also be high.

Here's a simulated example using $n=25$. The "true model" contains only two variables, $x1$ and $x2$, with the "true coefficients" of 2 and 3. Due to small sample size and high irreducible error, the p-value for x1 is high (>22%). The coefficient differences between the two models are also high.

set.seed(1983)

nobs <- 25

x1 <- rnorm(nobs, 10, 10)
x2 <- rnorm(nobs, 20, 20)
x3 <- rnorm(nobs, 30, 30)
x4 <- rnorm(nobs, 40, 40)

y <- 100 + 2*x1 + 3*x2 + rnorm(nobs,0,100)

df <- data.frame(y=y, x1=x1, x2=x2, x3=x3, x4=x4)

### fit a linear model

lm.mod <- lm(y ~ ., data=df)

summary(lm.mod)

### fit an elastic net model using 5-fold CV

library(caret)

set.seed(1984)

enet.mod <- train(y ~ ., data=df, method="glmnet", tuneLength=5, trControl=trainControl(method="cv", number=5))

coef(enet.mod$finalModel, enet.mod$bestTune$lambda)

### compute diffs between coefs

lm.mod$coefficients - t(coef(enet.mod$finalModel, enet.mod$bestTune$lambda))[1,]

When the sample size is increased to $n = 1000$, the p-value for $x1$ is very low and the coefficient differences between the two models are small.

set.seed(1983)

nobs <- 1000

x1 <- rnorm(nobs, 10, 10)
x2 <- rnorm(nobs, 20, 20)
x3 <- rnorm(nobs, 30, 30)
x4 <- rnorm(nobs, 40, 40)

y <- 100 + 2*x1 + 3*x2 + rnorm(nobs,0,100)

df <- data.frame(y=y, x1=x1, x2=x2, x3=x3, x4=x4)

### fit a linear model

lm.mod <- lm(y ~ ., data=df)

summary(lm.mod)

### fit an elastic net model using 5-fold CV

library(caret)

set.seed(1984)

enet.mod <- train(y ~ ., data=df, method="glmnet", tuneLength=5, trControl=trainControl(method="cv", number=5))

coef(enet.mod$finalModel, enet.mod$bestTune$lambda)

### compute diffs between coefs

lm.mod$coefficients - t(coef(enet.mod$finalModel, enet.mod$bestTune$lambda))[1,]
Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for a detailed explanation. I think first two paragraphs are more relevant in my case, but I like the alternative explanation. I suppose multicollinearity can also be an influential factor here. However, I remain a bit concerned that I have to choose from estimates that differ in millions of dollars in magnitude to make economic policy or big business decisions. Therefore still an unanswered part of my question is how do I determine which model to choose? Answer below suggests Elastic Net, but lower bias in OLS is more intuitive for me. Could any of you comment on why? $\endgroup$
    – user2530062
    Commented Dec 31, 2015 at 18:04
  • $\begingroup$ If your concern is to make accurate predictions, you should choose the model with the best out-of-sample performance. You could hold out a portion of your data (e.g., hold out 20% of your data) and build two models using the remaining 80%: elastic net, OLS. Score the hold-out data set and see which model returns the lowest root-mean-squared-error. $\endgroup$
    – William Chiu
    Commented Dec 31, 2015 at 18:17
  • $\begingroup$ Notice that, when $n=1000$, the coefficient estimates between elastic net and OLS are about the same. $\endgroup$
    – William Chiu
    Commented Dec 31, 2015 at 18:20
  • $\begingroup$ Now I have p=4 n=842 model and the estimates still vary a lot. This "bias" part you mentioned in original post is stil of a concern. I think OLS would be better for close to average observations (those with low variance themselves), but Elastic Net would be better in general and out-of-sample testing would be preferred. From some external reading I can see that it all goes down to this bias-variance tradeoff, so thank you for pointing me in this direction. $\endgroup$
    – user2530062
    Commented Jan 1, 2016 at 10:41
  • $\begingroup$ How are you measuring the difference in the two sets of coefficients? It may be that the scale of the variables magnify the differences. Have you considered dividing each variable by its mean and then running elastic net and OLS? The division would remove the units of measurement from each covariate. $\endgroup$
    – William Chiu
    Commented Jan 1, 2016 at 18:49
3
$\begingroup$

Much depends on your interpretation of the word "interpretation." Elastic net deliberately penalizes regression coefficients in a way that helps correct for the over-fitting and optimistically high-magnitude coefficients that standard regression can provide in this type of scenario. One might argue that "interpretation" from the perspective of reliably representing the underlying population rather than your particular sample would best be done on the penalized coefficients.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.