Return to Answer

Note: I recommitment using the glmnet package to fit elastic nets. It has a vey efficient implementation, and has become a standard. This is written from the perspective of that package, which uses the notation that has become a standard in the statistical learning literature. If you would rather use elasticnet, see the note below for an important caveat.

There are two parameters in elastic net $\alpha$ and $\lambda$. $\alpha$ controls the relative balance between the lasso and ridge regularization, and $\lambda$ controls the overall level (intensity) of regularization.

$\alpha$ is often chosen based on intent and domain knowledge. Do you want a sparse model with many zero coefficients? Then chose $\alpha$ close to a pure lasso regression. Do you want a dense model with many non-zero but small coefficients? Then chose $\alpha$ close to a pure ridge model.

On the other hand, $\lambda$ is a parameter that is tuned during the fitting of the model. The most common way to do this is with cross validation. In cross validation, you split your training data into many "folds" - partitions into a training and tuning (validation) subset - and fit a a model on each fold, one for each lambda. The various out of sample datasets in cross validation allow you to estimate the out of sample error for each value of $\lambda$, and then a heuristic is used to choose the best value of $\lambda$. Common heuristics are choosing the $\lambda$ that minimizes the out of fold deviance, and choosing the largest $\lambda$ one standard deviation away from the minimizer lambda (standard deviation referring to the estimated standard deviation in out of fold deviances).

Here's a picture of out of fold deviance estimates by lambda:

Each data point on the curve is one value of $\lambda$, the plots tend to look better by $log(\lambda)$ though, so that has become a standard. The error bars span the standard deviation of the estimated out of fold deviance, the variance is the estimates comes from the various cross validation folds. I believe this picture was from a binomial (logistic) model, with $10$ fold cross validation.

If you are using the glmnet package in R, which I recommend over elasticnet, there is a built in function to perform cross validation, it's called cv.glmnet. I highly suggest reading the documentation on this function before fitting your first glmnets.

To get coefficients, use R's coef function. This is a general function that is overloaded to work on most model objects (technically a generic function).

For more details, I highly reccoment the following paper by the pioneers in this field:

Regularization Paths for Generalized Linear Models via Coordinate Descent

Caveat: I checked the documentation for the elasticnet package, it uses a non-standard definition of $\lambda$:

lambda: Quadratic penalty parameter. lambda=0 performs the Lasso fit

This is not the use of $\lambda$ that has become standard in the literature - and I believe is equivalent to what is usually called $\alpha$. Be warned.

Note: I reccomend using the glmnet package to fit elastic nets. It has a vey efficient implementation, and has become a standard. This is written from the perspective of that package, which uses the notation that has become a standard in the statistical learning literature. If you would rather use elasticnet, see the note below for an important caveat.

There are two parameters in elastic net $\alpha$ and $\lambda$. $\alpha$ controls the relative balance between the lasso and ridge regularization, and $\lambda$ controls the overall level (intensity) of regularization.

$\alpha$ is often chosen based on intent and domain knowledge. Do you want a sparse model with many zero coefficients? Then chose $\alpha$ close to a pure lasso regression. Do you want a dense model with many non-zero but small coefficients? Then chose $\alpha$ close to a pure ridge model.

On the other hand, $\lambda$ is a parameter that is tuned during the fitting of the model. The most common way to do this is with cross validation. In cross validation, you split your training data into many "folds" - partitions into a training and tuning (validation) subset - and fit a a model on each fold, one for each lambda. The various out of sample datasets in cross validation allow you to estimate the out of sample error for each value of $\lambda$, and then a heuristic is used to choose the best value of $\lambda$. Common heuristics are choosing the $\lambda$ that minimizes the out of fold deviance, and choosing the largest $\lambda$ one standard deviation away from the minimizer lambda (standard deviation referring to the estimated standard deviation in out of fold deviances).

Here's a picture of out of fold deviance estimates by lambda:

Each data point on the curve is one value of $\lambda$, the plots tend to look better by $log(\lambda)$ though, so that has become a standard. The error bars span the standard deviation of the estimated out of fold deviance, the variance is the estimates comes from the various cross validation folds. I believe this picture was from a binomial (logistic) model, with $10$ fold cross validation.

If you are using the glmnet package in R, which I recommend over elasticnet, there is a built in function to perform cross validation, it's called cv.glmnet. I highly suggest reading the documentation on this function before fitting your first glmnets.

To get coefficients, use R's coef function. This is a general function that is overloaded to work on most model objects (technically a generic function).

For more details, I highly reccoment the following paper by the pioneers in this field:

Regularization Paths for Generalized Linear Models via Coordinate Descent

Caveat: I checked the documentation for the elasticnet package, it uses a non-standard definition of $\lambda$:

lambda: Quadratic penalty parameter. lambda=0 performs the Lasso fit

This is not the use of $\lambda$ that has become standard in the literature - and I believe is equivalent to what is usually called $\alpha$. Be warned.

Note: I recommitment using the glmnet package to fit elastic nets. It has a vey efficient implementation, and has become a standard. This is written from the perspective of that package, if you would rather use elasticnet, see the note below for an important caveat.

There are two parameters in elastic net $\alpha$ and $\lambda$. $\alpha$ controls the relative balance between the lasso and ridge regularization, and $\lambda$ controls the overall level (intensity) of regularization.

$\alpha$ is often chosen based on intent and domain knowledge. Do you want a sparse model with many zero coefficients? Then chose $\alpha$ close to a pure lasso regression. Do you want a dense model with many non-zero but small coefficients? Then chose $\alpha$ close to a pure ridge model.

On the other hand, $\lambda$ is a parameter that is tuned during the fitting of the model. The most common way to do this is with cross validation. In cross validation, you split your training data into many "folds" - partitions into a training and tuning (validation) subset - and fit a a model on each fold, one for each lambda. The various out of sample datasets in cross validation allow you to estimate the out of sample error for each value of $\lambda$, and then a heuristic is used to choose the best value of $\lambda$. Common heuristics are choosing the $\lambda$ that minimizes the out of fold deviance, and choosing the largest $\lambda$ one standard deviation away from the minimizer lambda (standard deviation referring to the estimated standard deviation in out of fold deviances).

Here's a picture of out of fold deviance estimates by lambda:

Each data point on the curve is one value of $\lambda$, the plots tend to look better by $log(\lambda)$ though, so that has become a standard. The error bars span the standard deviation of the estimated out of fold deviance, the variance is the estimates comes from the various cross validation folds. I believe this picture was from a binomial (logistic) model, with $10$ fold cross validation.

If you are using the glmnet package in R, which I recommend over elasticnet, there is a built in function to perform cross validation, it's called cv.glmnet. I highly suggest reading the documentation on this function before fitting your first glmnets.

To get coefficients, use R's coef function. This is a general function that is overloaded to work on most model objects (technically a generic function).

Caveat: I checked the documentation for the elasticnet package, it uses a non-standard definition of $\lambda$:

lambda: Quadratic penalty parameter. lambda=0 performs the Lasso fit

This is not the use of $\lambda$ that has become standard in the literature. Be warned.

Note: I recommitment using the glmnet package to fit elastic nets. It has a vey efficient implementation, and has become a standard. This is written from the perspective of that package, which uses the notation that has become a standard in the statistical learning literature. If you would rather use elasticnet, see the note below for an important caveat.

There are two parameters in elastic net $\alpha$ and $\lambda$. $\alpha$ controls the relative balance between the lasso and ridge regularization, and $\lambda$ controls the overall level (intensity) of regularization.

$\alpha$ is often chosen based on intent and domain knowledge. Do you want a sparse model with many zero coefficients? Then chose $\alpha$ close to a pure lasso regression. Do you want a dense model with many non-zero but small coefficients? Then chose $\alpha$ close to a pure ridge model.

On the other hand, $\lambda$ is a parameter that is tuned during the fitting of the model. The most common way to do this is with cross validation. In cross validation, you split your training data into many "folds" - partitions into a training and tuning (validation) subset - and fit a a model on each fold, one for each lambda. The various out of sample datasets in cross validation allow you to estimate the out of sample error for each value of $\lambda$, and then a heuristic is used to choose the best value of $\lambda$. Common heuristics are choosing the $\lambda$ that minimizes the out of fold deviance, and choosing the largest $\lambda$ one standard deviation away from the minimizer lambda (standard deviation referring to the estimated standard deviation in out of fold deviances).

Here's a picture of out of fold deviance estimates by lambda:

Each data point on the curve is one value of $\lambda$, the plots tend to look better by $log(\lambda)$ though, so that has become a standard. The error bars span the standard deviation of the estimated out of fold deviance, the variance is the estimates comes from the various cross validation folds. I believe this picture was from a binomial (logistic) model, with $10$ fold cross validation.

If you are using the glmnet package in R, which I recommend over elasticnet, there is a built in function to perform cross validation, it's called cv.glmnet. I highly suggest reading the documentation on this function before fitting your first glmnets.

To get coefficients, use R's coef function. This is a general function that is overloaded to work on most model objects (technically a generic function).

For more details, I highly reccoment the following paper by the pioneers in this field:

Regularization Paths for Generalized Linear Models via Coordinate Descent

Caveat: I checked the documentation for the elasticnet package, it uses a non-standard definition of $\lambda$:

lambda: Quadratic penalty parameter. lambda=0 performs the Lasso fit

This is not the use of $\lambda$ that has become standard in the literature - and I believe is equivalent to what is usually called $\alpha$. Be warned.

There are two parameters in elastic net $\alpha$ and $\lambda$. $\alpha$ controls the relative balance between the lasso and ridge regularization, and $\lambda$ controls the overall level (intensity) of regularization.

$\alpha$ is often chosen based on intent and domain knowledge. Do you want a sparse model with many zero coefficients? Then chose $\alpha$ close to a pure lasso regression. Do you want a dense model with many non-zero but small coefficients? Then chose $\alpha$ close to a pure ridge model.

On the other hand, $\lambda$ is a parameter that is tuned during the fitting of the model. The most common way to do this is with cross validation. In cross validation, you split your training data into many "folds" - partitions into a training and tuning (validation) subset - and fit a a model on each fold, one for each lambda. The various out of sample datasets in cross validation allow you to estimate the out of sample error for each value of $\lambda$, and then a heuristic is used to choose the best value of $\lambda$. Common heuristics are choosing the $\lambda$ that minimizes the out of fold deviance, and choosing the largest $\lambda$ one standard deviation away from the minimizer lambda (standard deviation referring to the estimated standard deviation in out of fold deviances).

Here's a picture of out of fold deviance estimates by lambda:

Each data point on the curve is one value of $\lambda$, the plots tend to look better by $log(\lambda)$ though, so that has become a standard. The error bars span the standard deviation of the estimated out of fold deviance, the variance is the estimates comes from the various cross validation folds. I believe this picture was from a binomial (logistic) model, with $10$ fold cross validation.

If you are using the glmnet package in R, which I recommend over elasticnet, there is a built in function to perform cross validation, it's called cv.glmnet. I highly suggest reading the documentation on this function before fitting your first glmnets.

To get coefficients, use R's coef function. This is a general function that is overloaded to work on most model objects (technically a generic function).

Note: I recommitment using the glmnet package to fit elastic nets. It has a vey efficient implementation, and has become a standard. This is written from the perspective of that package, if you would rather use elasticnet, see the note below for an important caveat.

There are two parameters in elastic net $\alpha$ and $\lambda$. $\alpha$ controls the relative balance between the lasso and ridge regularization, and $\lambda$ controls the overall level (intensity) of regularization.

$\alpha$ is often chosen based on intent and domain knowledge. Do you want a sparse model with many zero coefficients? Then chose $\alpha$ close to a pure lasso regression. Do you want a dense model with many non-zero but small coefficients? Then chose $\alpha$ close to a pure ridge model.

On the other hand, $\lambda$ is a parameter that is tuned during the fitting of the model. The most common way to do this is with cross validation. In cross validation, you split your training data into many "folds" - partitions into a training and tuning (validation) subset - and fit a a model on each fold, one for each lambda. The various out of sample datasets in cross validation allow you to estimate the out of sample error for each value of $\lambda$, and then a heuristic is used to choose the best value of $\lambda$. Common heuristics are choosing the $\lambda$ that minimizes the out of fold deviance, and choosing the largest $\lambda$ one standard deviation away from the minimizer lambda (standard deviation referring to the estimated standard deviation in out of fold deviances).

Here's a picture of out of fold deviance estimates by lambda:

Each data point on the curve is one value of $\lambda$, the plots tend to look better by $log(\lambda)$ though, so that has become a standard. The error bars span the standard deviation of the estimated out of fold deviance, the variance is the estimates comes from the various cross validation folds. I believe this picture was from a binomial (logistic) model, with $10$ fold cross validation.

If you are using the glmnet package in R, which I recommend over elasticnet, there is a built in function to perform cross validation, it's called cv.glmnet. I highly suggest reading the documentation on this function before fitting your first glmnets.

To get coefficients, use R's coef function. This is a general function that is overloaded to work on most model objects (technically a generic function).

Caveat: I checked the documentation for the elasticnet package, it uses a non-standard definition of $\lambda$:

lambda: Quadratic penalty parameter. lambda=0 performs the Lasso fit

This is not the use of $\lambda$ that has become standard in the literature. Be warned.