Hot answers tagged

43

Part 1 In the elastic net two types of constraints on the parameters are employed Lasso constraints (i.e. on the size of the absolute values of $\beta_j$) Ridge constraints (i.e. on the size of the squared values of $\beta_j$) $\alpha$ controls the relative weighting of the two types. The Lasso constraints allow for the selection/removal of variables in ...


43

Here's an unintuitive fact - you're not actually supposed to give glmnet a single value of lambda. From the documentation here: Do not supply a single value for lambda (for predictions after CV use predict() instead). Supply instead a decreasing sequence of lambda values. glmnet relies on its warms starts for speed, and its often faster to fit a whole ...


43

I think when trying to interpret these plots of coefficients by $\lambda$, $\log(\lambda)$, or $\sum_i | \beta_i |$, it helps a lot to know how they look in some simple cases. In particular, how they look when your model design matrix is uncorrelated, vs. when there is correlation in your design. To that end, I created some correlated and uncorrelated data ...


41

Lets say your caret model is called "model". You can access the final glmnet model with model$finalModel. You can then call coef(model$finalModel), etc. You will have to select a value of lambda for which you want coefficients, such as coef(model$finalModel, model$bestTune$.lambda). Take a look at the summaryFunction parameter for the trainControl ...


30

Using glmnet is really easy once you get the grasp of it thanks to its excellent vignette in http://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html (you can also check the CRAN package page). As for the best lambda for glmnet, the rule of thumb is to use cvfit <- glmnet::cv.glmnet(x, y) coef(cvfit, s = "lambda.1se") instead of lambda.min. To do the ...


28

The difference you are observing is due to the additional division by the number of observations, N, that GLMNET uses in their objective function and implicit standardization of Y by its sample standard deviation as shown below. $$ \frac{1}{2N}\left\|\frac{y}{s_y}-X\beta\right\|^2_{2}+\lambda\|\beta\|^2_{2}/2 $$ where we use $1/n$ in place of $1/(n-1)$ for ...


28

glmnet cannot take factor directly, you need to transform factor variables to dummies. It is only one simple step using model.matrix, for instance: x_train <- model.matrix( ~ .-1, train[,features]) lm = cv.glmnet(x=x_train,y = as.factor(train$y), intercept=FALSE ,family = "binomial", alpha=1, nfolds=7) best_lambda <- lm$lambda[which.min(lm$cvm)] ...


28

I emailed this question to Zou and to Hastie and got the following reply from Hastie (I hope he wouldn't mind me quoting it here): I think in Zou et al we were worried about the additional bias, but of course rescaling increases the variance. So it just shifts one along the bias-variance tradeoff curve. We will soon be including a version of relaxed lasso ...


27

Friedman, Hastie, and Tibshirani (2010), citing The Elements of Statistical Learning, write, We often use the “one-standard-error” rule when selecting the best model; this acknowledges the fact that the risk curves are estimated with error, so errs on the side of parsimony. The reason for using one standard error, as opposed to any other amount, seems to ...


25

You don't seem to want logistic regression at all. What you say is "I would like to maximize the difference between true positives and false positives." That is a fine objective function, but it is not logistic regression. Let's see what it is. First, some notation. The dependent variable is going to be $Y_i$: \begin{align} Y_i &= \left\{ \begin{...


22

My understanding is that you can't necessarily say much about which variables are "important" or have "real" effects based on whether their coefficients are nonzero. To give an extreme example, if you have two predictors that are perfectly collinear, the lasso will pick one of them essentially at random to get the full weight and the other one will get zero ...


22

In both plots, each colored line represents the value taken by a different coefficient in your model. Lambda is the weight given to the regularization term (the L1 norm), so as lambda approaches zero, the loss function of your model approaches the OLS loss function. Here's one way you could specify the LASSO loss function to make this concrete: $$\beta_{...


17

train does tune over both. Basically, you only need alpha when training and can get predictions across different values of lambda using predict.glmnet. Maybe a value of lambda = "all" or something else would be more informative. Max


17

Old question, but I recently had to deal with this problem and found this question as a reference. Here is an alternative approach: The glmnet vignette (https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html) specifically addresses this issue, recommending to specify the cross validation folds using the foldids argument and validate $\lambda$ over a ...


17

This has nothing to do with glm, you simply created a problem with an artificial perfect separation: df <- data.frame(x = rnorm(100), y = rnorm(100)) df$y_c = df$y > 0 glm(y_c~., data=df, family=binomial()) Warning messages: 1: glm.fit: algorithm did not converge 2: glm.fit: fitted probabilities numerically 0 or 1 occurred y is a perfect ...


17

I see two issue here. First, your training set is too small relative to your testing set. Normally, we would want a training set that is at least comparable in size to the testing set. Another note is that for Cross Validation, you're not using the testing set at all, because the algorithm basically creates testing sets for you using the "training set". So ...


16

For compleness' sake (and because I accidentally bumped in to this question): starting with version 1.9-3, fitting without intercepts is supported (intercept=FALSE).


16

The unregularized model is suffering from complete separation because you are trying to predict the dichotomized variable price_c from the continuous variable price from which it is derived. The regularized model avoids the problem of complete separation by imposing a penalty that keeps the coefficient for the price predictor from going off to $\infty$ or $-...


15

There are several things wrong with that approach, including: Seeking a cutoff for a continuous probability Using an arbitrary cutoff of 0.5 Assuming that the cost of a "false positive" and a "false negative" are the same for all subjects Using weights that are not fractional Using weights that are estimated Overriding maximum likelihood estimation Not ...


15

From the glmnet documentation (?glmnet), we see that it is possible to perform differential shrinkage. This gets us at least part-way to answering OP's question. penalty.factor: Separate penalty factors can be applied to each coefficient. This is a number that multiplies lambda to allow differential shrinkage. Can be 0 for some variables, which implies no ...


15

To understand why "[t]he response is either 0 or 1 [but] the predictions are probabilities between 0-1", you need to understand the type of model you are working with. Strip away the penalization methods and the cross validation, and you are running a basic logistic regression. The parameters are fit on the log odds / logistic scale. This is called the "...


15

Brief answers to your questions: Lasso and adaptive lasso are different. (Check Zou (2006) to see how adaptive lasso differs from standard lasso.) Lasso is a special case of elastic net. (See Zou & Hastie (2005).) Adaptive lasso is not a special case of elastic net. Elastic net is not a special case of lasso or adaptive lasso. Function glmnet in "...


14

Glmnet is for elastic net regression. This penalises the size of estimated coefficients (via a mix of L1 and L2 penalties). It tries to explain as much variance in the data through the model as possible while keeping the model coefficients small. I found these slides helpful to understand it. Glm doesn't use a penalty term. The effect, as I understand it, ...


14

The point here is that in cv.glmnet the K folds ("parts") are picked randomly. In K-folds cross validation the dataset is divided in $K$ parts, and $K-1$ parts are used to predict the K-th part (this is done $K$ times, using a different $K$ part each time). This is done for all the lambdas, and the lambda.min is the one that gives the smallest cross ...


14

Short answer Overdispersion does not matter when estimating a vector of regression coefficients for the conditional mean in a quasi/poisson model! You will be fine if you forget about the overdispersion here, use glmnet with the poisson family and just focus on whether your cross-validated prediction error is low. The Qualification follows below. Poisson,...


14

Be sure to install and load the glmnet package. install.packages("glmnet") library(glmnet) First you need to form a matrix with all your predictors, we call that matrix $\mathbf{X}$. I have done this for three variables I have created but since you have more you will need to change the predictor matrix accordingly. set.seed(1) x1 <- rnorm(30) ...


13

The first two arguments that glmnet() is expecting are a matrix of the predictors (x, in your case) and a vector of the response (g4, in your case). For the x matrix, it is expecting that you have already dummied out any categorical variables. In other words, glmnet() does not actually know if any of your predictors are categorical, because they have already ...


13

One difficulty in answering this question is that it's hard to reconcile LASSO with the idea of a "true" model in most real-world applications, which typically have non-negligible correlations among predictor variables. In that case, as with any variable selection technique, the particular predictors returned with non-zero coefficients by LASSO will depend ...


12

Omitting cases with NA values could lead to bias. An alternative would be to perform multiple imputations of the missing data, for example with mice, and then do lasso on each of the imputations. Lasso will probably return different sets of selected variables for the imputations, but you could examine how frequently each variable is selected, among the ...


Only top voted, non community-wiki answers of a minimum length are eligible