19

The goal of using LASSO is obtaining a sparse representation (of a predicted quantity) in the sense of not having many covariates. Comparing models with $R^2$ tends to favor models with lots of covariates: in fact, adding covariates unrelated to the outcome will never decrease $R^2$ and almost always increases it at least a little bit. The LASSO model will ...


18

In mathematics, statistics and physics, regularisation is the process of adding information in order to make an ill-posed problem soluble and well-behaved or to force a problem to exhibit some property known to be satisfied by suitable solutions; it particularly applies to objective functions in ill-posed optimisation problems. With regard to the lasso, ...


17

The two formulations are equivalent in the sense that for every value of $t$ in the first formulation, there exists a value of $\lambda$ for the second formulation such that the two formulations have the same minimizer $\beta$. Here's the justification: Consider the lasso formulation: $$f(\beta)=\frac{1}{2}||Y - X\beta||_2^2 + \lambda ||\beta||_1$$ Let the ...


17

I would usually only consider splines rather than polynomials. Polynomials cannot model thresholds and are often undesirably global, i.e., observations at one range of the predictor have a strong influence on what the model does at a different range (Magee, 1998, The American Statistician and Frank Harrell's Regression Modeling Strategies). And of course ...


14

I don't know python very well, but I did find one problem with your R code. You have the 2 lines: residuals = sum(y_true - y_preds) mse=residuals^2 Which sums the residuals, then squares them. This is very different from squaring the residuals, then summing them (which it appears that the python code does correctly). I would suspect that this may be a ...


14

Short answer Overdispersion does not matter when estimating a vector of regression coefficients for the conditional mean in a quasi/poisson model! You will be fine if you forget about the overdispersion here, use glmnet with the poisson family and just focus on whether your cross-validated prediction error is low. The Qualification follows below. Poisson,...


13

It's certainly possible to fit good models when there are more variables than data points, but this must be done with care. When there are more variables than data points, the problem may not have a unique solution unless it's further constrained. That is, there may be multiple (perhaps infinitely many) solutions that fit the data equally well. Such a ...


11

One difficulty in answering this question is that it's hard to reconcile LASSO with the idea of a "true" model in most real-world applications, which typically have non-negligible correlations among predictor variables. In that case, as with any variable selection technique, the particular predictors returned with non-zero coefficients by LASSO will depend ...


11

I am late for a party, but here are few of my thoughts about your problem. lasso selects what is informative. Lets consider lasso as a method to get the highest predictive performance with the smallest number of features. It is totally fine that in some cases, lasso selects interaction and not main effects. It just mean that main effects are not informative,...


10

I'll answer your third question first and address your other two later. What do you think he means when saying "Lasso (and related)...but not in the posterior distribution"? This figure from his slides shows what he means. Expressing the lasso regularizer as a prior distribution means your prior distribution will take the form of a Laplacian or double-...


9

You will get the ridge regression solutions, but parametrised differently in terms of the penalty parameter $\lambda$. This holds more generally for convex loss functions. If $L$ is a convex, differentiable function of $\beta$ let $\beta(\lambda)$ denote the unique minimiser of the strictly convex function $$h(\beta) = L(\beta) + \lambda \|\beta\|_2^2$$ ...


8

The term is clearly explained in the abstract of the paper[1] you mention. We propose the ‘fused lasso’, a generalization that is designed for problems with features that can be ordered in some meaningful way. The fused lasso penalizes the $L_1$-norm of both the coefficients and their successive differences. Thus it encourages sparsity of the coefficients ...


7

The so-called local linear approximation (LLA) algorithm described in One-step sparse estimates in nonconcave penalized likelihood models by Zou and Li solves the optimization problem for certain concave choices of $f$. Specifically, for $$f(s) = p(|s|)$$ and a concave function $p : [0,\infty) \to [0,\infty)$, which is differentiable on $(0,\infty)$. The ...


7

Don't do anything special. However, and this is crucial: choose a good quality measure. And that is not classification accuracy, sensitivity, specificity or similar measures, such as ROC curves. These can be very misleading in the case of unbalanced data, "identifying" that simply labeling everything as the majority class is "optimal". Which it isn't. ...


7

The answer depends entirely on the amount of penalization used. If too little, the model will be seen to be overfitted when evaluated on an independent sample. If too much penality, it will be found to be underfitted. The goal is to solve for the penalty that gets it "just right". Two ways to do this are cross-validation (e.g., 100 repeats of 10-fold ...


7

You do have a slight misunderstanding of how penalization works in the spline regression situation. Regression with (cubic, in this case) splines does not implement penalization by reducing the number of knots, but instead by reducing the magnitude of the coefficients of the spline terms. "Degrees of freedom" doesn't refer to the number of knots, but ...


6

sklearn has an example that is almost identical to what you're trying to do here: http://scikit-learn.org/stable/auto_examples/exercises/plot_cv_diabetes.html Indeed this example shows that you do get wildly varying results for alpha for each of the three folds done in that example. This means that you cannot trust the selection of alpha because it clearly ...


6

In section 7.4.5 of "The Elements of Statistical Learning", it's said that splines often give superior results than polynomial regression, because: It produces flexible fits; Produces more stable estimates; Polynomials can produce undesirable results at the boundaries.


5

You are mixing various ideas. Logistic regression uses the logistic function to translate the unrestricted linear predictor to a restricted $[0,1]$ space. This function must be monotonic. The $X$s on the other hand can include complex functions. Regularization is better called penalized maximum likelihood estimation because its purpose nowadays is to ...


5

You don't really need to bother. In most packages (like glmnet) if you do not specify $\lambda$, the software package generates its own sequence (which is often recommended). The reason I stress this answer is that during the running of the LASSO the solver generates a sequence of $\lambda$, so while it may counterintuitive providing a single $\lambda$ ...


5

The multi-collinearity in x1 and x2 is what makes the $\alpha$ value unstable in the Python code. The variance is so small for the distributions that generate these variables, so that the variance of the coefficients are inflated. The variance inflation factor (VIF) could be computed to illustrate this. After the variance is increased from x1 = range(n) ...


5

Shrinkage is all about improving finite sample behavior. That's pretty important: I've never analyzed a data set that is not finite. But more seriously, shrinkage methods are not just useful for data sets with relatively small samples sizes. They are very useful for data sets in which many of the variables have relatively little information about them. For ...


5

If a change of $\beta$ in any direction will not decrease the cost/objective function then you have found yourself in, at least, a local minimum. The calculations below will show for which λ the solution/point $\beta = 0$ stops to be a minimum. Consider the effect of 'a change of $\beta^{(l)}$ by an infinitesimal distance $\partial l$' on the 'change of 1) ...


5

The Penalized Quasi Likelihood (PQL) method has been proposed to fit generalized linear mixed-effects models. The way it works is by doing a kind of a Laplace approximation in a quasi-likelihood formulation of the model. This approximation results in a transformation of the original outcome variable. The aim of the transformation is to make the transformed ...


4

There can be many considerations to this matter. To name a few: Inference: the distribution of ridge estimates is fairly simple to derive. Lasso, and basically any other penalty that performs variable selection, has only limited probabilistic results. Sparsity: If you desire a model with only a few predictors (say, for speed of prediction, for ...


4

Yes. The penalty only affects the optimal parameters. Once the parameters are determined the structural form of the model is identical.


4

There are two issues: glmnet includes an intercept by default while genlasso does not. To disable the intercept in glmnet, use intercept=FALSE; it will reduce the performance significantly. To instead add an intercept in genlasso, cbind a column of ones onto your X matrix, and change D accordingly. If you choose to penalize it, you might need to set the ...


4

Let me suggest an admittedly slightly boring counterexample. Say that $\hat{\beta}_1$ is not just asymptotically more efficient than $\hat{\beta}_2$, but also attains the Cramer Rao Lower Bound. A clever shrinkage technique for $\hat{\beta}_2$ would be: $$ \hat{\beta}_2^\ast = w \hat{\beta}_2 + (1 - w) \hat{\beta}_1 $$ with $w\in(0,1)$. The asymptotic ...


4

You can obtain the same value when you first center the X and Y. > x <- x - matrix(rep(1,506)) %*% colMeans(x) > y <- y - mean(y) > (1/length(y))*max(abs(t(x)%*%y)) [1] 724.8204 note that I get a slightly different value than 724.80000 but this is the same when I run glmnet. when you manually include an intercept with penalty zero (or at ...


3

Penalized estimation can be used for very small sample sizes; it's just that the estimates will be shrunk to the mean very heavily, so the estimates will be conservative. But the intercept in the model is not shrunk, and for binary $Y$ the minimum sample size needed just to estimate the intercept in a logistic model is $n=96$ yielding a margin of error of $\...


Only top voted, non community-wiki answers of a minimum length are eligible