Questions tagged [penalized]

Methods of modifying objective functions to control the solutions of optimization problems.

Filter by
Sorted by
Tagged with
5
votes
2answers
2k views

Rare Events Logistic Regression

Suppose the event of interest occurs in approximately $10 \%$ of the cases where the number of cases is around $5,000$. Should you use a penalized logistic regression for this or is regular logistic ...
18
votes
1answer
14k views

Are categorical variables standardized differently in penalized regression? [duplicate]

In penalized/regularized regression (lasso, ridge, etc.) the predictors are typically standardized to be centered at 0 and often to have variance 1. Are categorical predictors treated differently. If ...
20
votes
2answers
3k views

KKT versus unconstrained formulation of lasso regression

L1 penalized regression (aka lasso) is presented in two formulations. Let the two objective functions be $$ Q_1 = \frac{1}{2}||Y - X\beta||_2^2 \\ Q_2 =\frac{1}{2}||Y - X\beta||_2^2 + \lambda ||\beta|...
4
votes
1answer
844 views

The name of 'Fused' Lasso

As many of you know, the Fused Lasso is one of well known penalized methods, which is introduced by Tibshirani, 2005. However, I don't get to the meaning of how it is called. Could anyone give any ...
0
votes
1answer
116 views

A reward becomes a penalty if

I am working to build a reinforcement agent with DQN. The agent would be able to place buy and sell orders for a day trading purpose. I am facing a little problem with that project. The question is "...
27
votes
3answers
8k views

LASSO with interaction terms - is it okay if main effects are shrunk to zero?

LASSO regression shrinks coefficients towards zero, thus providing effectively model selection. I believe that in my data there are meaningful interactions between nominal and continuous covariates. ...
10
votes
2answers
2k views

B-Splines VS high order polynomials in regression

I do not have a specific example or task in mind. I'm just new on using b-splines and I wanted to get a better understanding of this function in the regression context. Let's assume that we want to ...
6
votes
1answer
309 views

How to decide which penalty measure to use ? any general guidelines or thumb rules out of textbook

A number of regularization measures are available in literatures, which is kind of confusing to beginners. The classical penalty is ridge by Hoerl & Kennard (1970,Technometrics 12, 55–67). ...
6
votes
1answer
531 views

How is the minimum $\lambda$ computed in group LASSO?

The LASSO problem works by minimizing $$\min_\beta (\frac{1}{2}\left\rVert y-X\beta\right\rVert^2_2+\lambda\left\rVert\beta\right\rVert_1)$$ Here in this webpage I found that the minimal value of ...
5
votes
1answer
3k views

Is there a closed form solution for L2-norm regularized linear regression (not ridge regression)

Consider the penalized linear regression problem: $$ \text{minimize}_\beta \,\,(y-X\beta)^T(y-X\beta)+\lambda \sqrt{\sum \beta_i^2} $$ Without the square root this problem becomes ridge regression. ...
0
votes
2answers
3k views

Minimum number of observations needed for penalized regression?

I'm wondering what should be the minimum sample size to perform ridge, lasso or elastic net regression. I have a binomial outcome that I want to relate with a set of features (18 features in some ...
6
votes
1answer
2k views

Understanding confidence intervals in Firth penalized logistic regression

I recently discovered penalized likelihood ratio methods to cope with sparse and/or separated data. I'm having some problems though in understanding the results a logistic regression using Firth ...
3
votes
3answers
1k views

Why does L2 regularization smooth the loss surface?

Fitting neural nets with L2 penalization, I've noticed that I often attain lower in-sample mean-squared errors with higher rates of L2 "weight decay", then I do with lower rates of L2 weight decay. ...
3
votes
1answer
308 views

Regularized linear model: adding special constraints to the coefficient

I understand we can add $L_1$ or $L_2$ regularization to linear regression (Lasso and Ridge regression). In addition, it is possible to restrict the coefficient to be integers (see this post). ...
2
votes
0answers
552 views

Negative binomial log-likelihood in penalized regression

I am trying to understand how penalized logistic regression works and I got stuck with negative binomial log-likelihood. I understand the the first two formulas and the penalization part in the ...