Tag Info

New answers tagged regularization

1 vote

Why is Scikit's Support Vector Classifier returning support vectors with decision scores outside [-1,1]? Is this a mistake?

Misclassified points will always be support vectors, even when they are "badly misclassified" and lie beyond the margin, with decision function scores outside of $[-1, 1]$.
• 3,096
1 vote

Can a regularization harm more than help in the situation of a huge over-fit?

Yes! Regularization can destroy your model. For example if we use ridge regression our loss is: $\Sigma(y_{true,i} -y_{pred,i})^2 +\lambda a_j^2$ Where $a_j$ are the regression coefficients. If we ...
• 729

Can a regularization harm more than help in the situation of a huge over-fit?

If you increase the regularisation parameter too much, the model will ignore the data completely and the norm of the weights will be forced arbitrarily close to zero". This ought to be the ...
• 49.4k
1 vote

Why not perform weight decay on layernorm/embedding?

The mechanism of weight decay seems to be not clearly understood in the research field. For example, a research paper [1] reported that "the regularization effect was concentrated in the BN layer....
• 11
1 vote
Accepted

What if zero mean assumption is relased in graphical LASSO?

In practice I think one should always demean the data before applying the graphical lasso algorithm. I.e. just run with $X_i - \bar{X}$ where $\bar{X} = \tfrac{1}{n} \sum_{i=1}^n X_i$. This might be ...

Lasso regression Mathematical intuition

The other answer here covers the specific transformation issues you have asked about, so I'll focus solely on the intuition of the LASSO regression. One useful way to look at LASSO regression is that ...
• 100k
You have three different questions. Let's tackle them in a somewhat different order. I'll refer to ISLR 2nd edition. First off, here are the two different formulations of the lasso:  \text{minimize }...