New answers tagged regularization
1
vote
Why is Scikit's Support Vector Classifier returning support vectors with decision scores outside [-1,1]? Is this a mistake?
Misclassified points will always be support vectors, even when they are "badly misclassified" and lie beyond the margin, with decision function scores outside of $[-1, 1]$.
- 3,096
1
vote
Can a regularization harm more than help in the situation of a huge over-fit?
Yes! Regularization can destroy your model.
For example if we use ridge regression our loss is:
$ \Sigma(y_{true,i} -y_{pred,i})^2 +\lambda a_j^2 $
Where $a_j$ are the regression coefficients.
If we ...
- 729
3
votes
Can a regularization harm more than help in the situation of a huge over-fit?
If you increase the regularisation parameter too much, the model will ignore the data completely and the norm of the weights will be forced arbitrarily close to zero". This ought to be the ...
- 49.4k
1
vote
Why not perform weight decay on layernorm/embedding?
The mechanism of weight decay seems to be not clearly understood in the research field. For example, a research paper [1] reported that "the regularization effect was concentrated in the BN layer....
- 11
1
vote
Accepted
What if zero mean assumption is relased in graphical LASSO?
In practice I think one should always demean the data before applying the graphical lasso algorithm. I.e. just run with $X_i - \bar{X}$ where $\bar{X} = \tfrac{1}{n} \sum_{i=1}^n X_i$.
This might be ...
3
votes
Lasso regression Mathematical intuition
The other answer here covers the specific transformation issues you have asked about, so I'll focus solely on the intuition of the LASSO regression. One useful way to look at LASSO regression is that ...
- 100k
6
votes
Accepted
Lasso regression Mathematical intuition
You have three different questions. Let's tackle them in a somewhat different order. I'll refer to ISLR 2nd edition.
First off, here are the two different formulations of the lasso:
$$ \text{minimize }...
- 99.6k
Top 50 recent answers are included