New answers tagged

1 vote

Why is Scikit's Support Vector Classifier returning support vectors with decision scores outside [-1,1]? Is this a mistake?

Misclassified points will always be support vectors, even when they are "badly misclassified" and lie beyond the margin, with decision function scores outside of $[-1, 1]$.
user avatar
  • 3,096
1 vote

Can a regularization harm more than help in the situation of a huge over-fit?

Yes! Regularization can destroy your model. For example if we use ridge regression our loss is: $ \Sigma(y_{true,i} -y_{pred,i})^2 +\lambda a_j^2 $ Where $a_j$ are the regression coefficients. If we ...
user avatar
  • 729
3 votes

Can a regularization harm more than help in the situation of a huge over-fit?

If you increase the regularisation parameter too much, the model will ignore the data completely and the norm of the weights will be forced arbitrarily close to zero". This ought to be the ...
user avatar
1 vote

Why not perform weight decay on layernorm/embedding?

The mechanism of weight decay seems to be not clearly understood in the research field. For example, a research paper [1] reported that "the regularization effect was concentrated in the BN layer....
user avatar
  • 11
1 vote
Accepted

What if zero mean assumption is relased in graphical LASSO?

In practice I think one should always demean the data before applying the graphical lasso algorithm. I.e. just run with $X_i - \bar{X}$ where $\bar{X} = \tfrac{1}{n} \sum_{i=1}^n X_i$. This might be ...
user avatar
3 votes

Lasso regression Mathematical intuition

The other answer here covers the specific transformation issues you have asked about, so I'll focus solely on the intuition of the LASSO regression. One useful way to look at LASSO regression is that ...
user avatar
  • 100k
6 votes
Accepted

Lasso regression Mathematical intuition

You have three different questions. Let's tackle them in a somewhat different order. I'll refer to ISLR 2nd edition. First off, here are the two different formulations of the lasso: $$ \text{minimize }...
user avatar

Top 50 recent answers are included