2
$\begingroup$

Alright, I have an assignment that makes me calculate weights for a function with different terms. At first, I thought I might just leave the weight for the term $1$ out, and instead use the intercept. I have decided to use RidgeCV, as I had a large amount of multicolinearity.

However, now I have appended my $x$ by a row of $1$'s, and did the following:

RidgeCV(fit_intercept = False).fit(x, y)

Alright, now I have an array of weights. For comparison, I have tried also

RidgeCV(fit_intercept = True).fit(x, y)

As expected, the weight for $1$ became 0. However, all other values changed too - and the intercept is different from the weight for $1$ from before. I have also a different .score() - the first one is higher.

Why is that the case? I thought all fit_intercept was doing is adding a row of $1$ to my $x$, which obviously can't be true. Also, should I try centering my data myself, or is that now unnecessary?

$\endgroup$
1
$\begingroup$

If you use no regularization, i.e. alpha=0, they'll be the same/very similar when used the same random initialization, e.g. random_state=42. This is because Ridge class doesn't regularize the intercept term, but only the weights. When you include the intercept term, it becomes a member of your feature set and the column is assigned a weight, which is then regularized.

And, you should be adding a columns of 1's, not row of 1's since X is assumed to be $n\times k$ where $n$ is number of samples, and $k$ is number of features.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Yes sorry, I wrote row instead of column. So I assume to get the most accurate anwser when I am allowed to set a constant is to have my column of 1's and set fit_intercept to false? $\endgroup$ – ShonyLone Mar 20 at 11:40
  • $\begingroup$ If you want to regularise the bias term, yes. But, regularization in general fights overfitting, where an overfitted model is sensitive to small changes in features. But, the feature corresponding to bias term is always $1$ and so doesn't change. So, no strong signal to regularize it. $\endgroup$ – gunes Mar 20 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.