3
$\begingroup$

Let's say I have a equation like as below (linear regression)

Y = intercept + x1+x2+x3...xn

intercept = 30

positive coeff sum = 40

negative coeff sum = -10

So, the final outcome becomes like as below

30 + 40 - 10 = 60

My question is why does intercept have to be +30?

It could have been -30 as well?

Whether logistic or linear regression, how is the sign of intercept determined?

Does intercept only take positive values always?

$\endgroup$
3
  • 3
    $\begingroup$ ?? It doesn't have to be +30 or -30, this is what was estimated from your data, theoretically it could be any value. It doesn't have to be positive. $\endgroup$ Commented Feb 16, 2022 at 10:59
  • 1
    $\begingroup$ Why the neural-networks tag? That makes me think you have some other question for which this question is a proxy. Do you know about the XY problem? $\endgroup$
    – Dave
    Commented Feb 16, 2022 at 11:17
  • $\begingroup$ It's fine if you removed the neural-networks tag because it was a mistake to include it in the first place, but your original inclusion of that tag leads me to wonder if you have some other question for which this is a proxy. $\endgroup$
    – Dave
    Commented Feb 16, 2022 at 17:44

2 Answers 2

2
$\begingroup$

It is determined by the estimation method you select. In OLS, the regression parameters $\hat\beta_{OLS}$ are determined by the following.

$$ \hat\beta_{OLS}=(X^TX)^{-1}X^Ty $$

Whatever the result of that calculation turns out to be is the answer.

Logistic regression does not have a closed-form equation for its parameter vector, but it can be calculated as the parameter vector that minimizes the crossentropy loss function (same as how OLS minimizes the square loss function, which turns out to be the same as the $\hat\beta_{OLS}$ above). Whatever the intercept value is in the parameter vector is the intercept in the logistic regression; any real number is possible.

$\endgroup$
13
  • $\begingroup$ is there any logic to this? I ask this because, my input variables explain only 30% of the outcome. And model has predicted the class probability to be 70%. So, intercept is 40.. $\endgroup$
    – The Great
    Commented Feb 16, 2022 at 13:04
  • $\begingroup$ @TheGreat Logistic regression has its coefficients calculated a different way, and there is not a closed-form solution like there is in OLS. I gave that as an example of how the intercept is whatever the intercept in the parameter vector is that minimizes the loss function (square loss in OLS). // What do you mean that the input variables explain $30\%$ of the outcome? How do you calculate that? // What do you mean that the predicted class probability is $70\%?$ Is that for one observation? What about the other observations? // Do you mean an intercept of $40$ because $70-30=40?$ $\endgroup$
    – Dave
    Commented Feb 16, 2022 at 13:11
  • $\begingroup$ I have put a bounty on the below uestion. If you have time and are interested, can help me with the related question - $\endgroup$
    – The Great
    Commented Feb 16, 2022 at 13:29
  • $\begingroup$ datascience.stackexchange.com/questions/107928/… $\endgroup$
    – The Great
    Commented Feb 16, 2022 at 13:30
  • $\begingroup$ How are you getting the $30$ and $70?$ $\endgroup$
    – Dave
    Commented Feb 16, 2022 at 17:25
0
$\begingroup$

For linear regression, the intercept can be positive or negative (or 0), and simply represents the value of the mean of Y when the hyperplane (your linear relationship) passes through the point where all of the explanatory variables are 0 (i.e, x=(0,0,...,0).) If you have data around (0,0,...,0), this point is meaningful, otherwise it's just a starting point for estimating the value of y for any x in your data range and doesn't have much practical value, itself.

For a logistic regression, it can also be positive or negative (or 0). Being negative, here, means that the probability (e^B0/(1+e^B0)) at x = (0,0,...,0) will be < 0.5 and being positive means that it will be > 0.05. Again, though, its practical value will depend on whether you are interested in (and have data around) that point.

Side note: I can't tell from the original question whether you are confusing variables and coefficients, and possibly missing that they are two different but required elements. For example, we might normally see:

y = intercept + B1X1 + B2X2 + ... + BnXn + noise

where each Bi is the coefficient (slope) of the corresponding Xi variable. The intercept is often written as B0.

The sum of the coefficients (your "coeff sum"), then, would only matter if you happened to be looking at the point where all of the X's have a value of 1. That's a very, very specific case that is rarely of particular interest (even with all binary variables, that's just one particular case.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.