2
$\begingroup$

I am looking for the correct equation for a ridge logistic regression for multiple variables. I thought it simply was:

$$y=\frac1{1+e^{-(\beta_0+\beta_1X_1+\beta_2X_2+\cdots+\beta_nX_n)}}$$

with an additional penalty parameter added, in this case

$$\sum_i^n (y_i-\widehat{y_i})^2 + \lambda \sum_j^p \beta_j^2$$

Eventually

$$y=\frac1{1+e^{-(\beta_0+\beta_1X_1+\beta_2X_2+\cdots+\beta_nX_n)}}+\sum_i^n (y_i-\widehat{y_i})^2 + \lambda \sum_j^p \beta_j^2$$

Is this true for a ridge regression?

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ Where does the sum of $(y_i-\hat y_i)^2$ come from? That has nothing to do with logistic regression or any regularization of it. Your initial formulation does not describe logistic regression, either: for it to be correct, you must equate the expectation of $y$ with the right hand side. Your final formulation makes no sense at all, because it completely changes the model. Please see stats.stackexchange.com/questions/29325. Perhaps stats.stackexchange.com/questions/228763 answers your question? $\endgroup$
    – whuber ♦
    Jan 31, 2022 at 18:53
  • $\begingroup$ Hi @whuber. Would this be the correct equation for a multiple logistic regression then: ln[Y/(1−Y)]=a+b1X1+b2X2+b3X3...? $\endgroup$
    – Thomas
    Feb 1, 2022 at 8:07
  • $\begingroup$ That's stated explicitly in an answer in the first link I provided. $\endgroup$
    – whuber ♦
    Feb 1, 2022 at 14:23
  • $\begingroup$ I am sorry, I only looked at the second link for my question. I thank you a lot! $\endgroup$
    – Thomas
    Feb 1, 2022 at 15:22

1 Answer 1

Reset to default
6
$\begingroup$

The method of estimation does not affect the regression equation. Your logistic regression model is always the equation you posted in the first equation (save some technicalities).

If you estimate the parameters using the usual maximum likelihood estimation, you will get different estimated values of the parameters than if you use regularization, but those estimates are just guesses at the true values of the $\beta_i$.

As a heads up, the traditional loss function for logistic regression is log loss, not square loss.

$$ L(y,\hat y)=-\sum\bigg[ y_i\log(\hat y_i)+ (1-y_i)\log(1-\hat y_i)\bigg] $$

In the equation, $y=(y_1,\cdots,y_n)$ is the vector of observed classes (so all $0$s and $1$s), and $\hat y =(\hat y_1,\cdots,\hat y_n)$ is the vector of predicted probability values.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks @Dave. Nice and clear. I used the ridge regression with an optimization of the maximum likelihood estimation. Does that influence the second equation (penalty parameter)? If so, what would the equation for maximum likelihood be? $\endgroup$
    – Thomas
    Jan 31, 2022 at 12:33
  • $\begingroup$ @Thomas The $L(y,\hat y)$ equation in my answer is the loss function corresponding to maximum likelihood estimation. I do not follow what you mean when you ask about the "second equation:. Could you please type out exactly what you think the correct regression equation is? $\endgroup$
    – Dave
    Jan 31, 2022 at 18:49
  • 3
    $\begingroup$ NB: The first equation is not a correct representation of logistic regression, and therein may lie the source of the question. $\endgroup$
    – whuber ♦
    Jan 31, 2022 at 19:00
  • $\begingroup$ @Dave, sorry. I meant the loss function I added in my question, but you already answered my question though. Thanks. $\endgroup$
    – Thomas
    Feb 1, 2022 at 8:09
  • 1
    $\begingroup$ You're the best, thanks! $\endgroup$
    – Thomas
    Mar 8, 2022 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.