1
$\begingroup$

During linear regression the algorithm jumbles with the coefficients to create a linear model that closely fits real life data points. How good/crappy models are is tested by measuring the residuals, squaring them and taking the sum. If you graph the sum of squared residuals for a series of linear models (each of them getting progressively better at predicting the real life data points) it will look like a quadratic function. So ultimately the best model produces the minimized value of the quadratic loss function.

I also understand that in order to test how good/crappy a linear classification model is, you can measure how many misclassifications there are and assign one point for every misclassification and zero points for every correct classification (0-1 loss function - it looks like a step model, check picture below). Therefore the best linear classification model is selected by determining which model minimizes the number of misclassifications. I have recently learned that the loss function for logistic regression appears to be exponential in shape (look at picture). I have questions regarding this loss function.

First Question: The connection between linear regression and the quadratic loss function is very clear to me. I can graph the sum of squared residuals for progressively better linear models and it will produce a quadratic function (a parabola). However how is logistic regression models and the exponential loss function connected?? Can you prove/tell me how this is possible? I read that the traditional 0-1 loss function is altered for logistic regression models and a heavier penalty (not just one point) is applied for misclassification - however I'm still lost in seeing the connection between logistic regression and its corresponding loss function.

Second Question: If the loss function for logistic regression is an exponential function, how do we look to minimize this loss function? If I look at the quadratic loss function i can clearly see the minimum value, moreover I can even take the derivative of the quadratic loss function to find when the slope is zero. If I'm looking at the logistic loss function how would I minimize the function?

Please use plain english

0-1 Loss Function enter image description here

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Logistic regression is typically talked about as having its parameters estimated by maximum likelihood. This is equivalent to estimating its parameters through minimizing binary cross-entropy loss. Note that a logistic regression on its own does absolutely no classification. You must have a threshold above which you call the input $1$ and below which you call the input $0$, and that threshold does not have to be $0.50$. Our member Frank Harrell, in fact, argues vigorously against considering accuracy of classification methods like logistic regression. $\endgroup$
    – Dave
    Commented Apr 25, 2020 at 3:42
  • $\begingroup$ Thanks! I have just read about binary cross-entropy. I see that we take the negative log of the probability of each data point being of its true class. Then we take the average of the negative log (probability) for all data points and we get a binary cross entropy loss score for a specific logistic model. So if I made a series of progressively better logistic models (better able to predict the true class with higher probabilities) and plot their loss scores the graph would look exponential in shape. And the best logistic model is able to produce the minimum value of the exponential curve? $\endgroup$
    – link
    Commented Apr 25, 2020 at 15:21
  • $\begingroup$ 1) Be careful about overfitting. 2) It will depend on what kind of improvement each model makes on the other. I would. It expect an exponential-looking curve any more than I would expect a parabola for MSE loss or a V-shape for MAE loss in linear regression. Consider what happens if you add a variable that makes a huge improvement in model fit versus adding a variable that makes a small improvement. $\endgroup$
    – Dave
    Commented Apr 25, 2020 at 15:27
  • 1
    $\begingroup$ This have been asked many times, but it is difficult to choose a duplicate! See this list $\endgroup$
    – kjetil b halvorsen ♦
    Commented Oct 25, 2020 at 18:41

0

Reset to default