I mean least squares already penalize one big mistake more, then several small ones. So why don't just leave same "mean square error" for logistic regression - it is simpler than messy formula with logarithms.
$\begingroup$
$\endgroup$
4
-
2$\begingroup$ Besides the fact that the 'idea behind' it is completely different, do you mean that it is simpler to minimize $\sum_{i=1}^N \left(y_i - \frac{1}{1+e^{-(\beta_0+\sum_{k=1}^p \beta_k x_k)}}\right) ^2$ , did you set up the system of equations to find the solution that minimises it? $\endgroup$– user83346Sep 27, 2015 at 9:43
-
$\begingroup$ Oh thanks. Now I see I asked a stupid question. I completely missed the fact that in logistic regression the hypothesis is much more complex. $\endgroup$– YuriiSep 27, 2015 at 9:52
-
2$\begingroup$ It's not a stupid question, it's a good question, since it throws light on an important aspect of logistic regression. $\endgroup$– Glen_bSep 27, 2015 at 10:45
-
$\begingroup$ See also stats.stackexchange.com/questions/326350/… $\endgroup$– kjetil b halvorsen ♦Jun 14, 2019 at 12:44
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Note that there are times where least squares is applied to modelling where $E(Y)\propto \frac{\exp(X\beta)}{1+\exp(X\beta)}$ ... nonlinear least squares is used for such cases.
However, that case is not the same as logistic regression. One reason we don't apply plain least squares to logistic regression is that the variance of a binomial proportion varies with the proportion -- it's larger when the proportion is $\frac12$ (i.e. when $X\beta$ is 0) than when it's near the extremes.
[So why not weighted least squares? Note that the mean and variance are connected; the variance estimate depends on the estimate of the mean, but the current estimate of the mean depends on the variance ... leading to the need to reweight the least-squares model iteratively -- and that can indeed be done. However, it wouldn't generally be maximum likelihood, which is usually what's desired. A variation on such a scheme, however, can be used to get MLEs.]
-
$\begingroup$ Could you please explain more about binomial proportion? What is it, why does it matter, why that variance is bad. $\endgroup$– YuriiSep 27, 2015 at 11:17
-
$\begingroup$ There's no point at which I think any variance is "bad". In order to answer your other questions without writing a very long answer, I'd need to have some idea what you already understand about logistic regression. What can I take as given? Is there a link or reference whose content you're familiar with, for example? $\endgroup$– Glen_bSep 27, 2015 at 14:07
-
$\begingroup$ Currently I watched 4 weeks of Andrew Ng "Machine Learning" on Coursera. I implemented both linear and logistic regression in Octave, and then in python (in vectorized form). You can assume I know everything about logistic regression that is mentioned in this paper: cs229.stanford.edu/notes/cs229-notes1.pdf (pages 16-19). $\endgroup$– YuriiSep 28, 2015 at 6:22
$\begingroup$
$\endgroup$
There are 2 points which I found in favor of not using mean square error :
- We don’t use linear regression approach in logistic regression but a probabilistic one because linear regression simply does not fulfill the same role. The least squares criterion for fitting a linear regression does not respect the role of the predictions as conditional probabilities, while logistic regression maximizes the likelihood of the training data with respect to the predicted conditional probabilities. Additionally, the predictions from linear regression can be any real number, which negates their use as probabilities.
- We cannot use the same cost function that we use for linear regression because the Logistic Function will cause the output to be wavy, causing many local optima. In other words, it will not be a convex function eg:
Instead, if our cost function for logistic regression uses log-likelihood then it becomes convex:
Source: https://www.internalpointers.com/post/cost-function-logistic-regression
