Why use different cost function for linear and logistic regression? I mean least squares already penalize one big mistake more, then several small ones. So why don't just leave same "mean square error" for logistic regression - it is simpler than messy formula with logarithms.
 A: Note that there are times where least squares is applied to modelling where $E(Y)\propto \frac{\exp(X\beta)}{1+\exp(X\beta)}$ ... nonlinear least squares is used for such cases. 
However, that case is not the same as logistic regression. One reason we don't apply plain least squares to logistic regression is that the variance of a binomial proportion varies with the proportion -- it's larger when the proportion is $\frac12$ (i.e. when $X\beta$ is 0) than when it's near the extremes.
[So why not weighted least squares? Note that the mean and variance are connected; the variance estimate depends on the estimate of the mean, but the current estimate of the mean depends on the variance ... leading to the need to reweight the least-squares model iteratively -- and that can indeed be done. However, it wouldn't generally be maximum likelihood, which is usually what's desired. A variation on such a scheme, however, can be used to get MLEs.]
A: There are 2 points which I found in favor of not using mean square error :


*

*We don’t use linear regression approach in logistic regression but a probabilistic one because linear regression simply does not fulfill the same role. The least squares criterion for fitting a linear regression does not respect the role of the predictions as conditional probabilities, while logistic regression maximizes the likelihood of the training data with respect to the predicted conditional probabilities. Additionally, the predictions from linear regression can be any real number, which negates their use as probabilities.

*We cannot use the same cost function that we use for linear regression because the Logistic Function will cause the output to be wavy, causing many local optima. In other words, it will not be a convex function eg:

Instead, if our cost function for logistic regression uses log-likelihood then it becomes convex:

Source: https://www.internalpointers.com/post/cost-function-logistic-regression
