Why such a dissimilar solution to logistic regression and linear regression? In machine learning, linear regression and logistic regression are very common. The solutions to them are to find a parameter to minimize the energy function. Specifically, given a training data samples: $\{x^{(i)},y^{(i)}\}$, for linear regression, we try to find a $\theta$ to minimize $\sum_i(y^{(i)}-\theta^Tx^{(i)})^2$. However, for logistic regression, although the model is $y\sim\frac{1}{1+\exp(-\theta^Tx)}$, when we solve it, we maximize the likelihood instead of finding a $\theta$ to minimize $\sum_i(y^{(i)}-\frac{1}{1+\exp(-\theta^Tx)})^2$, just like linear regression. Why the solutions are so different?
 A: First, I think you are confusing some terms. Both maximum likelihood estimators and least squares estimators minimize an energy function to obtain their estimates. For least squares the energy function $h(\theta)$ is of the form $h(\theta)=\sum_i (\mathbb{E_\theta}[Y^{(i)}|X^{(i)}]-y^{(i)})^2$. For maximum likelihood the energy function is $h(\theta)=-\prod_i p_\theta(y^{(i)}|x^{(i)})$. So, I guess your question is really why don't we use least squares estimators for logistic regression.
The statistical model for every $y^{(i)}$ for  linear regression is $\{p_\theta(y^{(i)}|x^{(i))})=\mathcal{N}(y^{(i)};\theta^\top x^{(i)},\sigma_e^2): (\theta,\sigma_e) \in \mathbb{R}^p \times \mathbb{R_0^+}\}$. The maximum likelihood estimate $\hat{\theta}$ is $\hat{\theta}=\sum_i(y^{(i)}-\theta^\top x^{(i)})^2$. 
The statistical model for logistic regression is  $\{p_\theta(y^{(i)}|x^{(i)})=\frac{e^{\theta^\top x^{(i)}y^{(i)}}}{1+e^{\theta^\top x^{(i)}}}\}$, the typical estimator used for logistic regression is the maximum likelihood estimator.
Hence, the solutions are not different at all. For both models typically the maximum likelihood estimator is used.
Your statement that the model for logistic regression is $y\sim\frac{1}{1+\exp(-\theta^Tx)}$ is wrong. This is the formula for the conditional model implied expected value $\mathbb{E}[Y^{(i)}|X^{(i)}]$. I provided the correct model for logistic regression. You then continue and use the squared loss function. Hence, you derived the least squares estimator for logistic regression. The least squares estimator and the maximum likelihood estimator are the same for linear regression but in general they are not. I guess this is the answer to your question. We do not use the least squares estimator for logistic regression because it is not equivalent to the maximum likelihood estimator.
