This is about linear regression course given by Andrew Ng on Coursea about machine learning. why cost function is $$ \frac{1}{m} \sum _{i=1}^m \left(h_\theta(X^{(i)})-Y^{(i)}\right)^2 $$

and not simply: $$ \frac{1}{m} \sum _{i=1}^m \lvert h_\theta(X^{(i)})-Y^{(i)} \rvert $$


marked as duplicate by Tim, S. Kolassa - Reinstate Monica, whuber Jul 17 '15 at 13:23

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ @Tim: this doesn't really have anything to do with why we use squares for the standard deviation... $\endgroup$ – S. Kolassa - Reinstate Monica Jul 17 '15 at 13:22
  • $\begingroup$ @StephanKolassa right, I copy and pasted a link from a wrong window in browser... $\endgroup$ – Tim Jul 17 '15 at 13:25

In the $Likelihood$ framework, the different cost functions arise when you take the Maximum Likelihood Estimate of your model parameters given different probabilistic assumptions that describe your situation.

For instance, in the case of fitting a model against measured data: The squared error stems from the assumption that for the measured data can be modeled as a Gaussian noise added to your model with each datum independent from each other; in this case the optimal cost function when fitting is the summed squared error between each datum and the value of that model at that point. (Sum of squared difference and Gaussian noise model). The absolute error comes from assuming Laplacian noise (https://en.wikipedia.org/wiki/Laplace_distribution)

So the choice of your cost function stems down several factors 1) what is the right probability model that describes your question of interest. 2) what is computationally or algebraically amenable to solve 3) or some combination of 1 and 2.


Not the answer you're looking for? Browse other questions tagged or ask your own question.