# Question regarding form of a cost function while training a model

I am beginner in Machine Learning and I am very interested in modelling, simulation and all this jazz :)

One of the basic idea I learned so far is the use Cost Function and its minimization in order to train models based on existing data. In my understanding, Cost Function of any form provides an error measure between real and simulated data and, at particular stage of training, how accurate a model is. I have come across a formula:

$$J(\Theta)=\frac{1}{2}\sum_{m=1}^n\big(y^{(i)}-\Theta^{\text{T}}x^{(i)}\big)^2$$

I believe the expression squared inside brackets is expression of an error. I was always asking myself why we tend to use the error squared. How about applying absolute value or power to 4, etc.

I found an explanation based on probabilistic interpretation which describes error to be normally distributed with mean 0. It's fine to understand it but from this point onward authors usually skip the in-between process and tend to give the final result. So basically I still haven't fully understood it :/

I found that if we assume that error is normally distributed, that is

$$\varepsilon^{(i)} \sim \mathcal{N}(0,\sigma^2)$$

its probability is expressed by Gaussian Distribution:

$$\text{P}(\varepsilon^{(i)})=\frac{1}{\sqrt{2\pi}\sigma}\exp{\Bigg(-\frac{\big(\varepsilon^{(i)}\big)^2}{2\sigma^2}\Bigg)}$$

Now, how is it related to the cost function? I was digging more and found out that data are also expressed by normal distribution? Is it because of the assumption of error being normally distributed? I also found an equation below, where $y^{(i)}$ denotes real data and $\Theta^{\text{T}}x^{(i)}$ is a hypothesis. I believe models are fitted based on data and given hypothesis--here linear equation. The model's features are denoted by $x$ whereas parameters by $\Theta$:

$$y^{(i)}|x^{(i)};\Theta \sim \mathcal{N}(\Theta^{\text{T}}x, \sigma^2)$$

It is then followed by equation:

$$\text{P}(y^{(i)}|x^{(i)};\Theta)=\frac{1}{\sqrt{2\pi}\sigma}\exp{\Bigg(-\frac{\big(y^{(i)}-\Theta^{\text{T}}x^{(i)}\big)^2}{2\sigma^2}\Bigg)}$$

I can see the term $(y^{(i)}-\Theta^{\text{T}}x^{(i)})$ which is the same in Cost Function and the equation above.

So reassuming, does the normal distribution of error incline the normal distribution of data and therefore the use of aforementioned form of the Cost Function? If error was not normally distributed, would that form of Cost Function be wrong?

Almost all the ingredients to understanding the probabilistic interpretation of the squared cost function are already in you question. The thing that is missing, is the notion of likelihood maximization. It goes as follows:

Assuming, the model $y_i = x^T_i\Theta + \varepsilon_i$ and $\varepsilon_i \sim \mathcal{N} (0, \sigma^2)$, the value of $\Theta$ that maximizes the likelihood of observed $y_i$'s (given $\Theta$ and $x_i$'s) is the same value of $\Theta$, that minimizes the squared error. That's it.

For why would you want to maximize the likelihood you can read a wiki article, or ponder for a while about "finding parameter, for which the probability to observe values that you have observed is the greatest" and see if you are convinced that it is a good idea.

For the proof, the likelihood is as you wrote it: $$\text{P}(\mathbf{y}|\mathbf{x};\Theta)=\prod_i\frac{1}{\sqrt{2\pi}\sigma}\exp{\Bigg(-\frac{\big(y_i-x^T_i\Theta\big)^2}{2\sigma^2}\Bigg)}.$$

And you can easily see, that it is maximized by minimizing the squared error.

Your general interest in cost functions is valid and important. Different cost functions lead to different estimates, which will have different properties.

One of the differences that I find important, is the amount of influence that outliers have to the estimate. For example, outliers will make comparatively larger contribution to the square error, than to the absolute error. As a result, the regression line in quadratic error case will be more "attracted" to the outliers, and in the absolute error case will be more "robust".

P.S. You may want to use $\mathcal{N}$ (\mathcal{N}) or simply $N$ ($N$) for normal distribution instead of $\aleph$ ($\aleph$), which is usually used to denote a cardinal number.

1-) Cost function measures "distance" between the hypothesis and the groundtruth, so using norm, quadratic, or 4th degree doesn't matter. The cost value (dependant variable) changes, but optimum point doesn't change. Quandratic cost function is general practice. I think it is better than norm (abs) because it has fewer computation steps than norm ( square first than take its s-root). Also, higher terms may reduce efficient storage for variables. So, it is more about computation.

2-) You can use normal distribution as cost function. As the hypothesis gets closer to the ground truth, probability increases. So, maximizing probability, optimizes theta variables. If we compare it with normal distribution, I think $y^{(i)}$ is assumed as $\mu$ and as $\Theta^Tx^{(i)}$ approaches to $\mu$ the propability gets closer to its maximum value.

It is a minor point but, there is something wrong in your formulation, sigma assumed as 1 in the 3th eq. but not it 2nd. I'm not so sure about details for other part of your question.

Do you study it from Bishop's book? I've quit it several months ago at chapter 3 because it is not well-prepared for a beginner (At least for me).

• I missed std. dev. in both formulas. Thanks for pointing those out. – Celdor May 11 '15 at 7:07
• No I have not decided to read any book yet. It's only from one of the online presentation on Machine Learning. – Celdor May 13 '15 at 14:19
• @Celdor Cost function is very important, it defines your optimal estimator. For instance, the mean forecast will not be optimal with many cost functions, particularly asymmetric ones. – Aksakal May 13 '15 at 14:44

The choice of a cost function is independent of the error distribution assumption. The cost function, ideally, should be the function of the cost of forecasting error.

For instance, if you miss a payment on your credit card, you'll get late fee charge and the finance change. If pay earlier than a due date, then the cost is the opportunity cost, e.g. unearned interest on the savings account from which you withdrew your money. So, let's try to construct the cost function:

• when payment time is $t$ days before the due date, the opportunity cost is $c(t,M)=Mtr$, where $r$ is the interest rate on your savings account, and $M$ is the amount due.
• when you pay $t$ days after the due date, the cost is $c(t,M)=MtR+p$, where $R$ is the credit card interest rate, and $p$ is the late payment penalty.

This is an example of how you build the cost function, which must reflect the costs of the error. As you see this has nothing about the distribution of the errors.

The link to the distribution comes when you optimize the forecast to minimize the cost cost function. For instance, if you use the popular squared error cost function, then it happens so that the best forecast estimator is the mean. If you used absolute error cost instead of the squared then the optimal forecast would have been the median. However, for a symmetric distribution such as normal the mean and the median are the same.

Again, in an ideal world you should not be defaulting the squared error cost just because it's convenient to manipulate analytically. However, in reality most people don't bother building the cost functions and use either squared or absolute error.

UPDATE

here's an example, how to construct an estimator from the cost function.

Let's assume that you have a model: $y_i=f(x_i)+\varepsilon_i$. You want to find the estimator which minimizes the cost: $argmin E[c(\hat y,y)]$, i.e. estimator $\hat y$ such that the cost is the minimal, where $y$ is the true value. Now, you're going to assume that the model is true ("under the null"), and plug $y=f(x)$. If your cost function is symmetric, such as quadratic loss, and you probability distribution of error $\varepsilon_i$ is also symmetric, then it happens so that $\hat y_i=E[y_i]=f(x_i)$, i.e. the mean estimator. If these conditions are not met then the mean estimator may not be the optimal.

• HI. So the estimator in my example is error mean which supposed to be 0 at its best and is simply difference between actual and expected value. I think I've got it now but still I have a problem to understand where is the error in your credit card example. is it related to penalty payment meaning the cost function will grow when you get extra penalty? How do you build gradient from it to optimize the parameters? – Celdor May 14 '15 at 6:42