Supervised learning : How did they find the Cost function to minimize?

Question

I'm studying a tutorial in a video about supervised learning, more specifically, it's about "linear regression with one variable", that is the cost function.

So my first question : is this "cost function" equal to "linear regression with one variable, or does "cost function" belong to linear regression functions with one variable?

Well, having this as the regression function:

$$\color{green}{\underline{\color{black}{h_\theta(x)}}}=\theta_0+\theta_1x$$

we need to find out the minimum values for $\theta_0$ and $\theta_1$, and where $h$ stands for hypothesis. So my second question is : Why should $\theta_0$ and $\theta_1$ have minimum values? I know that having many training set we need to have kind of approximation... but this didn't help me much.

Finally, to find out the solution (the minimum values $\theta_0$ and $\theta_1$), we need to minimize $J(\theta_0,\theta_1)$ which is the cost function and whose expression is below:

$$J(\theta_0,\theta_1) = \frac{1}{2m} \sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})^2$$

$$\begin{array}{cc}\begin{array}{c} \text{minimize}\\ \theta_0,\theta_1 \end{array} & \underbrace{J(\theta_0,\theta_1)}\\ &\text{Cost function} \end{array} $$

where $m$ is the number of the training set and $i$ is the index of the elements in the set. So my third question is : How could we reach this expression? and Can anybody explain this expression to me?

Answer 1

You seem to be muddling different things together.

$h$ is NOT the cost function. $h$ is the object you're trying to fit (estimate parameter values for). It is $h$ that's the linear regression function. You fit it by choosing some cost function $J$, to measure (for a given $\theta_0$,$\theta_1$) how well or badly you fit the data relative to other values for $h_\theta$ ($J$ is big when it's a bad fit and small when it's a good fit). Minimizing the cost means you have the 'least costly' fit (best fit to the data by your cost criterion). You minimize $J$ to get $h$ 'close' to the data.

Now $\theta_0$ and $\theta_1$ don't have "minimum values", they have values that minimize $J$ -- it's $J$ that's at a minimum, not the $\theta$'s. The parameter estimates are at the argmin.

So you choose some $J$ that measures the overall 'badness of fit' - some measure of how far the data is from the given $h$.

The $J$ you have is the sum of squares of differences between $h$ (the line) and $y$ (the data). As you can see, it gets bigger when the fit is worse. It turns out to be a particularly convenient choice, as well as often satisfying people's notions of how a cost function should look.

The expression is just one way (though not the usual way for most of use us; most statisticians would use a different notation) to write that sum of squares. Since $J$ is the sum of squares of residuals, choosing $h$ to minimize $J$ makes the fitted $h$ the least squares regression line.

Other choices of J are definitely possible; see, for example, $L_1$ (least absolute values) regression, or regression based on M-estimators for some alternatives.

Answer 2

A cost function is a function that specifies how bad you're going to feel for making any given error.

In the case of regression, you're going to wince when your model's predictions are far from the mark. A good way to measure this pain for a given set of data is by measuring, say, the average squared distance from a model's predictions to what is actually observed. For a single observation $x$, that means $(h_{\theta}(x) - x)^{2}$.

If a model's predictions for a set of observations $\{x^{(i)}\}_{i \in I}$ are $\{h_{\theta}(x^{(i)})\}_{i \in I}$, the average squared distance is $\frac{1}{|I|}\sum_{i \in I} (h_{\theta}(x^{(i)}) - x^{(i)})^{2}$. Or,

$$ J(\theta_{0}, \theta_{1}) = \frac{1}{|I|}\sum_{i \in I} (\theta_{0} -(1 - \theta_{1})x^{(i)})^{2} $$

If we want to minimize the pain we feel from making mistakes, we should seek to minimize this function. To do that, we minimize over the free parameters $\theta_{0}$ and $\theta_{1}$ using whatever optimization method you'd like, and those values will be the parameters of your model, $h$.

There's no reason that this particular cost function must be preferred over any others that seek to measure the same kind of thing, so you can really do linear regression using whatever cost function you'd like. The reason that this form of cost function is the de-facto standard, however, is that it's convex. That is, the function will always have a unique global optimum at some particular values of $\theta_{0}$ and $\theta_{1}$, and it's pretty easy to find that optimum.

Answer 3

What you are asking above is really the technical details of how ordinary least squares (a linear regression estimation technique) calculates its parameters.

My advice is to not worry too much about how linear regression estimates its parameters. In my opinion, at the stage you are at, it might help out more if you were to look at some examples of linear regression in use, and look at what kind of answers this technique provides to a researcher's questions. Once you get a feel for what circumstances linear regression is used in, then you will be much more prepared to understand the exact algorithm it uses to calculate that answer.

Youtube will probably provide you with some really great examples if you search for 'simple linear regression example' or 'simple OLS example'.

Answer 4

Here is a link for a page that explains every step and component of logistic regression.

www.holehouse.org/mlclass/06_Logistic_Regression.html