Questions regarding linear regression as we change the parameters (related to Andrew Ng's course)

Question

For the sake of clarity, I'll input my questions in (1.), (2.) and so on. They are all related but honestly I don't have a sufficient grasp of the concepts so that I can formulate/compact everything in one question. So please forgive me for that.

As so many people, I am starting Machine Learning by taking Andrew Ng's course. In one of Andrew Ng's examples (right in the beginning of the course) he predicts the price of a house given its area, so $x$, our input variable, is the area in $ft^2$ and $y$, the output variable, is it's price in \$ (our targets).

We basically compute the cost function as

\begin{align*} J \left( \theta_0 , \theta_1 \right) = \frac{1}{2m} \sum_{i = 1}^{m} \left[ h_\theta(x^{(i)}) - y^{(i)} \right]^2 \end{align*}

and minimize it.

So my questions are:

1. Since we have two parameters and we are fitting a line, we can get a view of the "convexity" of the cost function by plotting $J$ as a function of $\theta_0$ and $\theta_1$ on a 3D plot, correct?

2. What if our hypothesis contains only $\theta_0$ and $\theta_1$ but the input $x$ is actually $x^2$ (let's say we decide that $h_\theta = \theta_0 + \theta_1 x^2$ is a better representative model for that problem). Do we still call it a linear regression? Do we compute the cost function in the same way (considering of course that $x$ is squared)? Can we still get a view of the "convexity" of the cost function by plotting $J$ as a function of $\theta_0$ and $\theta_1$ on a 3D plot?

3. What if our hypothesis does not contain that $x^2$ term described above but it contains $\theta_0$, $\theta_1$ and also $\theta_2$ (so $h_\theta = \theta_0 + \theta_1 x_1 + \theta_2 x_2$). Do we still call it a linear regression? Do we compute the cost function in the same way (considering of course that extra term)? Since now we have 3 parameters, we just lost the possibility of viewing the "convexity" of the cost function, correct (that would only be possible in 4D)?

Answer 1

1. Yes you can, but there are more accurate ways of determining convexity in multivariable twice-differentiable functions by examining second partial derivatives in the Hessian matrix.

2. Yes, it is still a linear model because it maintains linearity in the parameters (so nothing changes): $y = \theta_0 + \theta_1 f_1(x_1) + \theta_2 f_2(x_2) + ...+ \theta_nf_n(x_n)$

3. Yes, it is still linear model (as already described in 1 and 2).

Andrew Ng's take on linear regression is a nonparametric approach whose purpose is to describe some basic concepts of machine learning. If you're interested in understanding the subject in more depth, you probably should look into parametric regression where the focus falls on the underlying probabilistic assumptions rather the optimisation problem. These two views of the same topic reflect, in my opinion, the fundamental differences between machine learning and classical statistical modelling.