For the sake of clarity, I'll input my questions in (1.), (2.) and so on. They are all related but honestly I don't have a sufficient grasp of the concepts so that I can formulate/compact everything in one question. So please forgive me for that.

As so many people, I am starting Machine Learning by taking Andrew Ng's course. In one of Andrew Ng's examples (right in the beginning of the course) he predicts the price of a house given its area, so $x$, our input variable, is the area in $ft^2$ and $y$, the output variable, is it's price in \$ (our targets).

We basically compute the cost function as

\begin{align*} J \left( \theta_0 , \theta_1 \right) = \frac{1}{2m} \sum_{i = 1}^{m} \left[ h_\theta(x^{(i)}) - y^{(i)} \right]^2 \end{align*}

and minimize it.

So my questions are:

  1. Since we have two parameters and we are fitting a line, we can get a view of the "convexity" of the cost function by plotting $J$ as a function of $\theta_0$ and $\theta_1$ on a 3D plot, correct?

  2. What if our hypothesis contains only $\theta_0$ and $\theta_1$ but the input $x$ is actually $x^2$ (let's say we decide that $h_\theta = \theta_0 + \theta_1 x^2$ is a better representative model for that problem). Do we still call it a linear regression? Do we compute the cost function in the same way (considering of course that $x$ is squared)? Can we still get a view of the "convexity" of the cost function by plotting $J$ as a function of $\theta_0$ and $\theta_1$ on a 3D plot?

  3. What if our hypothesis does not contain that $x^2$ term described above but it contains $\theta_0$, $\theta_1$ and also $\theta_2$ (so $h_\theta = \theta_0 + \theta_1 x_1 + \theta_2 x_2$). Do we still call it a linear regression? Do we compute the cost function in the same way (considering of course that extra term)? Since now we have 3 parameters, we just lost the possibility of viewing the "convexity" of the cost function, correct (that would only be possible in 4D)?


1 Answer 1

  1. Yes you can, but there are more accurate ways of determining convexity in multivariable twice-differentiable functions by examining second partial derivatives in the Hessian matrix.

  2. Yes, it is still a linear model because it maintains linearity in the parameters (so nothing changes): $y = \theta_0 + \theta_1 f_1(x_1) + \theta_2 f_2(x_2) + ...+ \theta_nf_n(x_n)$

  3. Yes, it is still linear model (as already described in 1 and 2).

Andrew Ng's take on linear regression is a nonparametric approach whose purpose is to describe some basic concepts of machine learning. If you're interested in understanding the subject in more depth, you probably should look into parametric regression where the focus falls on the underlying probabilistic assumptions rather the optimisation problem. These two views of the same topic reflect, in my opinion, the fundamental differences between machine learning and classical statistical modelling.

  • $\begingroup$ Hi @Digio, first of all thank your input. Can I ask you a couple of things on top of your answer, if you would be so kind to clarify? You say that (2.) and (3.) are still linear models, but what is this "multivariate linear regression" I see around? How's this related to (1.), (2.) and (3.)? $\endgroup$
    – bru1987
    Sep 14, 2017 at 11:21
  • 1
    $\begingroup$ Multivariate regression refers to a model with multiple response variables. A model with multiple independent variables as described in (3) falls under multiple regression. $\endgroup$
    – Digio
    Sep 14, 2017 at 12:54
  • $\begingroup$ Ok so let's say that we want to do a polynomial regression (so it contains multiple features and the input parameters are squared, cubed etc). There are a number of input variables ($x_1$ for the area of the house, $x_2$ for the average price of the houses next to it and so on). How does one choose which variables ($x_1$, $x_2$...) should be squared, which should be cubed and so on, so that we build the hypothesys as (for example) $$h_\theta = \theta_0+\theta_1 x_1^3 + \theta_2 x_2 + \theta_3 x_3^2 + \cdots$$ and our model calculates the appropriate values of the parameters? $\endgroup$
    – bru1987
    Sep 14, 2017 at 15:48
  • 1
    $\begingroup$ In statistical modelling you either assume, know, or find the structure of a model through trial and error. In machine learning, you can use a universal function approximator such a neural network to do the job for you. I would advise you to start a new question for a more detailed answer. $\endgroup$
    – Digio
    Sep 16, 2017 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.