3
$\begingroup$

I'm in the beginnings of following along with the Coursera machine learning course, and I just did univariate linear regression. My regression line/output looks good and the cost function decreased, but was still extremely high at the end of iterating (J(theta) = 2058715091.21221 at the final iteration). Is this an issue if everything looks right and it seems to asymptote around there or should it really be going to zero? Here are the plots:

linear regression with green X for test input at the end

enter image description here

If there's not really a general answer and it depends upon specifics I'll make an edit and post all the code. For a general overview, the data is (house sqft, house price) with ranges (852-4478),(179900-699900). I normalized the house sqft input between [0,1], set the learning rate to 1 and the number of iterations to 100. I tried with a smaller learning rate and higher iterations and it doesn't seem to help.

Any comments or suggestions are greatly appreciated, thanks.

Improve this question
$\endgroup$
8
  • $\begingroup$ I assume you are minimizing the sum of square error where error is defined as the difference between the house price and a forecast house price that is a linear function of house size? $\endgroup$
    – Matthew Gunn
    Aug 10, 2016 at 3:11
  • $\begingroup$ Yep. 10 more characters $\endgroup$
    – Austin
    Aug 10, 2016 at 3:18
  • 4
    $\begingroup$ @GeneralAbrial my guess is that it's because Andrew Ng's Coursera course uses it as a teaching tool for gradient descent $\endgroup$
    – shadowtalker
    Aug 10, 2016 at 3:31
  • 2
    $\begingroup$ Gradient descent is also a fairly low-tech, antiquated way to solve optimization problems. That said, it's easy to understand and is probably a starting point. I agree that in practice, gradient descent is a weird way to solve an ordinary least squares problem. Almost any stats package is going to directly solve the linear system defining the first order conditions using LAPACK linear algebra routines (which use LU factorization), and if the system is large enough, solve it iteratively using Krylov subspace methods. $\endgroup$
    – Matthew Gunn
    Aug 10, 2016 at 3:44
  • 1
    $\begingroup$ I'd agree that leaving the course thinking that production level regression solvers is not good, but I think that's on the instructor no being clear about the situation. Using gradient descent to solve linear regression is fine if you're trying to teach both, or if you're doing some other learning focused task. For example, if I'm learning a new programming language, implementing linear and logistic regression with gradient descent or Newton's method is one of the first thing's i'll do to get my feet wet. $\endgroup$
    – Matthew Drury
    Aug 10, 2016 at 4:42

1 Answer 1

Reset to default
7
$\begingroup$

Directly examining the cost function can be useful, but be aware of some basic issues:

  • Units: Eg. if you measured house price in a less valuable currency (eg. Yen) all the numbers would be higher. What you regard as "high" must be relative to the units used.
  • Number of observations: you want to normalize by the number of observations so that more data doesn't mechanically give you higher cost!

Some basic measures of overall error:

Root mean square error is a monotonic transformation of the sum of squares, so minimizing the sum of squares is the same thing as minimizing root mean square error (and minimizing mean absolute deviation is the same as minimizing the sum of absolute error).

$R^2 = 1 - \frac{SS_{err}}{SS_{tot}}$ is 1 minus the sum of squared error divided by the total sum of squares. For a linear regression with a constant, this essentially gives you the proportion of the variance explained by the model.

What does a high root mean square error, high mean absolute deviation mean, or low $R^2$ imply?

In some sense it means that you have a lot of forecast error. What's reasonable to expect in terms of forecast error is entirely problem dependent. In physics with good data and precisely modeled problems, you may have almost no error. In economics (eg. forecasting home prices etc...) you tend have a LOT of error. In fact, if you have implausibly good forecasts, too little error, it probably means you've overfit the data!

Beware of overfitting...

In general, a huge problem in empirical research, machine learning etc.. is overfitting. If you give yourself enough parameters to estimate, you can end up with a model that fits the training data too well... it fits your sample, but if you try it on new data, the model may perform horribly. If there's overfitting, your algorithm is picking up random, meaningless peculiarities of your particular data set.

Note there's a big conceptual difference between error on: (1) the data used to estimate your model and (2) new data.

General note on solving least squares

The solution to minimizing a sum of squares can be expressed as a solution to a linear system of equations. (See derivation here: Understanding linear algebra in Ordinary Least Squares derivation.) Systems of linear equations can be efficiently solved and you can check the accuracy of your gradient descent algorithm by simply comparing it to the solution you get by solving the linear system.

Eg.

b_gradient_descent = my_gradient_descent(y, X);
b_linear_system = linsolve(X'*X, X'*y);

or in Matlab, the b_linear_system = X \ y;

Improve this answer
$\endgroup$
3
  • $\begingroup$ So to be honest a lot of this is a bit over my head since I'm just starting out. I used mean squared error divided by (2*number of samples) to normalize. Is what you're saying that because of the spread of the training data the cost function can't get very low and if the training data were closer to a particular line I would get a lower cost? $\endgroup$
    – Austin
    Aug 10, 2016 at 19:21
  • 2
    $\begingroup$ @Jake Looking at your plot, a simple line can only fit the data so well (which is fine!). For each observation $i$, the error $e_i$ is the vertical difference between the data point and the line. Your algorithm (I assume) is trying to find a line to minimize $\sum_i e_i^2$. Because your data doesn't lie perfectly on a line, the minimum of $\sum_i e_i^2$ won't be zero! And that's perfectly ok. (Indeed if it were perfectly on a line, it would be a bit suspicious...) $\endgroup$
    – Matthew Gunn
    Aug 10, 2016 at 19:36
  • 1
    $\begingroup$ Assuming housesize is an n by 1 matrix and houseprice is an n by 1 matrix, the answer to minimize the sum of squares can be calculated using the MATLAB code b = [ones(length(price), 1), housesize] \ houseprice; That should approximately match your gradient descent algorithm results if all is good. $\endgroup$
    – Matthew Gunn
    Aug 10, 2016 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.