2
$\begingroup$

I have implemented linear regression manually, for learning purposes, and I use the "Auto MPG" data set as the toy data I'm applying it to.

It occured to me that I don't know how to test the efficiency of my model! With classification I can check the predicted class vs the actual class, but what do I do with regression?

EDIT: Of course, I am splitting the data I have into train + test data...I am speaking of evaluating the results when I apply the model on the test data.

For instance, using my model I predict 16.76 mpg for a car for which the actual value in the data set is 15.5. How do I decide whether this is a "good" or "bad" prediction? I am thinking of using some thresholds ("if the predicted value is within the [actual-epsilon, actual+epsilon] interval => OK!"), but is this a good approach? And even if it is, how do I choose epsilon values?

I am aware of the fact that there's most likely no clear-cut answer, but any suggestion regarding the approach to take would be most helpful.

EDIT: I've decided to go with the following approach: for each test data sample, I am computing how far the predicted value is from the test data as 

abs(predicted_value - actual_value) * 100 /  actual_value

i.e. how far away is the prediction from the actual value, expressed in % of actual value. Any feedback is still welcome, since this is just an idea I've had, and I'm not sure if it is "best practice".

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

I suggest using RMSE (root mean square error) of your predictions on your test set when compared to the actual value. This is a standard method of reporting prediction error of a continuous variable. You can use R^2 to examine how well your model fits the training data.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks a lot for the suggestion, this is exactly what I was looking for! So I would do sqrt(sum((predicted_i - actual_i)^2)/#samples), where i=1..#samples? $\endgroup$
    – ACEG
    Commented May 11, 2012 at 13:37
  • $\begingroup$ Yes, thats correct. $\endgroup$
    – BGreene
    Commented May 11, 2012 at 13:57
  • $\begingroup$ I know this is a bit old, but just to double check, the RMSE is a scalar value correct? And the optimal value would be 0? $\endgroup$
    – KingPolygon
    Commented Sep 24, 2015 at 20:57
  • $\begingroup$ Yes. It is a scalar value. A perfectly fitted model would have RMSE of 0. $\endgroup$
    – BGreene
    Commented Dec 2, 2015 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.