2
$\begingroup$

I have implemented linear regression manually, for learning purposes, and I use the "Auto MPG" data set as the toy data I'm applying it to.

It occured to me that I don't know how to test the efficiency of my model! With classification I can check the predicted class vs the actual class, but what do I do with regression?

EDIT: Of course, I am splitting the data I have into train + test data...I am speaking of evaluating the results when I apply the model on the test data.

For instance, using my model I predict 16.76 mpg for a car for which the actual value in the data set is 15.5. How do I decide whether this is a "good" or "bad" prediction? I am thinking of using some thresholds ("if the predicted value is within the [actual-epsilon, actual+epsilon] interval => OK!"), but is this a good approach? And even if it is, how do I choose epsilon values?

I am aware of the fact that there's most likely no clear-cut answer, but any suggestion regarding the approach to take would be most helpful.

EDIT: I've decided to go with the following approach: for each test data sample, I am computing how far the predicted value is from the test data as

abs(predicted_value - actual_value) * 100 /  actual_value

i.e. how far away is the prediction from the actual value, expressed in % of actual value. Any feedback is still welcome, since this is just an idea I've had, and I'm not sure if it is "best practice".

$\endgroup$
3
$\begingroup$

I suggest using RMSE (root mean square error) of your predictions on your test set when compared to the actual value. This is a standard method of reporting prediction error of a continuous variable. You can use R^2 to examine how well your model fits the training data.

$\endgroup$
  • $\begingroup$ Thanks a lot for the suggestion, this is exactly what I was looking for! So I would do sqrt(sum((predicted_i - actual_i)^2)/#samples), where i=1..#samples? $\endgroup$ – ACEG May 11 '12 at 13:37
  • $\begingroup$ Yes, thats correct. $\endgroup$ – BGreene May 11 '12 at 13:57
  • $\begingroup$ I know this is a bit old, but just to double check, the RMSE is a scalar value correct? And the optimal value would be 0? $\endgroup$ – KingPolygon Sep 24 '15 at 20:57
  • $\begingroup$ Yes. It is a scalar value. A perfectly fitted model would have RMSE of 0. $\endgroup$ – BGreene Dec 2 '15 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.