2
$\begingroup$

I am working on a machine learning regression problem and I have chosen the metrics Mean Absolute Error (MAE) and 'Mean Squared Error (MSE). I have 3 features and two of them have values in the range 0.007 - 0.009 and the third feature's values range from 1.18 to 1.19. The predicted value/output should also lie in the range 0.007 - 0.009. When I implement a simple linear regression model using scikit learn in Python, I get the MSE to be about 2.037727147668752e-07. However I noticed if I multiplied all my features and the value to be predicted by say 100, the MSE changed to 0.0024. Can someone please explain

  1. If the MSE is a metric that is to be used on a relative scale, how do I interpret it? Does it mean an error of 0.002 means that if my actual value is 0.008, my predicted value is 0.008 +/- 0.002 = 0.006 or 0.01?

  2. This is a large difference between the actual and predicted values, are there any specific regression machine learning models that work well for this kind of problem? Will normalizing the data or scaling it help improve performance and if so, why?

  3. I noticed that MAE remained constant regardless of the scale. Is this an absolute error measure?

  4. What other metric can I use to evaluate the performance of my model?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

When I implement a simple linear regression model using scikit learn in Python, I get the MSE to be about 2.037727147668752e-07. However I noticed if I multiplied all my features and the value to be predicted by say 100, the MSE changed to 0.0024.

When you multiply your training data by 100, then your predictions will also change by a factor of (about) 100. The MSE is the mean of the squared differences between actuals and predictions. If you scale both actuals and (roughly) predictions by a factor of 100, the difference is also scaled by 100, so the square of the difference is scaled by 10,000. It works out. The features don't have anything to do with this effect.

If the MSE is a metric that is to be used on a relative scale, how do I interpret it? Does it mean an error of 0.002 means that if my actual value is 0.008, my predicted value is 0.008 +/- 0.002 = 0.006 or 0.01?

The MSE is not a relative measure. It is just the mean of the squared errors. Yes, this is hard to interpret. You may want to look at Mean absolute error OR root mean squared error?

This is a large difference between the actual and predicted values, are there any specific regression machine learning models that

work well for this kind of problem? Will normalizing the data or scaling it help improve performance and if so, why?

Scaling and normalizing will usually not help (except that scaling will scale the MSE, as above, but that is not helpful). Without knowing much more about your data, the best we can do is suggest How to know that your machine learning problem is hopeless?

I noticed that MAE remained constant regardless of the scale. Is this an absolute error measure?

This should not happen. The MAE is the mean of absolute errors. Scaling the actuals (and therefore also the predictions) should scale the MAE by the same amount.

What other metric can I use to evaluate the performance of my model?

This may be helpful - it's written in the context of time series forecasting, but you can apply it in other contexts, too.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thank you for your detailed answer, I have a clearer understanding of MAE and MSE now. I checked the MAE and you are right, it does change according to the scale. I am a beginner to machine learning regression problems, hence there are two things I would like to ask of you - a) how would you evaluate the performance of a regression model and b) are there any specific resources you could suggest for me to improve my understanding of performance metrics used for machine learning regression problems and to improve my skills in analysing and solving regression problems? $\endgroup$
    – user292709
    Jul 29, 2020 at 14:00
  • 1
    $\begingroup$ Hm. I think the MSE is typicall a good choice. It's not very easy to interpret, but it's the measure your model typically optimizes on, and it will draw you towards an unbiased prediction. I can't recommend much more than the first link I added. Good luck! $\endgroup$
    – Stephan Kolassa
    Jul 29, 2020 at 14:05
  • $\begingroup$ I've gotta say, your comment "Allow me a tiny bit of honesty: if you need help at this level, I don't know whether you should be doing statistics/machine learning. Sorry to be blunt." is completely unacceptable. A lot of people are struggling to learn statistics and machine learning. It's a very difficult topic. Suggesting someone should stop altogether is terrible advice. $\endgroup$
    – Danny Bullis
    Feb 19, 2022 at 5:25
  • $\begingroup$ @DannyBullis: thank you for your comment. I do understand how you feel, and I agree that my comment may have come across as harsh. Nevertheless, I stand by it. I have written a lot of answers here, and I very rarely write anything like this. If you do statistics or machine learning but have problems with what influence squaring a number has, then you first need to learn the basics. I see it as analogous to a doctor asking what end of the stethoscope goes in the ear. If you feel strongly about my choice of words, perhaps you would like to flag it to the mods (or have already done so)? $\endgroup$
    – Stephan Kolassa
    Feb 19, 2022 at 9:14
  • 3
    $\begingroup$ You're basically telling the person asking the question that if they don't understand the problem they shouldn't be doing that kind of work. Doesn't that defeat the whole purpose of asking questions or seeking help on StackOverflow? Your opinion offers no value other than to highlight the OP's lack of knowledge in this area. You also apologize for being blunt but you are not sorry at all. The answer could be improved simply by removing that comment. I could edit the post but I think it's important for you to understand the impact you have on others. As a SO veteran you should lead by example. $\endgroup$
    – Danny Bullis
    Feb 19, 2022 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.