1
$\begingroup$

I understand in general MSE, RMSE and MAE means average distance between the actual and predicted value, and the lower the MSE, RMSE and MAE, the better the model fits the dataset.

I try to understand these concepts more thoroughly by interpreting the following result - I have used model 1 and model 2 to predict each student group’s exam score based on several independent variables. Be noted, student1,2,3 and so on are group of student, not a single student.

Based on the result, model 2, in general, have higher MSE, RMSE and MAE(meaning model1 fits the model better than model 2). However, I try to understand how do I preciously interpret the result for each of these metrics and the overall data fitting performance between model 1 and model 2?

enter image description here

$\endgroup$
4
  • $\begingroup$ Your screenshot confuses me. It seems to me that what you call MSE in your screenshot is $\left(y_i-\hat{y_i}\right)^2$, whereas the MSE is defined as $\mathbb{E}\left(\left(y_i-\hat{y_i}\right)^2\right)$. This means that you should compute a single MSE/RMSE/MAE for each model, not for each observation. But if that was what you did, we should have $RMSE_i=MAE_i$. So I am not sure to understand what is displayed in your screenshot. $\endgroup$
    – FP0
    Commented Jul 23, 2022 at 23:49
  • $\begingroup$ It was a typo. What I meant by student 1, 2 3 and etc are group of students, not a single student $\endgroup$
    – user032020
    Commented Jul 23, 2022 at 23:55
  • $\begingroup$ Related: Mean absolute error OR root mean squared error? $\endgroup$ Commented Jul 24, 2022 at 6:34
  • $\begingroup$ Thanks Stephan, that’s helpful $\endgroup$
    – user032020
    Commented Jul 24, 2022 at 14:12

1 Answer 1

1
$\begingroup$

I think that you already understands what there is to understand. These statistics are used to measure the average precision of prediction models, and to compare the accuracy of different models. You already noted that:

Based on the result, model 2, in general, have higher MSE, RMSE and MAE(meaning model1 fits the model better than model 2).

You noted that the accuracy of model 1 is higher than the accuracy of model 2 in average. I do not think that you can interpret anything further.

The only additional observation I could think of is the following: For the groups "student5" and "student9", you have $MAE_1<MAE_2$ but $RMSE_1>RMSE_2$. This suggests that in these groups, model 1 produces some "significant" (do not interpret this word in a statistical way) outlier, which increases the MSE due to the squared error.

$\endgroup$
2
  • $\begingroup$ Thanks. This observation is extremely helpful. That means the absolute difference,MEA, in model 1 is smaller than model 2, however, there’s outliner in the score among these 2 groups, so when we calculate the square root of the average squared differences, model 1 has larger RMSE. Can I understand this relates to a potential data quality issue and this observation has nothing to do with concluding model 1 fits the dataset better ? $\endgroup$
    – user032020
    Commented Jul 24, 2022 at 1:37
  • $\begingroup$ You're welcome. Yes, this is different from the general conclusion that model 1 has a better fit than model 2. An outlier observation/prediction error could be related to some data quality issue, but it could simply be an outlier regarding the predictions. $\endgroup$
    – FP0
    Commented Jul 24, 2022 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.