1
$\begingroup$

I have 5 datasets and for each one of them I created 2 prediction models. For such task I divided each dataset in training and testing set (70/30%). Then I assessed the RMSE for each one and calculated the mean of the 5 RMSE for each model.

My question is can I compare my results with other results given that they used another dataset(s)? How can I say that my results are significantly better than the others?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

RMSE is a scale-dependent measure of forecast accuracy. If the scales of your dependent variables differ across the cases, RMSEs from the different cases will not be comparable. For example, if you predict price in dollars in one case and price in thousands of dollars in another case, comparing RMSEs directly will not make sense.

If you want to compare RMSEs for completely different dependent variables, such as daily values of S&P500 stock price index and monthly temperature in Chicago, you have to be aware of the fact that some things are intrinsically easier to predict than other. For example, if you guess the direction of change in S&P500 correctly 55% of the time, that might not seem impressive on absolute terms but it would be very impressive given the subject-matter context. Meanwhile, if you guess the direction of change in temperature in Chicago correctly only 55% of the time, that would be really low as almost any other forecast source would do better.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The output variable is the same: a certain value 60 minutes ahead. The data is on the same units. Where can I find a source that states what you said and learn a bit more about it as I need to explain this to other people? $\endgroup$
    – user25847
    Commented Oct 8, 2015 at 15:48
  • $\begingroup$ Why don't you give them a trivial example and try make it as intuitive as possible. My first example with dollars and thousands of dollars is not elaborate enough but you can expand it. My second example should be intuitive enough. I do not remember a good reference to cite right now... $\endgroup$
    – Richard Hardy
    Commented Oct 8, 2015 at 15:53
  • $\begingroup$ Does the dataset size influence the way we compare results, i.e., if their dataset is bigger than mine (and in the other way too) are we still able to compare the RMSE? $\endgroup$
    – user25847
    Commented Oct 8, 2015 at 16:07
  • $\begingroup$ The dataset size may influence the confidence bound around the estimated RMSE (large dataset ~ narrow confidence bound, small dataset ~ wide confidence bound) but should not bias the point estimate. $\endgroup$
    – Richard Hardy
    Commented Oct 8, 2015 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.