2
$\begingroup$

I'm not very familiar in statistics so please bear with me. I have two datasets which consist of four attributes:

order, original, prediction, absolute_difference(original-prediction)
  1. order is just numbers from 1 to n
  2. original is real measured value
  3. prediction is predicted value
  4. absolute_difference is absolute difference between original and prediction.

If the prediction is perfect then original and prediction should match (and absolute difference should be 0). Those data represent amount of electrical power consumed by a corporation's office building as a function of time. I want be able to distinguish which dataset contains better prediction and somehow quantify this value. For such tasks correlation should be fine (in this case correlation between original and prediction). But I've found that correlation is not good when your data follows something else but linear, and this is my case: there are many peaks, repeating cycles, random events at particular days, etc. Here are multiple methods for comparing datasets. But I'm not very familiar with them. My intuitive approach was: calculate mean from absolute differences and use this value as discriminator between datasets. The lower the mean is, the better the prediction is. Then I've realized that there is also standard deviation which can be calculated from those absolute differences of original and predicted values. Next step would be to pick up all values which have absolute differences of original and predicted value:

  1. above mean + standard deviation
  2. bellow mean - standard deviation

This should give me an overview of how many values fall in this interval and how many doesn't. The dataset where more values fall in the interval is dataset with better prediction. Does this make sense?

PS: Please consider also following two cases: is the method for evaluating best prediction different or it is the same for both:

  1. original is the same for both datasets but prediction might vary
  2. original (and logically prediction) vary between datasets

PPS: Quote from this site says:

The standard deviation is used in conjunction with the mean to summarise continuous data, not categorical data. In addition, the standard deviation, like the mean, is normally only appropriate when the continuous data is not significantly skewed or has outliers.

I'm not sure if I understand correctly, but categorical data is something else than discrete data but discrete data are not continuous I suppose. So I'm not sure if I can use standard deviation for my purposes?

PPPS: There are also metrics such as string distance or Mahalanobis distance which also compare the similarity between sets but I guess is not what I want. This leads me to assumption that particular method is good for particular dataset. If yes is there some cheat sheet or rule of thumb or something which will tell me appropriate method for particular dataset?

Improve this question
$\endgroup$
1
  • $\begingroup$ What you are seeking is, essentially, a penalty function. The "problem" you have is that you are finding that there are many penalty functions available. You could penalize overprediction more than underprediction, for example. I think you should first plot your two datasets. First instinct will be to plot predicted versus observed and that is good. Second plot should, perhaps, be a Bland-Altman plot. But you can devise any penalty function you think relevant. The shape of the curve is not necessarily relevant here. Your predictions are already made. Bias and Error. $\endgroup$
    – StatNoodle
    Commented Jan 13, 2016 at 5:10

1 Answer 1

Reset to default
1
$\begingroup$

As far as I understood, you need to compare which prediction method performed better?

If that's correct than I think you need to do a paired T-test between the absolute_difference column for both the methods. It might be better to do it on just the difference column.

T-test are pretty standard and you can find about them with a simple google search.

In case of two different input datasets, the comparison is not valid as you're not evaluating on common grounds.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you, this is exactly what I'm trying to do. Are you saying that I should use difference instead of absolute difference? Please check "PS" section, I suppose that the T-test will work in 1st case where datasets are the same but prediction differs. But what to do in 2nd case where I have same prediction but two different datasets and I want to know on which dataset the prediction method gives better results. Is this even possible? $\endgroup$
    – Wakan Tanka
    Commented Jan 13, 2016 at 11:00
  • $\begingroup$ @WakanTanka For evaluating two models, you must have a common ground, which in our case is the input data. If you keep input data different for each model, the comparison is not valid, because you don't how one model does on other model's data. $\endgroup$
    – Ujjwal Kumar
    Commented Jan 13, 2016 at 12:05
  • $\begingroup$ So that 2nd case is not a valid test $\endgroup$
    – Ujjwal Kumar
    Commented Jan 13, 2016 at 12:05
  • $\begingroup$ Seems to me that there must be way because some machine learning algorithms fits for some data and another fits for another so people that claims their ML is good for particular data type must do some prior research and compare the results between various datasets. Maybe not T-test but something different, I do not know. PS: I've checked T-test but as I said statistics is not my cup of coffee. There are also various T-tests and I do not know which to pick up. Would you please try tailor it on my data? Some python, R or maybe math explanation would be fine. Thank you very much. $\endgroup$
    – Wakan Tanka
    Commented Jan 13, 2016 at 12:10
  • $\begingroup$ An ML Algorithm is doing better on a certain problem (dataset) means that other models are not doing that great on the same dataset. Another way to not use exactly same datasets would be to use large datasets from the same domain, which by nature tend to be statistically equivalent $\endgroup$
    – Ujjwal Kumar
    Commented Jan 13, 2016 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.