2
$\begingroup$

I am trying to analyze some results I have generated from multiple different models. I have 21 models which are all used on the same 9 sets (years) of data. The response variable is the result of the running the model given the specified year. No randomization was used and all models were run for all years. The models are independent of each other and reset after each run, so running one model will have no effect on running the next.

This is a sample of how my data is formatted:

YEAR     V1   V2   V3   V4   V5   V6   V7   V8   V9   V10  etc...
year 1   2    2    3    10   2    4    11   2    2    2
year 2   5    3    2    2    2    2    2    10   2    9
year 3   3    9    4    1    2    2    7    4    3    3  
year 4   5    10   2    4    5    10   2    2    2    2  
year 5   7    5    2    9    2    2    2    2    4    7
year 6   2    5    7    2    8    9    8    3    7    7
year 7   9    5    3    1    2    2    8    2    2    4
year 8   5    7    4    2    7    5    2    2    2    1
year 9   2    8    9    5    2    5    7    2    3    2

I want to average the results for the 9 years for each model and determine if any models are significantly different from each other. What test would I run to determine this?

$\endgroup$
2
$\begingroup$

If I understand this correctly. you are mainly interested in looking for significant differences among Models (columns) in your table. Also, it seems a stretch to assume these data are normal.

So, I would suggest a nonparametric Friedman test, using Years as blocks. [In effect we're assuming Years (blocks) differ, and the Friedman procedure will provide no P-value for differences among Years.]

If (and only if) the Friedman test says there are significant differences among Models, you could do a few ad hoc tests to see which differences that look interesting are actually statistically significant. For those comparisons, perhaps use Wilcoxon signed rank tests on yearly differences for each pair investigated.

Also, to avoid 'false discovery', declare significant differences among Models only if the Wilcoxon test gives a P-value well below 5%, according to Bonferroni's method.

Potentially, there are ${21 \choose 2} = 210$ pairs of Models to investigate. If you were to look at all of them at the 5% level, you might find about ten 'significantly different' pairs out of the 210 by chance alone--even if all Models are essentially the same.

Some combination of Bonferroni protection and personal restraint would be appropriate when looking at differences between pairs of Methods.

$\endgroup$
  • $\begingroup$ Thank you so much! After extensive research I did begin to wonder if the Friedman test would be the best option but I still doubted myself on how to organize everything. I am really expecting the differences between the models to be statistically insignificant, but of course I need some numbers to back up that claim. $\endgroup$ – eCoder Jul 8 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.