We have a peculiar problem, for which we have many questions, but cannot seem to find satisfactory answers (we did find a few ideas as to what to do, but we are not very certain they apply to our problem). If you find any of the following questions to have already been answered, and those answers should apply to my problem, then a pointer in the right direction would be very much appreciated.
First some info about the data. Different models (that work differently) are feed the same data from which the models produce a numeric value for each input. These numbers become our (scale variable) data. We would like to use these values to show with some statistical certainty that different models produce different (or not) results. For this we planned on using the Wilcoxon signed rank test or the Sign test (depending on the distribution of differences; the related samples t-test is not really an option, because the differences are never normally distributed).
The problem becomes the sample size that we use. In total each model produces 12million values. We have read that big samples tend to lead to significant differences between groups even though there might not be one (the samples mentioned however, were always much smaller). Legacy dialogs tests in SPSS confirm this by returning the sig. of 0.0 followed by 35 more 0 (which is as far as we can tell the most decimal places one can get out of SPSS) for all the combinations of models (even though some look somewhat similar from their descriptive statistics and histograms). What further confuses us is the use of Wilcoxon sign rank test in the "new" dialog, where sig. value for all the pairs of data equals 1.0 (even for models that are clearly very very different). We cannot find a suitable reason for such differences between what should be, as we understand it, the same test (we do realize that a different functions are been called in the background (NPAR TESTS and NPTESTS), we are not however aware how this would cause such drastic differences; the fact that sig. values alternate between 0 and 1 makes us think that something is very wrong (let us just add here that Sign test returns sig. 0.00 in the "new" dialog as well).
So, to sum up we are interested to know:
- is there any point in even using tests like Wilcoxon signed rank test or the Sign test on sample sizes this big?
- could some other method me used instead (or should we just use descriptive statistics)?
- and what causes the differences in the test results for the same test in different dialogues?
Finally we would like to mention that we are not competent statisticians (or mathematicians) so we would ask you to be tolerant if we have made a mess of things ;). We would be happy to provide any more information. Any help would be appreciated.
Edit
Apparently the best solution to our problem would be to use the effect size. We chose to expand on the original post rather than ask a new question, because the circumstances are connected (although maybe that would be a better option).
Anyway our current plan is to first do a Friedman test to show there are differences between models, followed by the Wilcoxon and Sign tests and report that sig. is smaller than $\frac{0.01}{Bonfferoni\hspace{2mm}correction}$ (and kind of gloss over the fact that as far as SPSS is concerned it's complete 0) - is it viable to use different tests for Post-hoc testing (Wilcoxon and Sign tests) or would it be more appropriate to just use Sign tests?. After that we will calculate the effect size. We are considering using the $r = \frac{z}{\sqrt{N}}$ calculation as suggested here. We believe this formula can also be applied to the Sign test, but we are not certain? Would it then be reasonable to also add The Common Language Effect Size Statistic? Basically what we are asking is if this sequence of tasks is reasonable and viable.
Thanks to everybody for the previous and future help ;).