1
$\begingroup$

We have a peculiar problem, for which we have many questions, but cannot seem to find satisfactory answers (we did find a few ideas as to what to do, but we are not very certain they apply to our problem). If you find any of the following questions to have already been answered, and those answers should apply to my problem, then a pointer in the right direction would be very much appreciated.

First some info about the data. Different models (that work differently) are feed the same data from which the models produce a numeric value for each input. These numbers become our (scale variable) data. We would like to use these values to show with some statistical certainty that different models produce different (or not) results. For this we planned on using the Wilcoxon signed rank test or the Sign test (depending on the distribution of differences; the related samples t-test is not really an option, because the differences are never normally distributed).

The problem becomes the sample size that we use. In total each model produces 12million values. We have read that big samples tend to lead to significant differences between groups even though there might not be one (the samples mentioned however, were always much smaller). Legacy dialogs tests in SPSS confirm this by returning the sig. of 0.0 followed by 35 more 0 (which is as far as we can tell the most decimal places one can get out of SPSS) for all the combinations of models (even though some look somewhat similar from their descriptive statistics and histograms). What further confuses us is the use of Wilcoxon sign rank test in the "new" dialog, where sig. value for all the pairs of data equals 1.0 (even for models that are clearly very very different). We cannot find a suitable reason for such differences between what should be, as we understand it, the same test (we do realize that a different functions are been called in the background (NPAR TESTS and NPTESTS), we are not however aware how this would cause such drastic differences; the fact that sig. values alternate between 0 and 1 makes us think that something is very wrong (let us just add here that Sign test returns sig. 0.00 in the "new" dialog as well).

So, to sum up we are interested to know:

  • is there any point in even using tests like Wilcoxon signed rank test or the Sign test on sample sizes this big?
  • could some other method me used instead (or should we just use descriptive statistics)?
  • and what causes the differences in the test results for the same test in different dialogues?

Finally we would like to mention that we are not competent statisticians (or mathematicians) so we would ask you to be tolerant if we have made a mess of things ;). We would be happy to provide any more information. Any help would be appreciated.

Edit

Apparently the best solution to our problem would be to use the effect size. We chose to expand on the original post rather than ask a new question, because the circumstances are connected (although maybe that would be a better option).

Anyway our current plan is to first do a Friedman test to show there are differences between models, followed by the Wilcoxon and Sign tests and report that sig. is smaller than $\frac{0.01}{Bonfferoni\hspace{2mm}correction}$ (and kind of gloss over the fact that as far as SPSS is concerned it's complete 0) - is it viable to use different tests for Post-hoc testing (Wilcoxon and Sign tests) or would it be more appropriate to just use Sign tests?. After that we will calculate the effect size. We are considering using the $r = \frac{z}{\sqrt{N}}$ calculation as suggested here. We believe this formula can also be applied to the Sign test, but we are not certain? Would it then be reasonable to also add The Common Language Effect Size Statistic? Basically what we are asking is if this sequence of tasks is reasonable and viable.

Thanks to everybody for the previous and future help ;).

| cite | improve this question | |
$\endgroup$
  • 4
    $\begingroup$ As you allude to, with these volumes of data significant results become the rule rather than the exception. Would it not therefore make sense to stop thinking about statistical significance and instead think about whether the difference between the models have any practical consequence (i.e. stop thinking about $p$-values, and begin thinking about effect sizes). A meaningful question may not be "Do these models produce different results?", but rather "Is the difference between these models big enough that it may be detrimental to our work?" $\endgroup$ – Ian_Fin Oct 5 '16 at 9:23
  • $\begingroup$ Check stats.stackexchange.com/questions/2516/… and stats.stackexchange.com/questions/125750/sample-size-too-large $\endgroup$ – Tim♦ Oct 5 '16 at 13:34
  • $\begingroup$ The difference between the two formats of the test looks like either (a) a programming error (b) a change in the way ties are being handled. If you are sure you are doing the same test then I suggest trying on an SPSS specific forum, complaining to the manufacturer, trying different software. $\endgroup$ – mdewey Oct 5 '16 at 16:03
  • $\begingroup$ There are many questions on site that discuss hypothesis tests with large sample sizes and the distinction between statistical significance and practical importance. Dozens at least. $\endgroup$ – Glen_b Oct 5 '16 at 23:10
  • $\begingroup$ @mdewey , We have tried to find another similar case on the internet, but couldn’t really find anything that was the same problem, and we do find it kind of hard to believe we are the first to stumble upon a programming error. As for the how the ties are handled - there is only 18 out of 12M comparisons, so we don’t believe that would make a big difference. $\endgroup$ – kronos Oct 6 '16 at 9:09
0
$\begingroup$

You say "We have read that big samples tend to lead to significant differences between groups even though there might not be one". This is not correct. Significant differences will be found correctly, assuming that the tests are correctly applied. The problem you are facing is that differences that are statistically significant may not be substantively significant. Reframing the question in terms of effect sizes or some other idea of how big a difference needs to be to matter substantively may be a better path.

As for precision of the results in Statistics, double precision floating point calculations will never yield more than about 17 significant figures (using significant in yet a third way here), but the magnitude can vary much more. However, the numerical algorithms used to calculate significance, such as the Normal cdf, all have their own precision limitations. None of this matters in calculations of statistical significance, since the precision extends well beyond the point where differences would matter.

You should not be seeing a difference between the NPAR and NPTEST results, so I suspect that they are being applied differently. If you want to compare the two outputs, you might find it helpful to set the newer procedure to produce pivot tables instead of model viewer output. You can do that via Edit > Options > Output in the Output Display box.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ JKP and @Ian_Fin have both suggested using effect size. As we understand it effect size is used on top of tests for statistical sig., or can they be used independently (the sig. results just look unbelievable)? Effect size looks to us as it might be the way forward, but at this point we feel like we are not familiar enough with the use of it so we would just ask if anybody knows of an example with similar circumstances or even where we should look for some (everything we find is ether different or is just very theoretical)? $\endgroup$ – kronos Oct 6 '16 at 9:58
  • $\begingroup$ also thank you for the pivot tables info., we were not aware of that option. Unfortunately it doesn't seem to produce any more data from which we could see the reason for the differences (although in this view, it is possible to change the number of decimal places visible, which was not before). $\endgroup$ – kronos Oct 6 '16 at 10:01
  • $\begingroup$ One way to increase your understanding about the relationship among power, significance, sample size, and effect size would be to explore this using the STATS POWER extension command (V24 or later) or the STATS POWER23 equivalent for V23 and earlier. They differ only in that the dialog box for this command is only available for V24+, but the syntax help can be displayed by pressing F1 on an instance in the Syntax Editor. The command can be installed via the Extensions menu in V24 or the Utilities menu in V22 or 23. They require the R Essentials as well. $\endgroup$ – JKP Oct 7 '16 at 16:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.