1
$\begingroup$

I have a sample with millions of points. Every point $x$ has two values associated, let's say $A_{x}$ and $B_{x}$. I calculated the mean values of A and B in the sample:

$\overline{m}_{A} = 0.19 \quad \text{and} \quad \overline{m}_{B} = 0.21$

The distributions of the values are different and not normal. And the standard deviations of the means are negligible.

Can I say that the means are statistically significantly different? Should I calculate the Wilcoxon signed-rank test to prove it?

Improve this question
$\endgroup$
9
  • 4
    $\begingroup$ Don't use $\mu$ for sample means. The convention in statistics is usually that Greek letters are population parameters ($\mu_A$ is the population mean of the A's) and Roman letters are used for sample quantities ($\bar x_A$ as sample mean). The Wilcoxon signed rank test carries assumptions which - if your assertions are correct - will not be satisfied. Further, unless you make additional assumptions, it doesn't test mean shift, but something else. The sign test might have its assumptions satisfied, but it also won't be a test of means without further assumptions (that probably won't hold). $\endgroup$
    – Glen_b
    Jun 28, 2013 at 6:37
  • $\begingroup$ You may be able to get somewhere with a resampling based test, but even there, there are assumptions that mightn't be satisfied. $\endgroup$
    – Glen_b
    Jun 28, 2013 at 6:38
  • 3
    $\begingroup$ (1) You can't possibly satisfy the first assumption listed there; (2) ... which doesn't matter because the first assumption is nonsense; (3) There's a vital assumption that was removed more than a year ago - see assumption 4 ... and from the look of the changes that are often made on that page, I would treat that particular page as pretty much a random phrase generator. e.g. I don't think there's been a single time on that page when all of the assumptions have been correct. $\endgroup$
    – Glen_b
    Jun 28, 2013 at 12:46
  • 2
    $\begingroup$ On topics where many people think they have expertise, thay want to jump in and "fix" it on Wikipedia. Basic stats is one of those -- so pages on topics like histograms or simple nonparametric tests get "fixed" by well-meaning people every day or so, often to its detriment. A good way to actually understand the assumptions is to work out how to enumerate the permutations for each value the test statistic takes. The required assumptions are usually immediately obvious. Failing that, get a good book by someone who has published papers in the area - in a good journal. $\endgroup$
    – Glen_b
    Jun 29, 2013 at 0:16
  • 2
    $\begingroup$ With millions of points, there's probably no point in a hypothesis test at all (unless the sds are gigantic relative to the difference in means), but in any case the important assumptions then will relate to bias in sampling and independence of the pair-differences. You don't need nonparametrics - under mild conditions (e.g. variances exist) you can rely on CLT and Slutsky's theorem to say a one-sample t-statistic has a normal distribution. $\endgroup$
    – Glen_b
    Jun 29, 2013 at 0:20

1 Answer 1

Reset to default
4
$\begingroup$

With millions of points the SEM will often be "negligible". However, you should ask yourself about the effect size, as "statistically significant" $\neq$ "significant". There is always an effect, and with sufficient sample size, you will be able to show that it is significant.

Last but not least, the values are paired ($A_x$ and $B_x$ are paired for a given $x$). Therefore, rather than comparing the distribution of $A$ and $B$, you probably should calculate the difference $\Delta_x = A_x - B_x$ and focus your analyses on that. For this, Wilcoxon signed rank test seems appropriate.

Finally, ask yourself the question -- aren't you comparing apples to oranges? What is the question that you want to answer? (apart from "do I need to run some statistics to get my paper accepted")

Improve this answer
$\endgroup$
2
  • $\begingroup$ First thanks. Second, considering the central limit theorem, can't we calculate the standard error of the mean? Third, I want to focus on the difference $\Delta_{x}$, that's the reason why I consider to use the Wilcoxon signed rank test. And finally, I want to find what method is the best: A or B. And the best method will have the lowest mean. $\endgroup$
    – Medical physicist
    Jun 28, 2013 at 9:00
  • $\begingroup$ You are right about the SEM. And you are right about the Wilcoxon signed rank test. I need to buy glasses (rank sum, signed rank...) $\endgroup$
    – January
    Jun 28, 2013 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.