# Two groups are significantly different? Test to use

I have two groups of discrete data (integers):

  Group 1   Group 2
101       103
105       200
115       150
98       160
100       115
...       ...


and I need to know if they are significantly different or not. For this kind of tests I know there are useful tests such as t-test or Wilcoxon test. However, I was told that this kind of statistically tests are for continuous data (not my case).

Then, I used a Chi-squared test. However, assumptions are not met since there are a lot of cells with 0s. Even combining my data into bins (e. g. 1-19, 20-39, etc.), I have lots of cells with 0s. R throws this warning in this case:

Chi-squared approximation may be incorrect


I know as well there are the Montecarlo simulation. However, it is just a simulation and is always giving the same p-value, exactly the same p-value, for all my different datasets to be compared. I don't like this idea.

Fisher test is practically impossible due to the size of my datasets. It is possible to use Fisher test if I group my data into bins of 1000, quite wide bins. However, I don't like this idea neither.

In summary, do you know how can I deal with my data?

Notes:

• My data is not paired.
• Group 1 has about 30.000 observations while group 2 hardly has more than 4.000.
• Extremely skewed data. Example with two of my datasets: • Are you data paired? Jul 23, 2014 at 16:19
• No, my data is not paired. Jul 23, 2014 at 16:20
• 1) Are the integers counts? 2) you can have zero observed counts in cells with a chi-square. The issue is with low expected counts. 3) That you're getting 'exactly the same p-value for all your data sets' -- there's not enough detail here to tell what the issue is - what was this p-value and how were these simulations performed? It wouldn't happen to be that you used simulate.p.value=TRUE in chisq.test in R, with default B and got p= 0.0004998 would it? This is simply 1/(B+1). That's to be expected if there's a strong effect or large n Jul 23, 2014 at 18:51

It's true that the t-test is for continuous data.

However, looking at your few lines, your data might well be continuous enough.

Very few variables (if any) are purely continuous (you can get down to quantum level but no further) and none are recorded that way. Your data look like they could be IQs or weights (in pounds) or something like that and only a really anal person would object to comparing IQs using a t-test (even though IQ is measured in points).

• What if my data is extremely skewed? Jul 23, 2014 at 17:13
• Then the t-test isn't appropriate. Jul 23, 2014 at 17:20

You can still use tests like the t-test and the Wilcoxon test for your type of data. Some texts oversimplify data into categorical/nominal and numeric/continuous and describe the methods for those. Your data falls into what is called interval or probably even ratio data and while it is discrete, that has less of an impact on the tests. Having only integers can sometimes cause problems with small sample sizes (because it is clearly not normally distributed), but the Central Limit Theorem lets us use the t-test when sample sizes are large enough (and 4,000 is plenty big to overcome the discreteness, the only other concern would be extreme skewness or outliers).

The $\chi^2$ and Fisher's exact tests are designed for a small number of categories, not all the integers.

With the sample sizes that you describe you will have an incredible amount of power to detect even very small differences. One reason that the Monte Carlo methods may have kept giving you the same answer is if they all returned a value very close to 0 that would probably depend more on the number of simulations than the data itself.

• Thanks for answering. My data is extremely skewed and it is a concern, as you say, for t-tests. What to do in this case? Jul 23, 2014 at 17:05
• You mentioned the Wilcoxon test, that is fine if you are happy with the assumptions and what it is testing. I like permutation tests (One of the Monte Carlo tests). With the large sample sizes that you have, pretty much anything is going to give you a very significant result. Jul 23, 2014 at 17:22

The Wilcoxon-Mann-Whitney rank sum test does not assume continuous data. You can just go ahead and conduct a rank sum test.

Note that because the distribution of your two groups have different shapes, that the rank sum test will necessarily be a test of stochastic dominance, with H$_{0}\text{: P}(X_{\text{Coding}} > X_{\text{IncRNA}}) = 0.5$ (i.e. the probability of a random observation drawn from Coding is greater than a random observation drawn from InRNA is one half), with H$_{\text{A}}\text{: P}(X_{\text{Coding}} > X_{\text{IncRNA}}) \ne 0.5$ (i.e. one of the groups has greater than one half probability of a random draw being larger than the other, the sign of the test statistic tells you which).

Another option I haven't seen mentioned here yet is a bootstrap test. This will allow you to identify the sampling distribution of the test statistic empirically by resampling from your data, thus requiring no distributional assumption. I.e., like the Wilcoxon–Mann–Whitney U, this is a nonparametric analysis (though parametric bootstrapping is also "a thing").

As others have said, you'll probably have plenty of power to detect any difference, so you might consider focusing on an effect size estimate and maybe bootstrapping a confidence interval around it.