Interpret Mann–Whitney U Test with highly skewed data

Question

I have data for ~102,000 hands of bananas. The dataset has four data points:

• Spiders present (True/False)
• Count of rotten bananas in hand
• Count of total bananas in hand
• Percent of bananas that are rotten in each hand (rotten/total)

Overall, spiders were present in 6,600 (6.5%) of banana hands. In ~102,000 hands of bananas, there are 557,000 bananas, of which 32,700 (5.9%) were rotten. In the hands where spiders were present, the rotten rate was 9.9%, and in the no spiders banana hands, the rate was 5.2%. From here, it

Hypothesis: Spiders prefer banana hands with higher rates of rotten bananas (though they do not exclusively inhabit rotten banana hands).

I would like to statistically test and prove this. Spiders-present and no-spiders present have very different numbers of groups. The data is skewed data. The vast majority of banana hands do not have any rotten bananas. Of the 6,600 spider-present banana hands, 56% do not have any rotten bananas (3,696 hands) and in the no-spider-present banana hands, 86.8% (82,807 hands) have no rotten bananas. There are lots of zeroes.

In this scenario, this is not a sample of the total world's bananas but the full population of bananas.

I calculated the following frequency table for spiders present and the rotten rate (pandas/python):

OBS	BANANAS	ROTTEN	SPIDERS	MIN	25P	MEDIAN	75P	MAX	MEAN	STD
6,600	36,041	3,568	TRUE	0	0	0	0.13333	1.0	0.139540699	0.086308
95,400	520,959	27,090	FALSE	0	0	0	0.0	1.0	0.05970877	0.180351

Given the high number of zeros, and the very skewed data, I was looking for nonparametric statistical tests to explain the difference in the rotten rate between the two categories (present, not present) of banana hands.

I found the Mann–Whitney U test. Running the test, I find I have incredibly small p-value (1.25e104, yes 104). The returned statistic value is astronomical compared to the descriptions, 224031919.

I believe this is the appropriate test and is valid based on these four assumptions of my data because:

1. My dependent variable, rate of rotten bananas, is ordinal/continuous
2. My independent variable is categorical (spiders present, spiders not present)
3. There is independence of observations. Each banana occurs only once in each hand.
4. My data is not normally distributed

My questions to you:

• How can I interpret these results?
• Should I be wrangling my data to make it more manageable?
• Is there a different test that you'd suggest? If yes, what is it, and why?
• Is there something else I should be considering? If yes, what is it and why?

Answer 1

Testing might be appropriate, if you have samples and want to make inferences about populations from which the samples were randomly taken.

If you want to do a formal test of hypothesis to compare two percentages (say 5.5% with spiders and 9.9% without, or 5.5% in group A with 4.4% in group B), then you have to use numerator and denominator for each percentage.

Tests of two proportions. For example, if you had 10000 bananas of which 550 are rotten in A, and you had 5000 of which 220 are rotten in B, you could use prop.test in R to see if this difference is statistically significant at the 5% level, the answer is Yes because P-value $0.0045 < 0.05 = 5\%.$ however, if you're comparing $55/1000$ with $22/500,$ then No.

prop.test(c(550, 220), c(10000, 5000))$p.val
[1] 0.004530416

prop.test(c(55, 22), c(1000, 500))$p.val
[1] 0.4318846

Chi-squared tests. Somewhat similarly, one could use a chi-squared tests on contingency tables of counts of rotten and non-rotten bananas in several groups to see if the proportion of rotten bananas in the several groups are essentially the same or are significantly different. The groups must be disjoint. There are significant differences among the three groups in the example below.

TBL =  rbind(c(559, 220, 380), c(93520, 44150, 78954));  TBL
      [,1]  [,2]  [,3]
[1,]   559   220   380   # Rotten
[2,] 93520 44150 78954   # OK
chisq.test(TBL)

        Pearson's Chi-squared test

data:  TBL
X-squared = 12.18, df = 2, p-value = 0.002265

Note: Percentages without the relevant sample sizes are not useful.