I have data for ~102,000 hands of bananas. The dataset has four data points:

  • Spiders present (True/False)
  • Count of rotten bananas in hand
  • Count of total bananas in hand
  • Percent of bananas that are rotten in each hand (rotten/total)

Overall, spiders were present in 6,600 (6.5%) of banana hands. In ~102,000 hands of bananas, there are 557,000 bananas, of which 32,700 (5.9%) were rotten. In the hands where spiders were present, the rotten rate was 9.9%, and in the no spiders banana hands, the rate was 5.2%. From here, it

Hypothesis: Spiders prefer banana hands with higher rates of rotten bananas (though they do not exclusively inhabit rotten banana hands).

I would like to statistically test and prove this. Spiders-present and no-spiders present have very different numbers of groups. The data is skewed data. The vast majority of banana hands do not have any rotten bananas. Of the 6,600 spider-present banana hands, 56% do not have any rotten bananas (3,696 hands) and in the no-spider-present banana hands, 86.8% (82,807 hands) have no rotten bananas. There are lots of zeroes.

In this scenario, this is not a sample of the total world's bananas but the full population of bananas.

I calculated the following frequency table for spiders present and the rotten rate (pandas/python):

6,600 36,041 3,568 TRUE 0 0 0 0.13333 1.0 0.139540699 0.086308
95,400 520,959 27,090 FALSE 0 0 0 0.0 1.0 0.05970877 0.180351

Given the high number of zeros, and the very skewed data, I was looking for nonparametric statistical tests to explain the difference in the rotten rate between the two categories (present, not present) of banana hands.

I found the Mann–Whitney U test. Running the test, I find I have incredibly small p-value (1.25e104, yes 104). The returned statistic value is astronomical compared to the descriptions, 224031919.

I believe this is the appropriate test and is valid based on these four assumptions of my data because:

  1. My dependent variable, rate of rotten bananas, is ordinal/continuous
  2. My independent variable is categorical (spiders present, spiders not present)
  3. There is independence of observations. Each banana occurs only once in each hand.
  4. My data is not normally distributed

My questions to you:

  • How can I interpret these results?
  • Should I be wrangling my data to make it more manageable?
  • Is there a different test that you'd suggest? If yes, what is it, and why?
  • Is there something else I should be considering? If yes, what is it and why?
  • 1
    $\begingroup$ A good descriptive analysis will be more insightful than testing a (trivial?) hypothesis. So you could e.g. state the proportion of rotten bananas per spider status. $\endgroup$
    – Michael M
    Jul 1 at 17:14
  • 1
    $\begingroup$ @MichaelM are you referring to the overall rotten rate being 5.5%, where spiders present the rotten rate is 9.9%, and where spiders are not present the rotten rate is 5.2%? I've also posted the min, 25P, median, 75P, max, mean, STD (the .describe() function in pandas) for each spider status. Could you please be specific with the other descriptive statistics you're expecting? $\endgroup$
    – adin
    Jul 1 at 17:20
  • 3
    $\begingroup$ I was actually referring to your excellent description "In the hands where spiders were present, the rotten rate was 9.9%, and in the no spiders banana hands, the rate was 5.2%". The other statistics are useful as well. Since you have access to the whole population, there is no point in wasting much thought about tests as the null hypothesis is rejected in any case. $\endgroup$
    – Michael M
    Jul 1 at 17:40
  • $\begingroup$ @MichaelM Thank you for the clarification. What if the question changes to having a whole district's population and I want to describe the variances between districts. Say the whole, spiders-present, spiders-not-present distribution is 5.5,9.9/5.2 in District A but then it's 4.4, 8.2, 4.0 in District B. There are N districts. Could you recommend an alternative statistic to create a value that 1) tells me if there is significant difference between the groups, and 2) if so, what's the variance? $\endgroup$
    – adin
    Jul 1 at 17:48
  • 1
    $\begingroup$ When you test with $100,000$ observations, it is routine to get tiny p-values. $\endgroup$
    – Dave
    Jul 2 at 14:12

Testing might be appropriate, if you have samples and want to make inferences about populations from which the samples were randomly taken.

If you want to do a formal test of hypothesis to compare two percentages (say 5.5% with spiders and 9.9% without, or 5.5% in group A with 4.4% in group B), then you have to use numerator and denominator for each percentage.

Tests of two proportions. For example, if you had 10000 bananas of which 550 are rotten in A, and you had 5000 of which 220 are rotten in B, you could use prop.test in R to see if this difference is statistically significant at the 5% level, the answer is Yes because P-value $0.0045 < 0.05 = 5\%.$ however, if you're comparing $55/1000$ with $22/500,$ then No.

prop.test(c(550, 220), c(10000, 5000))$p.val
[1] 0.004530416

prop.test(c(55, 22), c(1000, 500))$p.val
[1] 0.4318846

Chi-squared tests. Somewhat similarly, one could use a chi-squared tests on contingency tables of counts of rotten and non-rotten bananas in several groups to see if the proportion of rotten bananas in the several groups are essentially the same or are significantly different. The groups must be disjoint. There are significant differences among the three groups in the example below.

TBL =  rbind(c(559, 220, 380), c(93520, 44150, 78954));  TBL
      [,1]  [,2]  [,3]
[1,]   559   220   380   # Rotten
[2,] 93520 44150 78954   # OK

        Pearson's Chi-squared test

data:  TBL
X-squared = 12.18, df = 2, p-value = 0.002265

Note: Percentages without the relevant sample sizes are not useful.

  • $\begingroup$ to clarify, I looked up proportion test in Python and it states Test for proportions based on normal (z) test. Normal here refers to a normal Z-Test or does it refer to requiring a normal distribution? statsmodels.org/stable/generated/… $\endgroup$
    – adin
    Jul 2 at 14:28
  • $\begingroup$ I'd say refers to both. $\endgroup$
    – BruceET
    Jul 2 at 16:12
  • $\begingroup$ is there an alternative for non-normally distributed data? $\endgroup$
    – adin
    Jul 2 at 18:31
  • $\begingroup$ Gemerally: Rank based tests such as Wilcoxon SR and Wilcoxon RS. Permutation tests. Also. specific tests for populations with specific known (but non-normal) distributionss. $\endgroup$
    – BruceET
    Jul 2 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.