Skip to main content
edited body; edited title
Source Link
Arya McCarthy
  • 9k
  • 1
  • 24
  • 58

Interpret Mann-WhitneyMann–Whitney U Test with highly skewed data

I have data for ~102,000 hands of bananas. The dataset has four data points:

  • Spiders present (True/False)
  • Count of rotten bananas in hand
  • Count of total bananas in hand
  • Percent of bananas that are rotten in each hand (rotten/total)

Overall, spiders were present in 6,600 (6.5%) of banana hands. In ~102,000 hands of bananas, there are 557,000 bananas, of which 32,700 (5.9%) were rotten. In the hands where spiders were present, the rotten rate was 9.9%, and in the no spiders banana hands, the rate was 5.2%. From here, it

Hypothesis: Spiders prefer banana hands with higher rates of rotten bananas (though they do not exclusively inhabit rotten banana hands).

I would like to statistically test and prove this. Spiders-present and no-spiders present have very different numbers of groups. The data is skewed data. The vast majority of banana hands do not have any rotten bananas. Of the 6,600 spider-present banana hands, 56% do not have any rotten bananas (3,696 hands) and in the no-spider-present banana hands, 86.8% (82,807 hands) have no rotten bananas. There are lots of zeroes.

In this scenario, this is not a sample of the total world's bananas but the full population of bananas.

I calculated the following frequency table for spiders present and the rotten rate (pandas/python):

OBS BANANAS ROTTEN SPIDERS MIN 25P MEDIAN 75P MAX MEAN STD
6,600 36,041 3,568 TRUE 0 0 0 0.13333 1.0 0.139540699 0.086308
95,400 520,959 27,090 FALSE 0 0 0 0.0 1.0 0.05970877 0.180351

Given the high number of zeros, and the very skewed data, I was looking for nonparametric statistical tests to explain the difference in the rotten rate between the two categories (present, not present) of banana hands.

I found the Mann WhitneyMann–Whitney U Testtest. Running the test, I find I have incredibly small p-value (1.25e104, yes 104). The returned statistic value is astronomical compared to the descriptions, 224031919.

I believe this is the appropriate test and is valid based on these four assumptions of my data because:

  1. My dependent variable, rate of rotten bananas, is ordinal/continuous
  2. My independent variable is categorical (spiders present, spiders not present)
  3. There is independence of observations. Each banana occurs only once in each hand.
  4. My data is not normally distributed

My questions to you:

  • How can I interpret these results?
  • Should I be wrangling my data to make it more manageable?
  • Is there a different test that you'd suggest? If yes, what is it, and why?
  • Is there something else I should be considering? If yes, what is it and why?

Interpret Mann-Whitney U Test with highly skewed data

I have data for ~102,000 hands of bananas. The dataset has four data points:

  • Spiders present (True/False)
  • Count of rotten bananas in hand
  • Count of total bananas in hand
  • Percent of bananas that are rotten in each hand (rotten/total)

Overall, spiders were present in 6,600 (6.5%) of banana hands. In ~102,000 hands of bananas, there are 557,000 bananas, of which 32,700 (5.9%) were rotten. In the hands where spiders were present, the rotten rate was 9.9%, and in the no spiders banana hands, the rate was 5.2%. From here, it

Hypothesis: Spiders prefer banana hands with higher rates of rotten bananas (though they do not exclusively inhabit rotten banana hands).

I would like to statistically test and prove this. Spiders-present and no-spiders present have very different numbers of groups. The data is skewed data. The vast majority of banana hands do not have any rotten bananas. Of the 6,600 spider-present banana hands, 56% do not have any rotten bananas (3,696 hands) and in the no-spider-present banana hands, 86.8% (82,807 hands) have no rotten bananas. There are lots of zeroes.

In this scenario, this is not a sample of the total world's bananas but the full population of bananas.

I calculated the following frequency table for spiders present and the rotten rate (pandas/python):

OBS BANANAS ROTTEN SPIDERS MIN 25P MEDIAN 75P MAX MEAN STD
6,600 36,041 3,568 TRUE 0 0 0 0.13333 1.0 0.139540699 0.086308
95,400 520,959 27,090 FALSE 0 0 0 0.0 1.0 0.05970877 0.180351

Given the high number of zeros, and the very skewed data, I was looking for nonparametric statistical tests to explain the difference in the rotten rate between the two categories (present, not present) of banana hands.

I found the Mann Whitney U Test. Running the test, I find I have incredibly small p-value (1.25e104, yes 104). The returned statistic value is astronomical compared to the descriptions, 224031919.

I believe this is the appropriate test and is valid based on these four assumptions of my data because:

  1. My dependent variable, rate of rotten bananas, is ordinal/continuous
  2. My independent variable is categorical (spiders present, spiders not present)
  3. There is independence of observations. Each banana occurs only once in each hand.
  4. My data is not normally distributed

My questions to you:

  • How can I interpret these results?
  • Should I be wrangling my data to make it more manageable?
  • Is there a different test that you'd suggest? If yes, what is it, and why?
  • Is there something else I should be considering? If yes, what is it and why?

Interpret Mann–Whitney U Test with highly skewed data

I have data for ~102,000 hands of bananas. The dataset has four data points:

  • Spiders present (True/False)
  • Count of rotten bananas in hand
  • Count of total bananas in hand
  • Percent of bananas that are rotten in each hand (rotten/total)

Overall, spiders were present in 6,600 (6.5%) of banana hands. In ~102,000 hands of bananas, there are 557,000 bananas, of which 32,700 (5.9%) were rotten. In the hands where spiders were present, the rotten rate was 9.9%, and in the no spiders banana hands, the rate was 5.2%. From here, it

Hypothesis: Spiders prefer banana hands with higher rates of rotten bananas (though they do not exclusively inhabit rotten banana hands).

I would like to statistically test and prove this. Spiders-present and no-spiders present have very different numbers of groups. The data is skewed data. The vast majority of banana hands do not have any rotten bananas. Of the 6,600 spider-present banana hands, 56% do not have any rotten bananas (3,696 hands) and in the no-spider-present banana hands, 86.8% (82,807 hands) have no rotten bananas. There are lots of zeroes.

In this scenario, this is not a sample of the total world's bananas but the full population of bananas.

I calculated the following frequency table for spiders present and the rotten rate (pandas/python):

OBS BANANAS ROTTEN SPIDERS MIN 25P MEDIAN 75P MAX MEAN STD
6,600 36,041 3,568 TRUE 0 0 0 0.13333 1.0 0.139540699 0.086308
95,400 520,959 27,090 FALSE 0 0 0 0.0 1.0 0.05970877 0.180351

Given the high number of zeros, and the very skewed data, I was looking for nonparametric statistical tests to explain the difference in the rotten rate between the two categories (present, not present) of banana hands.

I found the Mann–Whitney U test. Running the test, I find I have incredibly small p-value (1.25e104, yes 104). The returned statistic value is astronomical compared to the descriptions, 224031919.

I believe this is the appropriate test and is valid based on these four assumptions of my data because:

  1. My dependent variable, rate of rotten bananas, is ordinal/continuous
  2. My independent variable is categorical (spiders present, spiders not present)
  3. There is independence of observations. Each banana occurs only once in each hand.
  4. My data is not normally distributed

My questions to you:

  • How can I interpret these results?
  • Should I be wrangling my data to make it more manageable?
  • Is there a different test that you'd suggest? If yes, what is it, and why?
  • Is there something else I should be considering? If yes, what is it and why?
Source Link
adin
  • 235
  • 1
  • 8

Interpret Mann-Whitney U Test with highly skewed data

I have data for ~102,000 hands of bananas. The dataset has four data points:

  • Spiders present (True/False)
  • Count of rotten bananas in hand
  • Count of total bananas in hand
  • Percent of bananas that are rotten in each hand (rotten/total)

Overall, spiders were present in 6,600 (6.5%) of banana hands. In ~102,000 hands of bananas, there are 557,000 bananas, of which 32,700 (5.9%) were rotten. In the hands where spiders were present, the rotten rate was 9.9%, and in the no spiders banana hands, the rate was 5.2%. From here, it

Hypothesis: Spiders prefer banana hands with higher rates of rotten bananas (though they do not exclusively inhabit rotten banana hands).

I would like to statistically test and prove this. Spiders-present and no-spiders present have very different numbers of groups. The data is skewed data. The vast majority of banana hands do not have any rotten bananas. Of the 6,600 spider-present banana hands, 56% do not have any rotten bananas (3,696 hands) and in the no-spider-present banana hands, 86.8% (82,807 hands) have no rotten bananas. There are lots of zeroes.

In this scenario, this is not a sample of the total world's bananas but the full population of bananas.

I calculated the following frequency table for spiders present and the rotten rate (pandas/python):

OBS BANANAS ROTTEN SPIDERS MIN 25P MEDIAN 75P MAX MEAN STD
6,600 36,041 3,568 TRUE 0 0 0 0.13333 1.0 0.139540699 0.086308
95,400 520,959 27,090 FALSE 0 0 0 0.0 1.0 0.05970877 0.180351

Given the high number of zeros, and the very skewed data, I was looking for nonparametric statistical tests to explain the difference in the rotten rate between the two categories (present, not present) of banana hands.

I found the Mann Whitney U Test. Running the test, I find I have incredibly small p-value (1.25e104, yes 104). The returned statistic value is astronomical compared to the descriptions, 224031919.

I believe this is the appropriate test and is valid based on these four assumptions of my data because:

  1. My dependent variable, rate of rotten bananas, is ordinal/continuous
  2. My independent variable is categorical (spiders present, spiders not present)
  3. There is independence of observations. Each banana occurs only once in each hand.
  4. My data is not normally distributed

My questions to you:

  • How can I interpret these results?
  • Should I be wrangling my data to make it more manageable?
  • Is there a different test that you'd suggest? If yes, what is it, and why?
  • Is there something else I should be considering? If yes, what is it and why?