I am an old dog (DB guy) trying to learn new tricks (stats) and was hoping someone here could tell me if this is a good approach:

I have to analyse extremely right skewed counts of events over a period of observation, pre vs post. I have N=1213 pre and N=1138 post intervention datasets. Minitab rates Skewness at 5.3 and 7.3 for the two datasets. I would like to figure out the %change due to our intervention.

I proposed to do: transform non-zero event counts using log10 to make it less skewed and calculate the median change Mann-Whitney between pre and post. In addition, i calculate a probability of zero value events. In Minitab the Levene test for 2 sets of log10 values pre vs post gave P=0.220, so i assume that the variances are similar. With Mann-Whitney η1 = η2 vs η1 > η2 i had 0.4301 difference with P=0. So calculating after reversing the log gave me a 5.85 % improvement after intervention. The log curve wasn't considered normal(p<0.005).

But a teammate said i should be doing a bootstrap mean over the entire dataset including the zero event counts. So I did the Bootstrapping for mean in R, and this gave me over 50% improvement. This does not feel right.

What is the correct representation of the difference?

  • $\begingroup$ What do you mean by P = 0.220, and P = 0 ? what is " counts of events". please indicate type of data and an illustration of data. Why do you undertake log of data ? $\endgroup$ – Subhash C. Davar Aug 2 '18 at 13:25

If you have count data and are interested in the percentage change due to the intervention, you might want to run a negative binomial regression with the counts as the outcome variable and the intervention condition as the predictor. Taking $exp(\beta_1)$, where $\beta_1$ is the coefficient on the intervention variable, will give you the percentage change in your mean outcome corresponding to being in the treatment category vs. the control category.

Negative binomial regression is designed to model count data with right skew. If you have many zeros, you can try a zero-inflated negative binomial regression.The benefit of negative binomial regression is that it is designed to work with the data you have and produces a result on the scale you want (percentage change).

Another thing to note is that a standard t-test on the raw values would not necessarily be invalid; with such a large sample, the test statistic should be approximately t-distributed.

  • $\begingroup$ Using binomial regression, i got an improvement percentage of 53.8%. If i use a standard t-test on the bootstrapped mean, i got an improvement percentage also 53.5%. The graph of the bootstrapped mean really does follow normal distribution. I will go with this value. $\endgroup$ – Pho Aug 3 '18 at 9:22

I think the taking the log and then comparing medians is odd, and might sound like shenanigans to your audience.

You can use Mann-Whitney on the original data (with or without the zeros, depending on what makes sense). Just understand that it isn't a test of medians. It is a test of stochastic equality. That is, Ho is that the probablity of an observation in one group being higher than an observation in another group is 0.50.

This is usually a more interesting hypothesis than a test of medians, but it depends on your application.

If you really want to test medians, use Mood's median test or a permutation test for medians.

The percent change you could do a few different ways.

What percent change is actually meaningful in your case? Is it the mean? The median? Other percentiles?

Percentile  Value for Group 1  Value for Group 2  Percent difference
10th        whatever           whatever           + 10%
25th        whatever           whatever           +  5%
median      whatever           whatever              0%
75th        whatever           whatever           - 17%
90th        whatever           whatever           - 25%

It may help also to simply present the histograms.

Finally, if you use the Mann-Whitney test, one effect size statistic is Vargha and Delaney's A. It reports the probability that an observation in one group is greater than the other group. So that, VDA = 0.50 means the groups are stochastically equal; VDA = 1 means that all observations in one group are greater than those in the other.

  • $\begingroup$ As a layman, I had thought that in general for a skewed dataset, the median should be more representative than the mean. I wanted to figure out the probability of getting a lower median observation after intervention. That's why i tried Mann-Whitney. I also thought that Mann-Whitney needed similar variances, so i log10 the values to do the test. Am i wrong in this understanding? This seems to imply my change had a much less impact. $\endgroup$ – Pho Aug 3 '18 at 9:59
  • $\begingroup$ @Pho , I think you are correct in your thinking, except that M-W doesn't really require equal variances. It does if you want to use it as a test of medians. But, as I mention, I think the hypothesis about stochastic equality that it is really testing is usually more interesting. If you want to test for differences in the median, use a test for the median, like Mood's median test or a permutation test for the median. If you want to report the difference in medians as a number or as a percent, just report it as such. I wouldn't try to manipulate the data so much to answer a simple question. $\endgroup$ – Sal Mangiafico Aug 7 '18 at 17:51
  • $\begingroup$ Thank you for your clarification. The probability from M-W is interesting after i read up about this. I have included it in my report to the team on our improvements. $\endgroup$ – Pho Aug 8 '18 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.