1
$\begingroup$

I'm doing a textmining project about a videogame review dataset. I divided de reviews into a positive (reviewgrade > 8) and negative (reviewgrade < 3) and want to draw conclusions based on the frequency of different grading factors mentioned in the text. I now have a dataframe with counts of certain categories mentioned per review.

0    13249
1     1565
2      292
3       65
4       19
5        6
6        4
7        3
8        2
9        1
Name: positive: gameplay, dtype: int64
0    5043
1     348
2      66
3      18
4       8
5       3
Name: negative: gameplay, dtype: int64

For example this are the distributions of the amount of reviews which have a certain amount of mentions of gameplay related words. There are for example 1565 reviews with precisely 1 mention of a word related to the category gameplay. Now I want to use this data to determine if on average gameplay gets mentioned more in negative context than in positive or vice versa. Because the data is not normal distributed, I assumed a t-test is not appropriate, and found online that a Mann–Whitney U test might be suitable.

> wilcox.test(positive$gameplay, negative$gameplay, alternative = "two.sided", conf.int = TRUE)

    Wilcoxon rank sum test with continuity correction

data:  positive$gameplay and negative$gameplay
W = 43702925, p-value < 2.2e-16
alternative hypothesis: true location shift is not equal to 0
95 percent confidence interval:
 2.963844e-05 4.797412e-05
sample estimates:
difference in location 
          4.995671e-05 

I do however get a very high W value, which doesn't seem to be correct. Is this not the appropriate test, or am I interpreting this wrong?

$\endgroup$
3
  • $\begingroup$ It is looking like you are using all 20,000 reviews to perform the Wilcoxon test, If you are using the count information, then maybe the Chisq or Fisher test is a better alternative since they could compensate for the mismatch on the number of positive vs negative reviews. $\endgroup$
    – Dave2e
    Nov 26, 2019 at 18:44
  • $\begingroup$ What are e.g. 0, 1, 2, 3 in your table? Are these the counts, of e.g. gameplay words? $\endgroup$ Nov 26, 2019 at 20:44
  • $\begingroup$ @SalMangiafico yes, gameplay words for example are 'level design' and 'combat system'. So for the positive reviews there are 13249 reviews that don't have any of those specific words in them, and 1565 that have exactly 1 etc. The positive dataframe has in total 15206 reviews, and the negative dataframe 5486 reviews $\endgroup$
    – blaurens
    Nov 26, 2019 at 20:49

1 Answer 1

2
$\begingroup$

It is reasonable to use the Wilcoxon-Mann-Whitney test as you have done. There are alternative models you could have used --- such as negative binomial regression for count data, or ordinal regression --- but the results would be similar.

This may be an example where the p value doesn't tell you what you need to know. Because your sample size is very large, the low p value indicates that one can tease out a "signal" against the "noise" in the data. Here, the W value is very large and this results in a very small p value. That's all good and reasonable.

But the next questions are, How large is the effect size? Is this a meaningful difference? I have some R code below to start to sketch out answers to these questions.

### Install packages, read data, prepare data

if(!require(lattice)){install.packages("lattice")}
if(!require(rcompanion)){install.packages("rcompanion")}

Data1 = read.table(header=T, text="
Group Mentions Count
A     0    13249
A     1     1565
A     2      292
A     3       65
A     4       19
A     5        6
A     6        4
A     7        3
A     8        2
A     9        1
")

Data2 = read.table(header=T, text="
Group Mentions Count
B     0    5043
B     1     348
B     2      66
B     3      18
B     4       8
B     5       3
")

Data = rbind(Data1, Data2)

Long = Data[rep(row.names(Data), Data$Count), c("Group", "Mentions")]

  rownames(Long) = seq(1:nrow(Long))

str(Long)

We could plot histogram-like bar plots to see the relative distributions of the mentions for each group. Due to the distribution of the data here, these plots may not be very helpful.

library(lattice)

Long$Mentions.f = factor(Long$Mentions)

histogram(~ Mentions.f | Group, data=Long, layout=c(1,2), col="darkgray")

histogram-like bar plots

It may be more useful to look at the proportions of each level of mentions for each group. Here, Group A has about 87% zero mentions and Group B has about 92% zero mentions. Is this an important difference? That's for you to say.

Table = xtabs(Count ~ Group + Mentions, data=Data)

round(prop.table(Table, margin=1),2)

   ###     Mentions
   ### Group    0    1    2    3    4    5    6    7    8    9
   ###     A 0.87 0.10 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00
   ###     B 0.92 0.06 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00

And my WMW test agrees with your results.

wilcox.test(Mentions ~ Group, data=Long)

   ### Wilcoxon rank sum test with continuity correction
   ### 
   ### W = 43702925, p-value < 2.2e-16

For a more formal effect size statistic, we might look at Cliff's delta. This value is related to the probability that an observation from one group would be larger than an observation from another group. Here, the value 0.05 is quite small, and the positive value indicates that the first group ("A") is slightly stochastically dominant.

library(rcompanion)

cliffDelta(Mentions ~ Group, data=Long)

   ### Cliff.delta 
   ###      0.0478

Vargha and Delaney's A gives us the same information in a different form. Here the answer is literally the probability that an observation from one group would be larger than an observation from another group. The answer is about 52%.

vda(Mentions ~ Group, data=Long)

   ###   VDA 
   ### 0.524

The upshot is that your use of the WMW test is reasonable. But the large sample size gives the test power to detect small stochastic differences between the two groups. It's important to also assess how large these differences are, and if these differences are practically meaningful for you.

$\endgroup$
7
  • $\begingroup$ First of all, awesome that you gave such an elaborate answer, much appreciated! I unfortunately am not skilled enough in statistics (yet) to deduce my next course of action from your explanation, to finally be able to have enough prove to finish my research paper. What should I "also assess", and which variables/tests are indeed suitable to use? I also read a research that claimed a normal distribution is not required to be able to do a t-test, as long as the sample is big enough. What is your opinion about this, a t-test might in that case be easier to just compare the means. $\endgroup$
    – blaurens
    Nov 26, 2019 at 22:38
  • $\begingroup$ @blaurens , the WMW test is a reasonable test for the situation as you've described it. There's probably no advantage to using a t-test in these circumstances. It is easier to discuss means, but with data this skewed, I'm not sure the mean is that meaningful. ... I might have some reservations about you dichotomizing the reviews according to reviewgrade. And the fact that you have multiple categories of words creates the potential to make things more complicated. But your approach is fine if you want to keep things simple. ... By "also assess" I just mean looking at, for example, $\endgroup$ Nov 26, 2019 at 23:27
  • $\begingroup$ the table of probabilities I provided. After the statistical test, you have to decide if the differences between the groups are meaningful. There's no test for that. In this case, the statistical test is very significant, the the difference between the groups is relatively small. It's up to you to decide if these differences are meaningful for what you want to know. $\endgroup$ Nov 26, 2019 at 23:28
  • $\begingroup$ Thank you, so comparing the 0,10 with 0,06 regarding mentioning gameplay at least once seems like quite a significant difference. My research actually contains 4 groups, positive and negative user reviews, and positive and negative critic reviews, which all have 8 variables (gameplay, graphics etc..). How do you suggest applying this on such a big scale with 4x8 values to compare? $\endgroup$
    – blaurens
    Nov 27, 2019 at 9:46
  • $\begingroup$ Do you want to compare the content of user reviews to critic reviews --- that is, Are they 4 of the same type of group? Or should they be considered separately? $\endgroup$ Nov 27, 2019 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.