1
$\begingroup$

I have two groups, A & B. Let's say A are the users who wrote a review for place A, and B for place B.

For each user $u$ I have recorded his total number of reviews $r_u$. Since we're talking about number of reviews, which is a very skewed variable, I thought it would be better to use the natural logarithm $\log(r_u) = \hat{r_u}$.

Now, I suspect that the users who write for A are "fake" or at least "green" accounts with 1 or very few reviews, and I want to declare whether these users are statistically significant from those who write for B. From a visual inspection, it seems clearn that something is off (black trace = A; green trace = B).

enter image description here

Still, I would like to formalize this difference. I need to compare $R_A = \{\hat{r_u} \ \forall \ u \in A \}$ and $R_B = \{\hat{r_u} \ \forall \ u \in B \}$ somehow, but I am unsure on how to do it properly from a statistical point of view.

Some doubts and questions I have in mind (I am fully aware that it's not one specific question, I'm sorry about that):

  • is it appropriate to test for difference of means? I mean, should I really compare means to show what I want to show? What about a given percentile? Or maybe compare proportions of 1-review users in the two groups? What would you go for? Would you go for more than one?
  • if I compare means I'll go for an independent group t-test but I think it is clear that in my case the assumption of normality doesn't hold. Would you go with a non-parametric, e.g. the Mann-Whitney test?
  • what's a good sample size? and how much do samples' sizes matter? For example, now I have gathered $|R_A| \approx 400$ and $|R_B| \approx 700$ and I am not really sure what to do with that (because I could even gather more for both). Should I sample $\approx$ 400 records from $R_B$ to level out the groups?

Thank you for your help.

Improve this question
$\endgroup$
4
  • $\begingroup$ 1) What do you want to know about the groups? If you’re interested in testing for different means, a test of means is appropriate. If you’re interested in something else, such a test is less appropriate. 2) Let’s first address what you even want to learn from your data. Then we can discuss specific approaches. 3) It depends on what you want to do. However, i cannot think of any circumstances where discarding data to balance the sizes will help you. $\endgroup$
    – Dave
    Nov 23, 2022 at 12:43
  • $\begingroup$ @Dave Not knowing how to go about 1) and 2) that's exactly why I ask the opinion of some experts here. I present my (confuse) ideas hoping someone can guide me. $\endgroup$
    – rusiano
    Nov 23, 2022 at 13:15
  • $\begingroup$ What do you want to learn from your data? $\endgroup$
    – Dave
    Nov 23, 2022 at 13:23
  • $\begingroup$ @Dave I think it is stated: whether these two groups are different, specifically if there are significantly more 1-review users in group A than in group B. The problem, as you say, is that there is no unambiguous definition of "different": I could look at mean (which is the most "popular" choice), but why not at median, at another percentile or maybe at proportions? Or why not at multiple metrics? $\endgroup$
    – rusiano
    Nov 23, 2022 at 13:28

1 Answer 1

Reset to default
2
$\begingroup$

  • What you want to test depends on your substantive question. Based on your explanation, it looks to me like a test for whether the proportions of 1-review users differs between the groups might be helpful. You can simply set up a 2-by-2 contingency table and use Pearson's $\chi^2$ test. Alternatively, you could test "low numbers of reviews", e.g., 1-2 vs. 3 or more reviews.

  • If you want to test differences in means, the t-test can be used even if the distributions are heavily non-normal as long as the sample size is "reasonably large". Your 400-700 is definitely fine.

  • More data always gives us more information. Just be aware that tiny observed differences will be statistically significant if your sample size is large. I would say that you can probably be reasonably confident in your results based on your current sample size. Just be sure to also report the effect size ("50% of the reviewers for A had only a single review to their name, against 10% of the reviewers for B, and the difference is statistically significant, $p=\dots$").

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you so much for sharing your knowledge, I really appreciate it. I am wondering, could it be that the 3rd link about tiny observed differences is not the one intended? I am redirected to the questions tagged with [hypothesis-testing]. $\endgroup$
    – rusiano
    Nov 23, 2022 at 13:31
  • $\begingroup$ That link is as intended. Notice the questions are sorted by votes. We get lots of questions "I have a large sample and a tiny p value, what's wrong?", and the top hits in that search look at this (and other) aspect from different angles. $\endgroup$
    – Stephan Kolassa
    Nov 23, 2022 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.