1
$\begingroup$

The data are from book ratings done by users. Users were split into three experimental groups (A/B - test). Goal is to understand if users show any difference in rating books between these groups. Carrying out t-test sounds like the first approach to understand the differences.

Since there is slight imbalance in the user populations i.e: Experiment Group1 has 100 users, Group2 has 110 users, Group3 has 105 Two Sample t-test seems sensible.

  1. Executing two sample t-test between Group1 Vs Group2, Group1 Vs Group3, and Group2 Vs Group3 with-out calculating average book rating per user.

    Here taking each rating done by each user and running t-test. Exp:

    import scipy.stats as stats
    group1Rating = [2.5, 2, 4.5, 4, 4.5, 3.5.....4] #multiple rating per user
    group2Rating = [3, 2, 5, 4, 4, 4.....4.5] #multiple rating per user
    grp1-grp2 = stats.ttest_ind(a=group1Rating, b=group2Rating, equal_var=False)
    
    ...
    

    Here I see p-values 3.24 * 10^-12

    similarly between rest of the group combinations.

  2. Executing two sample t-test between Group1 Vs Group2, Group1 Vs Group3, and Group2 Vs Group3 with calculating average book rating per user (Each user has a single record which is average rating).

Here I see no significance between the p-values Example: Ttest_indResult(statistic=0.984, pvalue=0.324)

Questions:

  • Which one is the correct approach? If none of the above whats the correct t-test for above scenario.
  • Is there a different/alternate test to understand the rating differences between groups.
$\endgroup$
1
$\begingroup$

Executing two sample t-test

This is a general practice and I don't see red flags until your data are very not symmetric.

Executing two sample t-test between Group1 Vs Group2, Group1 Vs Group3, and Group2 Vs Group3 with calculating average book rating per user (Each user has a single record which is average rating).

One does not apply t-test function (I don't specify whether it's R or Py) to samples with only one observation which is the mean value of a sampled variable. Instead one have to follow that part of t-test methodology that starts after calculating the mean and standard deviation values form their samples.

This is basically just that: find a pooled standard deviation, calculate the t-statistic using mean, pooled s.d. and degrees of freedom, and the desired effect size (>= 0), and find the p-value using a table, or a special function.

$\endgroup$
2
  • $\begingroup$ Yes, thats what i thought so what i gather from your answer is that Approach one is correct? how would you interpret two results if approach two is not an incorrect thing to do? python method is actually calculating means, std etc that are necessary to carry out t-test. $\endgroup$ Dec 6 '17 at 18:18
  • $\begingroup$ @Null-Hypothesis, I work with R and I cannot advise on the exact t-distribution function in Python. You can do the job by using the first approach, it's most straightforward way. Otherwise for exact math look here: en.wikipedia.org/wiki/… and here: statisticshowto.com/tables/t-distribution-table $\endgroup$ Dec 6 '17 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.