I need your help on how best to test the following hypotheses in R. Each observation in my data has RaterID and RaterGender, along with their ratings on 5-point scale for four different attributes (P1, P2, P3, P4) of product (ProductID).There are multiple observations for each rater and same product may have ratings from multiple observers.

How do I statistically model in R to test the following hypothesis:

H1: Men are more likely have higher ratings on P1 than P2.

H2: Women are more likely to have higher ratings on P4 than P3.

How should I control for the random effects of raters and products?


closed as unclear what you're asking by StatsStudent, Michael Chernick, Peter Flom Feb 17 at 13:37

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ But you haven't told us anything about how the data were collected. $\endgroup$ – StatsStudent Feb 16 at 19:17
  • $\begingroup$ Data was collected through a survey following product purchases by the individuals. Hope this clarifies. $\endgroup$ – Bensun Feb 16 at 19:20
  • $\begingroup$ Survey data needs to be carefully handled. You cannot just build traditional mixed effects models or carry out traditional hypothesis tests. How does each observation have multiple ratings? You need to tell us more about the data collection process and I'd exercise caution using any of the answers provided thus far since this information is absent in your question and this can change the entire analysis. $\endgroup$ – StatsStudent Feb 16 at 19:53
  • 1
    $\begingroup$ What is the population of interest? You should update the question with these additional details and then I'll delete some of the comments with my questions. $\endgroup$ – StatsStudent Feb 16 at 20:04
  • 1
    $\begingroup$ So, unless that website was randomly selected from all websites, then it will be impossible for you to infer anything about all consumers who buy organic products. Research shows that online buyers are markedly different than those who purchase items in retail, brick-and-mortar stores. I doubt consumers have similar behavior from web-site to web-site at that. $\endgroup$ – StatsStudent Feb 16 at 20:14

Based on your description, you could start from a model that contains as fixed effects the sex and the attribute, where sex is a factor with levels male and female, and attribute a factor with levels P1-P4, and random effects for ProductID and RaterID, e.g., something like

y ~ sex * attribute + (1 | ProductID) + (1 | RaterID)

By including the interaction, you could test your specific hypothesis of interest by using suitable contrasts.

An additional important point is what type of mixed model you will use. If I understood correctly, you have an ordinal outcome (i.e., 5-point scale). There are not that many options with regard to software for proportional odds models. However, you could use the continuation ratio model instead that requires a couple of extra data management steps, and fitting the model with a mixed effects logistic regression.

  • 1
    $\begingroup$ (+1) perhaps a cumulative link mixed model with clmm2 from the ordinal package would be suitable - although I am not sure if it handles crossed random effects. $\endgroup$ – Robert Long Feb 16 at 19:46
  • 1
    $\begingroup$ How does this take into account finite population corrections or the survey design? $\endgroup$ – StatsStudent Feb 16 at 19:53
  • 1
    $\begingroup$ @RobertLong I also don’t know about the ordinal package, but I’ve been using the continuation ratio model with satisfactory results. In this model it is straightforward to check the ordinality assumption for specific covariates by simply including an interaction term. $\endgroup$ – Dimitris Rizopoulos Feb 16 at 19:54

You should probably be using the Mann-Whitney-Wilcoxon Test. In R if your data is in data frame X with X\$gender being the gender data and X\$difference being the difference between the rating of $P_i$ versus $P_j$ then your code would be

wilcox.test(difference~gender, data=X)

the null would be that there is no gender-based difference.

You would control for product effects by solving it for each product. However, as you are doing multiple comparisons, you will need to add the FSA library to get the Holm-Bonferroni correction.

Run each test and capture the p-values into some variable such as calling it pvalues. Then you would run

final_values$Holm = p.adjust(pvalues, method = "holm")

Unless you know what the random effects are, you cannot control for them. There isn't a simple answer as to adding a variable for nonparametric anova that I have found. The Friedman test may be appropriate, but you would have to research it appropriateness for your data.

  • 1
    $\begingroup$ what do you mean "You would control for product effects by solving it for each product" ? If you mean splitting the dataset by product then this would lead to an enormous loss of power. Also, the data are clustered by rater, not just product, so a mixed effects model with crossed random effects suggested by @DimitrisRizopoulos might be more appropriate.... $\endgroup$ – Robert Long Feb 16 at 19:49
  • $\begingroup$ I too had the same concern $\endgroup$ – Bensun Feb 16 at 20:02
  • $\begingroup$ Yes, I do need to rewrite that or at least qualify it. $\endgroup$ – Dave Harris Feb 16 at 20:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.