0
$\begingroup$

I have 2 variables in my data. One variable is continuous and it is the dependent variable (grammaticality ratings). The second variable is categorical and has 6 levels (6 grammatical structures). I asked a group of participants to rate each structure.

I started data analysis and I noticed that the mean rating for each categorical level is different. I want to test the significance of this difference in means. I want to compare the means of the 6 levels with the continuous variable.

Should I separate the categorical variable into 6 different categorical variables and then do a test?

Is the most appropriate test for this data set One-Way ANOVA?

$\endgroup$

1 Answer 1

0
$\begingroup$

Testing to see if differences are "significant" is an idea that the field of statistics needs to put behind it. It is not meaningful and uses an arbitrary significance threshold (often the highly arbitrary 0.05). It would be beneficial to state you real underlying goals. Also make sure your experimental design is in line with your goals. Then assess whether you have prior knowledge that the data come from a nearly Gaussian (normal) distribution, and whether the 6 groups are sharply defined and don't represent some underlying continuum.

Depending on all of that you may want to consider a Kruskal-Wallis test, which is the extension of the Wilcoxon-Mann-Whitney 2-sample rank-sum test for comparing $k$ groups. It tests a more general hypothesis than the corresponding parametric test (ANOVA $F$-test). The general hypothesis is related to a tendency of measurements in at least one of the groups to be higher than measurements in at least one of the other groups. Use the continuous $p$-value from the test as a measure of surprise, i.e., how surprising are the data were the measurements to all come from the same population. This can be interpreted as a indirect evidence against the null hypothesis of 6 equal distributions. Don't apply a cutoff to it.

In many cases a statistical test is not really what is consistent with the overall goals and instead you have an estimation problem. This can be dealt with using uncertainty intervals (e.g., compatibility intervals, AKA confidence intervals).

$\endgroup$
1
  • $\begingroup$ Thanks a lot, Frank. Thanks for reminding me that we should put behind us significance testing. I will make sure to use confidence intervals to estimate the true population parameter. $\endgroup$
    – Alaa
    Commented Jan 31, 2022 at 4:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.