This is the data set
How can I determine if this 4 groups differed significantly in their performance
5
-
$\begingroup$ Do you know the letters to which your numbers correspond, like 84 was in group A, 90 was in group C, etc? $\endgroup$– DaveCommented Dec 23, 2021 at 1:11
-
$\begingroup$ I edited it, take a look again. $\endgroup$– Steven AlexCommented Dec 23, 2021 at 1:16
-
$\begingroup$ What are the trials? Is this just four measurements in each of four groups, that’s different than if you track four subjects over time, for instance. $\endgroup$– DaveCommented Dec 23, 2021 at 1:27
-
$\begingroup$ Sorry, but I don't understand $\endgroup$– Steven AlexCommented Dec 23, 2021 at 2:09
-
$\begingroup$ What is the “Trial” column? $\endgroup$– DaveCommented Dec 23, 2021 at 3:57
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
How can I determine if this 4 groups differed significantly in their performance
Based on the stripplot of your data below, I had two thoughts:
- Formal statistical inference aside, these groups don't look substantially different. But I also don't know the measure scale we are dealing with here, so it is difficult for me to judge.
- Your sample size is small. Not so small that you cannot do the calculations for a hypothesis test, but small enough that I have doubts about stability of inferences subject to sampling error. Even so, consider reading through this post for a discussion of the smallest sample size one might do with an ANOVA.
But for what it is worth, an ANOVA of your data had an $F=1.176$ and $p=0.3510$. Under the usual confidence limit of $\alpha=0.05$, one would consider there to not be a significant difference among your groups.
-
1$\begingroup$ Kruskal-Wallis is a highly-similar approach, but would have even less statistical power. $\endgroup$– GalenCommented Dec 23, 2021 at 6:38
-
$\begingroup$ If their data are repeated measures design (not saying they are, they did not describe with much detail), they may have enough power to detect a difference in treatment means just going visually by the distribution of green dots. $\endgroup$– AlexisCommented Dec 23, 2021 at 8:20
$\begingroup$
$\endgroup$
2
Do.
Not.
Use.
Unless you want:
- $p$ values biased to be small
- $\beta$ estimates biased away from zero
- equal likelihood of including false predictors as true predictors
- equal likelihood of excluding false predictors as true predictors
- $F$ statistics biased upward
- $R^2$ biased upward
- a series of nested "condition on the decisions about models using such-and-such variables" which are virtually never presented to the audience
-
$\begingroup$ How is stepwise variable selection related to this situation of a single (categorial) predictor? $\endgroup$– cdalitzCommented Dec 23, 2021 at 6:06
-
2$\begingroup$ @cdalitz Perhaps you missed the OP's title? And the only tag they selected for their question? $\endgroup$– AlexisCommented Dec 23, 2021 at 8:16
$\begingroup$
$\endgroup$
If I interpret your table correctly, you have one predictor (inpendent variable) which is the category (or group) A-D, and one response (dependent variable) that is the score value in the table and is measured on different independent trials.
In this situation, there are two standard statistical tests that try to answer different questions:
- ANOVA (see this answer), which tests whether there is any difference in the mean values between all pairs of groups.
- Two sample Kolmogorov-Smirnov test (KS), which tests whether the distribution of the response is different between two groups.
ANOVA automatically takes care of the multiple testing problem, while KS only compares a single pair.
