For an experiment, I want to test if people at each condition have performed significantly above chance level on average, in which they answer 3 questions, each with two options. I will compare each group to chance level performance, without comparing each group to each other.

Then, should I do a one-sampled t-Test by doing these steps, for each single group?

  1. Turn each individual's performance to a proportion (0, 0,3333333, 0,66666666, 1)
  2. Turn it into a data frame.
  3. run it in R with the code t.test(my_dataframe, mu = 0.5, alternative = "two.sided")

I can imagine how trivial the question looks, but it's the first time I'm doing it, so I wanted to make sure. Thank you!

New contributor
breathe is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • $\begingroup$ Do you mean $H_0: \mu_1=\mu_2=\mu_3 = 0.5$ vs $H_a: H_0 \text{ is false}$? $\endgroup$ – Dave Jun 8 at 17:36
  • $\begingroup$ The second one! I don't compare the groups. I just want to compare each group to chance level perfromance, without comparing the groups to each other. $\endgroup$ – breathe Jun 8 at 17:54
  • $\begingroup$ That is one test. You do not choose between testing $H_0$ and $H_a$. Those are the null and alternative hypotheses. Do you not want to test that $\mu_1=0.5$, $\mu_2=0.5$, and $\mu_3=0.5$? $\endgroup$ – Dave Jun 8 at 17:58
  • $\begingroup$ Yes, that's what I want to do exactly! $\endgroup$ – breathe Jun 8 at 18:32
  • $\begingroup$ How many subjects in the various groups? $\endgroup$ – BruceET Jun 8 at 20:20

Some ideas: Looking at one group only.

Suppose you have 50 subjects in a group. Then each subject will give you a binomial score $X_i \sim \mathsf{Binom}(3, p_i).$ If all subjects had the same probability of answering correctly $(p_i \equiv p),$ then you could say that the total score for a group is $T \sim \mathsf{Binom}(150, p)$ and you could do a binomial test of $H_O: p = 0.5$ against $H_a: p >0.5.$

However, there is no reason to suppose all 50 subjects are equally capable. so a t test of $H_0: \mu=1.5$ against $H_a: \mu > 1.5$ for $n = 50$ individual scores should give reasonably accurate results.

Example: In R, suppose I have $n = 50$ (fictitious) scores in vector x as summarized and plotted below.

summary(x); sd(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.00    1.00    2.00    1.98    3.00    3.00 
[1] 0.9145089
hist(x, prob=T, br=seq(-.5,3.5), col="skyblue2")

enter image description here

Although the sample is not normally distributed, a t test on $n = 50$ such values without outliers should be reasonably accurate. The one-sided test shows a highly significant result with P-value near $0.$

t.test(x, mu=1.5, alt="g")

        One Sample t-test

data:  x
t = 3.7114, df = 49, p-value = 0.0002634
alternative hypothesis: true mean is greater than 1.5
95 percent confidence interval:
 1.76317     Inf

sample estimates:
mean of x 

If one has qualms about using a t test, a less-powerful one-sided sign test based on 35 of 50 observations in x above $1.5$ rejects that the population median score is $1.5$ with P-value 0.0033.

y = sum(x > 1.5); y
[1] 35
sum(dbinom(y:50, 50, .5))
[1] 0.003300224

Also, a simple quantile 95% nonparametric bootstrap CI for $p$ is $(1.72, 2.22),$ which does not contain $1.5.$

a.re = replicate(5000, mean(sample(x,50,rep=T)))
quantile(a.re, c(.025,.975))
 2.5% 97.5% 
 1.72  2.22 

Note: The fictitious data in x were generated as follows in R. The mean of $p \sim \mathsf{Beta}(8,5)$ is $\mu = 8/13 = 0.6154 > 0.5.$

x = rbinom(50, 3, rbeta(50,8,5))
  • $\begingroup$ What is the theory behind the mean of $p$ following a beta distribution? $\endgroup$ – Galen Jun 8 at 22:59
  • $\begingroup$ @Galen: $p$ can take different values in $(0,1)$ for different subjects. Beta distributions have support $(0,1).$ BETA(9,5) averages $p \approx 0.61.$ It's not the only way to model fictitious data that might match this experiment, but it is a reasonable one. // My fictitious $X$ will tend to have a larger variance than a binomial dist'n with the same mean. // In Bayesian statistics beta distributions are often used as prior distributions on binomial success probability. $\endgroup$ – BruceET Jun 9 at 1:56

Your Answer

breathe is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.