# one sampled t-Test to compare against chance level performance

For an experiment, I want to test if people at each condition have performed significantly above chance level on average, in which they answer 3 questions, each with two options. I will compare each group to chance level performance, without comparing each group to each other.

Then, should I do a one-sampled t-Test by doing these steps, for each single group?

1. Turn each individual's performance to a proportion (0, 0,3333333, 0,66666666, 1)
2. Turn it into a data frame.
3. run it in R with the code t.test(my_dataframe, mu = 0.5, alternative = "two.sided")

I can imagine how trivial the question looks, but it's the first time I'm doing it, so I wanted to make sure. Thank you!

New contributor
breathe is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
• Do you mean $H_0: \mu_1=\mu_2=\mu_3 = 0.5$ vs $H_a: H_0 \text{ is false}$? – Dave Jun 8 at 17:36
• The second one! I don't compare the groups. I just want to compare each group to chance level perfromance, without comparing the groups to each other. – breathe Jun 8 at 17:54
• That is one test. You do not choose between testing $H_0$ and $H_a$. Those are the null and alternative hypotheses. Do you not want to test that $\mu_1=0.5$, $\mu_2=0.5$, and $\mu_3=0.5$? – Dave Jun 8 at 17:58
• Yes, that's what I want to do exactly! – breathe Jun 8 at 18:32
• How many subjects in the various groups? – BruceET Jun 8 at 20:20

## 1 Answer

Some ideas: Looking at one group only.

Suppose you have 50 subjects in a group. Then each subject will give you a binomial score $$X_i \sim \mathsf{Binom}(3, p_i).$$ If all subjects had the same probability of answering correctly $$(p_i \equiv p),$$ then you could say that the total score for a group is $$T \sim \mathsf{Binom}(150, p)$$ and you could do a binomial test of $$H_O: p = 0.5$$ against $$H_a: p >0.5.$$

However, there is no reason to suppose all 50 subjects are equally capable. so a t test of $$H_0: \mu=1.5$$ against $$H_a: \mu > 1.5$$ for $$n = 50$$ individual scores should give reasonably accurate results.

Example: In R, suppose I have $$n = 50$$ (fictitious) scores in vector x as summarized and plotted below.

summary(x); sd(x)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.00    1.00    2.00    1.98    3.00    3.00
 0.9145089
hist(x, prob=T, br=seq(-.5,3.5), col="skyblue2") Although the sample is not normally distributed, a t test on $$n = 50$$ such values without outliers should be reasonably accurate. The one-sided test shows a highly significant result with P-value near $$0.$$

t.test(x, mu=1.5, alt="g")

One Sample t-test

data:  x
t = 3.7114, df = 49, p-value = 0.0002634
alternative hypothesis: true mean is greater than 1.5
95 percent confidence interval:
1.76317     Inf

sample estimates:
mean of x
1.98


If one has qualms about using a t test, a less-powerful one-sided sign test based on 35 of 50 observations in x above $$1.5$$ rejects that the population median score is $$1.5$$ with P-value 0.0033.

y = sum(x > 1.5); y
 35
sum(dbinom(y:50, 50, .5))
 0.003300224


Also, a simple quantile 95% nonparametric bootstrap CI for $$p$$ is $$(1.72, 2.22),$$ which does not contain $$1.5.$$

set.seed(1234)
a.re = replicate(5000, mean(sample(x,50,rep=T)))
quantile(a.re, c(.025,.975))
2.5% 97.5%
1.72  2.22


Note: The fictitious data in x were generated as follows in R. The mean of $$p \sim \mathsf{Beta}(8,5)$$ is $$\mu = 8/13 = 0.6154 > 0.5.$$

set.seed(2021)
x = rbinom(50, 3, rbeta(50,8,5))

• What is the theory behind the mean of $p$ following a beta distribution? – Galen Jun 8 at 22:59
• @Galen: $p$ can take different values in $(0,1)$ for different subjects. Beta distributions have support $(0,1).$ BETA(9,5) averages $p \approx 0.61.$ It's not the only way to model fictitious data that might match this experiment, but it is a reasonable one. // My fictitious $X$ will tend to have a larger variance than a binomial dist'n with the same mean. // In Bayesian statistics beta distributions are often used as prior distributions on binomial success probability. – BruceET Jun 9 at 1:56