# Frequencies/percentages different from zero - which test?

I conducted an experiment and had participants indicate binary judgments by either pressing "1" or "0" when seeing different types of stimuli. Now I want to test whether the proportions of number presses differ from chance likelihood of 50% for the different stimulus conditions. My supervisor always used to run a one-sample t-test against 0.5 on the subject-level aggregated data (with the resulting mean representing the percentage of 1-presses); however, I am wondering this is the best way to do it. Since I am dealing with a binary variable (1/0), wouldn't it be more appropriate to run a chi-square goodness of fit test on the unaggregated data? Any input is highly appreciated!

[Update] This is a fictive example for my data structure:

Subj    type    response
1   target_top      1
1   target_bottom   0
1   target_bottom   0
1   target_top      1
1   target_bottom   0
1   target_top      1
2   target_bottom   0
2   target_top      0
2   target_top      0
2   target_bottom   1
2   target_bottom   1
2   target_top      0


As can be seen, it is a repeated measures design. Stimuli differ in their vertical position and we test the hypothesis that spatial position influences judgments (i.e., for example, 1-presses when presented on top position significantly differ from chance likelihood).

• Could you perhaps add a (fictive) small data sample for 1-2 participants so that we can grasp the data structure? What research question(s) are you trying to answer by the hypotheses tests? – Michael M Oct 24 '18 at 18:34
• Your update is about a completely different question from your title. What exactly do your want to do? Perhaps you need to edit the title? – mdewey Oct 25 '18 at 11:21
• Thank you, I have changed the title now and please excuse the previous misunderstanding! I am interested in testing whether the observed frequencies of pressing the respective buttons (whether in relative or absolute terms) differs significantly from chance. – Lafayote Oct 25 '18 at 12:54

The presumptive appropriate approach would be mixed effects logistic regression, where the dependent variable is Response, the independent variable is Type, and Subject is treated as the random subject. This is the approach described in the answer by @Noah.

Using a simple chi-square goodness-of-fit test or binomial test ignores the fact that the same subject is responding several times.

A t-test is probably not a good solution considering that the dependent variable is binomial and not continuous, as well as the fact that a t-test ignores the repeated responses by each subject.

My inclination is to use multilevel logistic regression. Your outcome is whether the observation was 0 or 1. Your level-1 predictor is the target location. Your clustering variable is participant ID. Alternatively, logistic regression with cluster-robust standard errors or fixed effects for participant ID would work as well.

Your output would be a coefficient on target location, which indicates the difference in the log odds of selecting 1 vs. 0 between the target locations, and a constant, which would represent the probability of selecting 1 in the baseline target location. With these, you could transform the log odds ratio into a difference in probabilities, but the significance test on the coefficient would be the test of whether target location affects the probability of selecting 0 or 1.

If I am understanding you correctly, you wish to see if the proportion of 1's in your data is significantly different from .5. You can calculate the sample proportion, say, $$\hat{p}=\frac{\# \ of \ 1}{N}$$. You can then calculate a $$z$$-statistic: $$z=\frac{\hat{p}-p_0}{\frac{p_0(1-p_0)}{N}}$$ where $$p_0=.5$$ and $$N$$ is your total sample size. Then you can calculate a $$p$$ value given the $$z$$ statistics and conclude statistical significance or lack thereof.

This can be done for each type of stimuli. If the stimulus are independent of each other and the order is randomized, a Bonferroni correction can be applied: you should compare your $$p$$ value to $$\frac{\alpha}{ \# \ of \ stimulus}$$, e.g., you have 4 different types and your overall significance level is .05, then you should compare your $$p$$ values to .0125.

• This doesn't actually answer the question, which is whether a chi-square test would be better. The OP obviously knows about the test you are proposing as he refers to it in the question. – jbowman Oct 24 '18 at 18:58
• Oh sorry. I was not sure if the OP mentioned this. I saw that the OP mentioned an one sample t test, which is slightly different because the denominator here can be known given the $p_0$. Then the OP mentioned a chi-square test, which I am not sure how it would apply since it seemed to be an one-sample problem comparing sample proportions to .5. So I added this. But my bad if this is what OP is already doing. – QmmmmLiu Oct 24 '18 at 19:04
• @jbowman, it's not clear OP knows about this test. The t-test they propose does not use binary variables rather a proportion correct for each individual as the dependent variable. A binomial test would therefore not make sense for them to contemplate. – Noah Oct 24 '18 at 21:37

The probably most obvious choice to test, whether an event ("1") happend by chance $$p$$ or not would be an binomial test. In R this can be done via the binom.test function.

So if 30 participants voted "1" and 35 voted "0" the test of votings happening bei 50:50 chance ist

> binom.test(c(30, 35), p=.05)

Exact binomial test

data:  c(30, 35)
number of successes = 30, number of trials = 65, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.05
95 percent confidence interval:
0.3370209 0.5896749
sample estimates:
probability of success
0.4615385


This is an exact test that does not rely on approximations as opposed to $$\chi^2$$ tests, which seem still more reasonable then a t-test.

• Shouldn't the fact that these are repeated measures be given any consideration? Contemplating that suggests (to me at least) that we need to clarify the specific statistical question that is being asked. – whuber Oct 24 '18 at 19:31
• @whuber The way I understood the question was, that there was a number of conditions that yield 0 or 1 from each participant and that a dedicated test for each condition should be performed. Are not in each of these tests the results indipendend, because they are singular? But yes, maybe the question was not as clear as it appeared to me. – Bernhard Oct 25 '18 at 11:06
• Thank you for your reply! I have to apologize because I did not mention important aspects of the design (repeated measures) in my first post. However, I still learned from you answer! – Lafayote Oct 25 '18 at 12:57