I am currently analysing some data from a psychological experiment. In this experiment participants have to decide between two options based on some information. I can derive a variable indicating which option participants "should" choose, and I want to know how well this predictor fits with the actual behavior.
However, when I calculate a logistic regression in R for the probability that the participants choose a particular option, I do not only find an influence of the criterion, but also repeatedly a quite significant intercept. Now this intercept is absolutely impossible, because during presentation the two options are fully randomized (unless I have clairvoyant participants, an explanation I however refuse to believe). I would dismiss this significant intercept as a typical type I error, however I repeatedly find this effect in multiple experiments.
One of the possible explanations I had, was that the predictor was somewhat biassed to one of the options. Currently, the predictor is coded so that -1 fully indicates one option, 0 is no indication of either option, and +1 indicates the other option. With a t-test I could find, that within the design the mean of this criterion is significantly different from 0.
If I try to reproduce this problem with some skewed predictor values, however, I fail to reproduce this problem:
# For reproducibility
set.seed(19857)
# Number of participants
npart <- 40
# Number of times each PO1 is sampled
nreps <- 10
intercept <- 0
beta.main1 <- 8.83
# Criterion Value in the range [0,+1].
# Will later be recoded to [-1,+1]
BPO1 <- c(3:5/10,11:20/20)
df <- expand.grid(VpNr=1:npart, BayesPO1=BPO1, rep=1:nreps)
# Criterion recoded to [-1,+1]
df$SupportO1 <- (df$BayesPO1 - 0.5) * 2
type1Error <- 0
for(i in 1:100) {
df$logit <- intercept + beta.main1 * df$SupportO1
df$ModelPO1 <- 1/(1+exp(-df$logit))
df$Option1Chosen <- ifelse(runif(nrow(df)) < df$ModelPO1,1,0)
mdl <- glm(Option1Chosen ~ SupportO1, data=df, family=binomial(link="logit"))
if(coef(summary(mdl))["(Intercept)","Pr(>|z|)"] <= 0.05) {
type1Error <- type1Error + 1
}
}
With this I get about 3% type 1 Errors, which is well within the range of what is to be expected.
In the actual experiment there are some other factors which may influence the participant behavior (all randomized across options). I tried adding additional noise and random slopes to the simulated data as well, with which I could drive the type 1 error ratio up to a maximum of 30%. However, this depends a lot on the actual additional effects I add to the model, some of which are again impossible (such as some types of interactions with the predictor).
In the current data (experiment has been repeated multiple times), the impossible intercept appears to be very robust. Even if I control for between subject factor using a mixed model, I always get this intercept.
So I think some model assumptions are definitely not met, however, I fail to figure out, what exact assumptions are causing this strange behavior (especially since I fail to replicate the effect in a simulation).
What kind of additional checks could I perform to figure out what is causing this effect and what could I do to still get valid information from the analysis. Or also, which possible real effect is overfit here by the model, that may actually be interesting?
Update:
This question is not a duplicate of this one, since that question is asking about the interpretation of the intercept. I know that the intercept is describing the marginal distribution. However, since the experiment is randomized, this marginal distribution should vanish when predictor is added. It is simply impossible for the participants to prefer one option over the other, except for factors which can be fully explained by the (biased) predictor.
So the main question is, since my analysis is obviously giving me impossible results, what else do I have to watch out for in terms of other possible incorrect results, and what can I do about this problem to remove any incorrect results (i.e. better models or additional cleaning).
Update:
What is biased in the experiment is the predictor, which could in principle also be calculated by the participants. The complete possible range of the value (before recoding) is 0 to 1. Because this value is symmetric, i.e. values higher than 0.5 indicate one option and values lower than 0.5 indicate the other option, only a limited and biased range was used (about 0.3 to 0.8). This of course also translated to a bias in the predictor used in the logistic regression, i.e. after recoding the values come from a range of about -0.4 to 0.6 with a mean significantly different from 0 (0.5 before recoding). This bias somehow seems to produce errors during the logistic regression, leading to impossible results. However, I fail to be able to identify the conditions under which the logistic regression is producing impossible results.
10$
, or (b)10000$
, imagine I presented such question randomly to different people, obviously the distribution of the answers would be skewed no matter what kind of randomization you'll use. Maybe simply your two options aren't equivalently preferred? $\endgroup$