Impossible effects in a logistic regression. What causes them, and what to do about them

Question

I am currently analysing some data from a psychological experiment. In this experiment participants have to decide between two options based on some information. I can derive a variable indicating which option participants "should" choose, and I want to know how well this predictor fits with the actual behavior.

However, when I calculate a logistic regression in R for the probability that the participants choose a particular option, I do not only find an influence of the criterion, but also repeatedly a quite significant intercept. Now this intercept is absolutely impossible, because during presentation the two options are fully randomized (unless I have clairvoyant participants, an explanation I however refuse to believe). I would dismiss this significant intercept as a typical type I error, however I repeatedly find this effect in multiple experiments.

One of the possible explanations I had, was that the predictor was somewhat biassed to one of the options. Currently, the predictor is coded so that -1 fully indicates one option, 0 is no indication of either option, and +1 indicates the other option. With a t-test I could find, that within the design the mean of this criterion is significantly different from 0.

If I try to reproduce this problem with some skewed predictor values, however, I fail to reproduce this problem:

# For reproducibility 
set.seed(19857)

# Number of participants
npart <- 40

# Number of times each PO1 is sampled
nreps <- 10

intercept     <-  0
beta.main1    <-  8.83

# Criterion Value in the range [0,+1].
# Will later be recoded to [-1,+1]
BPO1 <- c(3:5/10,11:20/20)

df <- expand.grid(VpNr=1:npart, BayesPO1=BPO1, rep=1:nreps)

# Criterion recoded to [-1,+1]
df$SupportO1 <- (df$BayesPO1 - 0.5) * 2

type1Error <- 0

for(i in 1:100) {
  df$logit <- intercept + beta.main1 * df$SupportO1
  df$ModelPO1 <- 1/(1+exp(-df$logit))
  df$Option1Chosen <- ifelse(runif(nrow(df)) < df$ModelPO1,1,0)

  mdl <- glm(Option1Chosen ~ SupportO1, data=df, family=binomial(link="logit"))
  if(coef(summary(mdl))["(Intercept)","Pr(>|z|)"] <= 0.05) {
    type1Error <- type1Error + 1
  }
}

With this I get about 3% type 1 Errors, which is well within the range of what is to be expected.

In the actual experiment there are some other factors which may influence the participant behavior (all randomized across options). I tried adding additional noise and random slopes to the simulated data as well, with which I could drive the type 1 error ratio up to a maximum of 30%. However, this depends a lot on the actual additional effects I add to the model, some of which are again impossible (such as some types of interactions with the predictor).

In the current data (experiment has been repeated multiple times), the impossible intercept appears to be very robust. Even if I control for between subject factor using a mixed model, I always get this intercept.

So I think some model assumptions are definitely not met, however, I fail to figure out, what exact assumptions are causing this strange behavior (especially since I fail to replicate the effect in a simulation).

What kind of additional checks could I perform to figure out what is causing this effect and what could I do to still get valid information from the analysis. Or also, which possible real effect is overfit here by the model, that may actually be interesting?

Update:

This question is not a duplicate of this one, since that question is asking about the interpretation of the intercept. I know that the intercept is describing the marginal distribution. However, since the experiment is randomized, this marginal distribution should vanish when predictor is added. It is simply impossible for the participants to prefer one option over the other, except for factors which can be fully explained by the (biased) predictor.

So the main question is, since my analysis is obviously giving me impossible results, what else do I have to watch out for in terms of other possible incorrect results, and what can I do about this problem to remove any incorrect results (i.e. better models or additional cleaning).

Update:

What is biased in the experiment is the predictor, which could in principle also be calculated by the participants. The complete possible range of the value (before recoding) is 0 to 1. Because this value is symmetric, i.e. values higher than 0.5 indicate one option and values lower than 0.5 indicate the other option, only a limited and biased range was used (about 0.3 to 0.8). This of course also translated to a bias in the predictor used in the logistic regression, i.e. after recoding the values come from a range of about -0.4 to 0.6 with a mean significantly different from 0 (0.5 before recoding). This bias somehow seems to produce errors during the logistic regression, leading to impossible results. However, I fail to be able to identify the conditions under which the logistic regression is producing impossible results.

Answer 1

I believe I have found the problem. Because the predictor is a summary based on a psychological model, it does not fully capture the behavior in the participants. Because the data is skewed and only randomized during presentation, some additional behavior causes an actual tendency of the participants to choose one option more frequently, and this tendency is not captured by the model. Since there is nothing in the logistic regression which could capture this additional behavior besides the intercept and because this behavior is quite stable, the logistic regression "incorrectly" uses the intercept to fit this additional behavior (which it can only do, because of the bias in the data).

I can by now reproduce some similar tendencies in the data. The current model and simulation only uses a so called "bayesian decision strategy". If I add additional participants, who are using take-the-best (TTB) instead, then I get a similar skew and a reproducible significant intercept (which really does not mean much, since the "correct" model is missing). I am not yet sure if this is really caused by TTB participants. Non-linear processing of the provided information may also add additional biases, which could not be captured by the bayesian strategy I am testing in the regression.

So there is an additional pattern in the data, and because of problems with the experimental design, this pattern is biased. The lesson I would take from this, is that it is not enough to just randomize displays, but the actual data used for presentation should also be symmetric, to keep additional patterns from appearing as intercepts.

I am still not sure, though, what would be the best method to actually analyze the data. One approach I am currently using, is to "flip" half of the data randomly. I.e. I just switch Option1 and Option2 in all regards. This will keep the actual pattern, but now this pattern is not biased anymore, so it cannot produce a faulty intercept. However, since there still are additional patterns in the data not captured by the model I am using, this can still cause my parameter estimates to be off, if the factors I am actually testing correlate with the additional pattern by accident.