# GLM binomial regression in python shows significance for any random vector

I have a dependent variable as a binomial count, and I have used the GLM model as suggested in this post. My model shows up highly significant, which I found very suspicious. To check, I ran the model on a vector with randomly generated numbers.

I get the following output: If I'm interpreting it correctly, it's still highly significant, with the associated p-value of the chi-sq statistic <0.01.

Any ideas on what might be happening?

• You have over 300,000 observations. Anything that is not exactly zero will likely be statistically significant. – Heteroskedastic Jim Jul 24 '18 at 22:24
• @user162986 While it's true that smaller and smaller effects become significant with large sample sizes, for randomly generated data that should not be the case. – Bryan Krause Jul 24 '18 at 22:32
• I'm not familiar with GLM in python - is your model lacking an intercept? – Bryan Krause Jul 24 '18 at 22:35
• @BryanKrause in a single run, one can't say. We can only say what we'd expect over many runs. I now realize I didn't read the question properly and I should have. OP can gain learn more about the performance of stats models glm by running a simulation. I'll try one and post an answer. – Heteroskedastic Jim Jul 24 '18 at 22:37
• I suppose I presumed that the OP ran more than one iteration with the synthetic data, since doing so would be so easy. – Bryan Krause Jul 24 '18 at 22:38

I attempted to test OP's claim.

import matplotlib.pyplot as plt
import numpy as np
import statsmodels.api as sm

n = 300000
res = []
for _ in xrange(1000):
y = np.random.randint(0, 2, n) # Bernoulli with p = 50%
# x \sim U(0, 300000 - 1)
x = np.random.randint(0, n, n)
# Standardize strange x to help with numerical issues
x = (x - np.mean(x)) / np.std(x)
res.append(sm.GLM(y, x, family=sm.families.Binomial()).fit().pvalues)


A couple of comments. The data generation process for the regressor in OP's example is strange but correct. In my simulation as in OP's, x is a uniformly distributed with minimum 0 and maximum 299,999. I standardized the regressor to reduce any chances of software problems with model fitting. I used 1000 replications. Sample size is 300,000.

Here is the result:

plt.hist(res)
plt.show() The p-values are uniformly distributed between 0 and 1 as one would expect. statsmodels behaves as expected. I tried the variation in OP's examples keeping y the same and changing x. Same outcome.

• sorry, I am confused - what is this saying? that my random generator is correct or..? – vvv Jul 25 '18 at 20:24
• @vvv Your random number generation though correct is strange. You are generating a variable that can take on any value between 0 and ~370,000. Usually, we do not use such variables in models without transforming them somehow. The major point here is that statsmodel GLM behaves as one might expect. Most of the p-values at large sample sizes are greater than .05 (~95% of the p-values), if you repeat the process enough times. – Heteroskedastic Jim Jul 25 '18 at 21:35