0
$\begingroup$

I have a dependent variable as a binomial count, and I have used the GLM model as suggested in this post. My model shows up highly significant, which I found very suspicious. To check, I ran the model on a vector with randomly generated numbers.

I get the following output: GLM output If I'm interpreting it correctly, it's still highly significant, with the associated p-value of the chi-sq statistic <0.01.

Any ideas on what might be happening?

$\endgroup$
  • $\begingroup$ You have over 300,000 observations. Anything that is not exactly zero will likely be statistically significant. $\endgroup$ – Heteroskedastic Jim Jul 24 '18 at 22:24
  • 1
    $\begingroup$ @user162986 While it's true that smaller and smaller effects become significant with large sample sizes, for randomly generated data that should not be the case. $\endgroup$ – Bryan Krause Jul 24 '18 at 22:32
  • $\begingroup$ I'm not familiar with GLM in python - is your model lacking an intercept? $\endgroup$ – Bryan Krause Jul 24 '18 at 22:35
  • $\begingroup$ @BryanKrause in a single run, one can't say. We can only say what we'd expect over many runs. I now realize I didn't read the question properly and I should have. OP can gain learn more about the performance of stats models glm by running a simulation. I'll try one and post an answer. $\endgroup$ – Heteroskedastic Jim Jul 24 '18 at 22:37
  • $\begingroup$ I suppose I presumed that the OP ran more than one iteration with the synthetic data, since doing so would be so easy. $\endgroup$ – Bryan Krause Jul 24 '18 at 22:38
3
$\begingroup$

I attempted to test OP's claim.

import matplotlib.pyplot as plt
import numpy as np
import statsmodels.api as sm

n = 300000
res = []
for _ in xrange(1000):
    y = np.random.randint(0, 2, n) # Bernoulli with p = 50%
    # x \sim U(0, 300000 - 1)
    x = np.random.randint(0, n, n)
    # Standardize strange x to help with numerical issues
    x = (x - np.mean(x)) / np.std(x)
    res.append(sm.GLM(y, x, family=sm.families.Binomial()).fit().pvalues[0])

A couple of comments. The data generation process for the regressor in OP's example is strange but correct. In my simulation as in OP's, x is a uniformly distributed with minimum 0 and maximum 299,999. I standardized the regressor to reduce any chances of software problems with model fitting. I used 1000 replications. Sample size is 300,000.

Here is the result:

plt.hist(res)
plt.show()

enter image description here

The p-values are uniformly distributed between 0 and 1 as one would expect. statsmodels behaves as expected. I tried the variation in OP's examples keeping y the same and changing x. Same outcome.

$\endgroup$
  • $\begingroup$ sorry, I am confused - what is this saying? that my random generator is correct or..? $\endgroup$ – vvv Jul 25 '18 at 20:24
  • 1
    $\begingroup$ @vvv Your random number generation though correct is strange. You are generating a variable that can take on any value between 0 and ~370,000. Usually, we do not use such variables in models without transforming them somehow. The major point here is that statsmodel GLM behaves as one might expect. Most of the p-values at large sample sizes are greater than .05 (~95% of the p-values), if you repeat the process enough times. $\endgroup$ – Heteroskedastic Jim Jul 25 '18 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.