Small number of positives in a large dataset I have a panel dataset with a very large number of observations 300,000. I am testing to see if a dummy variable is positive and significant using regular OLS. I have only about 1500 obs where the dummy variable is 1 - in one version of this and about 800 where the dummy is 1 in another version of this.
The coefficient on the dummy ends up being positive and statistically significant in all of my tests -  the model with the smaller set of dummies=1 has a higher p-value than the large. This is true when controlling for two sets of group variables (year and firm - panel dataset i.year i.firm and even i.year#i.firm). This doesn't make sense theoretically for these variables (I am sorry I can't share details due to confidentiality agreements).
So I tested to see if this would be true if I created a placebo group - i.e. instead of the obs being 1 for the 1500 obs where my dummy variable is truly 1, selected 1500 other observations (not truly random but close) that  also came out positive and stat significant. I tried this exercise with a number of placebo selections and in every case this is true. The overlap between the placebo and the true group is about about 7% in many of these.
Is there a reason why a small number of cases in a very large dataset would result in statistically significant results regardless of true relationship? I know the SEs get very small in large datasets but is that the only reason?
What solutions are there to this problem if one exists?
Thanks,
Greg
 A: The standard error for a regression coefficient divides by feature variance, and the variance of a Bernoulli variable is maximized when the probability parameter if $0.5$ (perfect balance) and decreases as the parameter tends toward $0$ or $1$. In the case of a considerable imbalance like you have, that probability is close to $0$, resulting in a small variance. Since the standard error equation divides by this small variance, the standard error is relatively large. The equation also divides by the variance-inflation factor, though it is not obvious why the imbalance should have any impact on this. (In particular, if this imbalanced feature is totally independent of the other features, the VIF is zero.)
However, since you have $300,000$ observations this means that, as you've mentioned,  your test has considerable power to detect small differences (small standard errors), perhaps even differences that are of no practical importance.
Sure, theory says that the coefficient is zero. When you look at the estimated coefficient and its confidence interval, I speculate that one of two of the following will be true:

*

*The interval estimate only contains values that are trivially different from zero and not of any practical importance, meaning that the theory is basically true (more or less).


*Babies don't wear headphones as much as adults (that is, the coefficient will have practical importance, but this is a signal that something is wrong about your assumptions).
