I have an odd scenario in my data analysis and I'm not sure what is causing it. I have a large set of tuples $(Y_1, X_i) \dots, (Y_N, X_N)$ where $Y_i$ is a random vector from some arbitrary distribution (might not be normal) and $X_i$ is a vector denoting group membership such that $X_i \in [0,1,2]$. The length of the vectors are 200. I shuffle these vectors independently and observe uniform analytical OLS p-values for $\hat{\beta}$ as expected under the null. Then I run robust standard errors using this same shuffled dataset and observe extreme inflation in p-values. I am using the statsmodels package in Python to compute robust errors as follows

model_ols = sm.OLS(y, sm.add_constant(x)).fit(cov_type='HC3')

I'm not sure why this would be the case. Here are some observations about my data:

  1. The distribution of each vector $Y_i$ roughly follows a beta distribution. It can be Gaussian like at a mean of 0.5 and skewed at a mean of 0.9 or 0.1.
  2. The groups denote in $X_i$ can be heavily imbalanced.
  3. The variance per group is highly variable (i.e. heteroskedastic).
  4. The estimates for $\hat{\beta}$ are indeed identical for both runs. I know $\hat{\beta}$ is unbiased so this is expected.
  5. I've confirmed the standard errors reported by the statsmodels is identical (negligible numerical imprecision difference) to the jackknife standard error.

I suppose to simplify the question in broader terms: when would OLS control type I error under the null but robust errors result in poor type I error control on the same data?

  • $\begingroup$ Robust SEs will be larger than usual, that's what makes them robust. So, p values will be higher. Why it's extreme in your case, I don't know. $\endgroup$
    – Peter Flom
    Commented Mar 4 at 10:27
  • 1
    $\begingroup$ @PeterFlom, while empirically often the case, that is not necessarily so, see e.g. stats.stackexchange.com/questions/627057/… $\endgroup$ Commented Mar 4 at 15:49
  • $\begingroup$ @ChristophHanck Wow. Interesting. $\endgroup$
    – Peter Flom
    Commented Mar 4 at 15:59