1
$\begingroup$

I am testing a linear model of the form $Y=X\beta+\epsilon$, where $X$ is an $n\times m$ matrix, with $x_{ij}=1$ if observation $i$ has some characteristic $j$, and $0$ if it doesn't (so I'm investigating the dependence of $Y$ on $m$ categorical predictors, and take $n$ observations). Here, $\beta\in\mathbb R^m$ is the (unknown) parameter vector, and $\epsilon\sim N(0,\sigma^2 I)$ is noise.

For each $1\leq j\leq m$, I test $H_0:\beta_j=0$ against $H_1:\beta_j\neq0$ (say using the F test or t test, doesn't make a difference), with significance level $\alpha$.

If I reject the null hypothesis for only one value $j$, does this mean there is evidence that characteristic $j$ is associated with $Y$? Does the answer change for small/large values of $m$?

$\endgroup$
0
$\begingroup$

In the procedure you describe, you are performing $m$ hypothesis tests, each with a probability $\alpha$ of yielding a false positive. If you have many predictors, these probabilities may accumulate, and the chances of seeing false positive results may become substantial. For example, suppose you are testing for 1000 independent parameters, the chance of seeing at least one false positive is $1-(1-\alpha)^m=1-0.95^{100}\approx0.994$ (way more than the stated $\alpha=0.05$).

To avoid this issue, you should apply multiple test corrections, such as:

  • Bonferroni correction: This approach simply changes the significance threshold from $\alpha$ to $\alpha/m$. It successfully reduces the chances of seeing false positives, but is sometimes considered overly conservative.
  • Benjamini-Hochberg correction: This is a slightly more complex procedure, described here. This method is quite popular in some fields and is less conservative than Bonferroni.

You can find many resources online discussing how and when to apply these correction procedures. They are very important step in the statistical analysis, specially when you are in a high dimensional setting (large $m$).

Hope it was helpful!

$\endgroup$
1
  • $\begingroup$ Hmm sorry if this wasn't clear, but the condition is that for exactly one $j$, we reject $H_0$. The probability of this happening is still small for large $m$. But thanks for the pointers! Looks interesting :) $\endgroup$
    – Verge
    Mar 22 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.