Related question: Hypothesis testing: Why is a null model that fits the data well better than one that doesn't?

I simulated a response, y, that is influenced by a covariate but not by the "treatment" (the thing I care about) as follows:

n <- 1e3
covar_effect <- 1
trt_effect <- 0

covar <- rnorm(n = n)
trt <- rnorm(n = n)
y <- rnorm(n = n, mean = covar*covar_effect + trt*trt_effect)

I computed the likelihood ratio statistic (LRS) between a null model that omits trt and an alternative model that includes trt in two ways.

  1. both the null and alternative model omit covar
  2. both the null and alternative model include covar

In both cases, the distribution of the LRS was $\chi^2_1$, as you can see in this figure: null_scenario Here's the gist showing how I ran this simulation in R.

Then, I repeated the process, but simulated a situation where there actually is a treatment effect:

n <- 1e3
covar_effect <- 1
trt_effect <- 0.1

covar <- rnorm(n = n)
trt <- rnorm(n = n)
y <- rnorm(n = n, mean = covar*covar_effect + trt*trt_effect)

Consistent with intuition, the LRS was greater between the null and alternative that include the covariate than between the null and alternative that omit it: alt_scenario

If I were thinking about things from a parameter estimation perspective and using a Wald test to test whether the effect of trt is zero, I could readily understand why including the covariate in the null and alternative models would increase my power to reject the null -- the standard error of the effect estimate is $\frac{\hat{\sigma}}{V(x)}$, so anything that decreases $\sigma$ will increase the precision of the estimate.

But I am not thinking about things from the perspective of a Wald test. I am thinking about a likelihood ratio. There are some likelihood relationships that are obvious to me (I hope this notation is clear):

LRSs will be positive:

  • L(y|trt) > L(y)
  • L(y|trt, covar) > L(y|covar)

These comparisons aren't directly relevant but, the models are nested so the inequality seems obvious:

  • L(y|covar) > L(y)
  • L(y|trt, covar) > L(y|trt)

But the increased LRS with the covariate modeled doesn't follow directly from any of that. It relates to this inequality:

(L(y|trt,covar) - L(y|covar)) > (L(y|trt) - L(y))

which I can rearrange into:

L(y|trt, covar) + L(y) > L(y|covar) + L(y|trt)

Now why would that be true? Can anyone offer a mathematical or conceptual explanation of why it's beneficial to model a covariate in both the null and alternative models?

  • $\begingroup$ Can you clarify your notation for the likelihood expressions? You're considering L(y|trt,covar), L(y|trt), L(y|covar), and L(y) right? Not the marginal likelihoods for the exposures? Also what is "alt"? $\endgroup$ – AdamO Feb 1 '18 at 14:58
  • $\begingroup$ @AdamO, where I wrote "alt", I meant "trt". I've edited my questiont o fix that typo. I've also updated by notation to follow yours, which is better. $\endgroup$ – rcorty Feb 1 '18 at 20:20
  • 1
    $\begingroup$ Do you need a proof of this, or just to understand the issue? $\endgroup$ – gung - Reinstate Monica Feb 1 '18 at 20:24
  • $\begingroup$ I think understanding would be a reasonable first step. I could try to prove it myself if I understood! $\endgroup$ – rcorty Feb 1 '18 at 22:39

The reason that the inference adjusting for the extraneous variable is more powerful is that that variable is considered a precision variable. A precision variable is unassociated with the predictor of interest, but is strongly predictive of the outcome. Controlling for precision variables reduces the residual standard error. Even in your simulation, the argument to sd in the rnorm function which generates y is taken to be its default, which is 1, and that' conditional upon the precision variable. If you omit that variable from the model, the residual standard error of y is in fact:

$$\text{var}(Y-\beta_0 - \beta_1 X) = \text{var}(\epsilon) + \beta_2^2\text{var}(P)$$

Where $Y$ is the outcome of interest, $\beta_0 + \beta_1 X$ is the marginal model, $\beta_2$ is the effect for the precision variable $P$ and $\epsilon$ is the actual model error.

In your model, omitting the precision variable effectively doubled the residual variance. Reducing the variance by a factor of 2 at the sacrifice of 1 degree of freedom is a fair exchange in any sample size. A more thorough discussion of precision variables can be found in the Regression Methods in Biostatistics text by Vittinghoff et al.

  • $\begingroup$ This answer addresses the situation from a parameter estimation perspective. Why would a precision variable increase the increase in likelihood we observe when we add in the parameter of interest? $\endgroup$ – rcorty Feb 6 '18 at 21:50
  • 1
    $\begingroup$ @rcorty because the log-likelihood of the model is just the sum of the log normal density of the residuals. This is $- n \cdot\sqrt{2 \sigma}$. Decrease the residual variance, increase the log-likehood proportionately.. $\endgroup$ – AdamO Feb 6 '18 at 22:02
  • $\begingroup$ I agree that the log likelihood will go up when a covariant is included. But the LRS is a difference of LLs — why would the difference be greater when the covariant is included? $\endgroup$ – rcorty Feb 7 '18 at 1:58
  • $\begingroup$ I am awarding the bounty to this answer, though I do not feel it truly addressed the question. It was the only answer! $\endgroup$ – rcorty Feb 8 '18 at 16:04
  • $\begingroup$ @rcorty We established the log likelihood is proportional to sqrt(resid-SD). So the LRS is a difference of these. resid-SD null model: 1 + 1 + .01, resid SD trt only model: 1+ 1, resid SD covar only model: 1+.01 resid SD covar trt model: 1. sqrt(1+1+.01)-sqrt(1+1) = 0.004 but sqrt(1+.1)-sqrt(1) = 0.005. Does the algebra make sense? $\endgroup$ – AdamO Feb 8 '18 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.