0
$\begingroup$

the basic setup is as follows: I have a continuous dependent variable (DV, 7 observations) and two continuous independent variables (IV1 & IV2). I would like to evaluate whether adding IV2 as covariate provides added value compared to a regression model with IV1 alone. Normally, one could simply use the Anova function in R to compare both models.

However, the problem is that the observations are not completely independent, and that full correction for all random factors (cf. mixed models approach) is impossible due to the limited setup. Nevertheless, the dependence between the observations is completely captured by IV1, not IV2. E.g. whereas IV1 & DV are heavily correlated (R²~99%), IV2 is not correlated with either IV1 nor DV. FYI: Adding IV2 to the model explains about 95% of the remaining variance, and is significant (p<0.001) when added as independent covariate in the model. Moreover, it results in a reduced RSS for almost all observations.

My question is whether (and why) the p-value for IV2 can (not) be trusted.

Thanks!

$\endgroup$
1
$\begingroup$

With only 7 observations you may well be faced with overfitting of your 2-predictor model. A rule of thumb for avoiding overfitting in linear regression is about 15 cases per predictor being considered (e.g.,section 4.4 of Harrell's course notes), so you are only half-way there even when considering just IV1. So the initial reaction would be not to trust anything having to do with IV2.

If the underlying relationships are strong (as they seem to be in your case) you can get away with fewer cases. One way to proceed could be to perform your regression model on multiple (several hundred) bootstrapped samples of your data, and for each model from a bootstrap sample collect the regression coefficients and the performance on predicting the original 7 observations. The distribution of coefficients for IV2 among those models might be a better measure of the trustworthiness of the IV2 coefficient than the nominal p-value in your initial model, but I would be very wary about trying to generalize any model based on only 7 observations.

$\endgroup$
  • $\begingroup$ Thanks EdM, I'm aware of the limitations, but I'll definitely explore the bootstrap option! $\endgroup$ – Mr.T Jan 1 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.