0
$\begingroup$

For linear regression analysis, I thought that the addition of covariates which are not related to the dependent variable $Y$ does not decrease sensitivity. Such random/dummy regressors can be expected to capture some of the variance in the data. This is penalized with a decrease of the degrees of freedom (df2). I thought the p-value of the Welch test should be similar as if no dummy regressors were considered.

However, below you can find an example in $\texttt{R}$ which suggests I was wrong. If there is no regressor which is really related to the dependent variable $Y$, an increase of the number of dummy regressors does not affect the distribution of the p-values, which is $U[0,1]$. This is not surprising. On the other hand, if there is one regressor related to the dependent variable $Y$, an increase of the number of additional dummy regressors changes the distribution of the p-values. More precisely, it lowers sensitivity.

set.seed(1)
no_iter                 = 10000
F_p_val                 = rep(NA, no_iter)
F_p_val_01_dummies_only = rep(NA, no_iter)
F_p_val_12_dummies_only = rep(NA, no_iter)
F_p_val_with_01_dummies = rep(NA, no_iter)
F_p_val_with_12_dummies = rep(NA, no_iter)

for (iter in 1:no_iter) {

   X          = (1:100) / 100
   X          = X - mean(X)
   Y          = X + rnorm(100)
   X_dummy_1  = rnorm(100)
   X_dummy_2  = rnorm(100)
   X_dummy_3  = rnorm(100)
   X_dummy_4  = rnorm(100)
   X_dummy_5  = rnorm(100)
   X_dummy_6  = rnorm(100)
   X_dummy_7  = rnorm(100)
   X_dummy_8  = rnorm(100)
   X_dummy_9  = rnorm(100)
   X_dummy_10 = rnorm(100)
   X_dummy_11 = rnorm(100)
   X_dummy_12 = rnorm(100)

   #-2 models with no regressors related to Y;
   #-1st model with 1 dummy regressor,
   #-2nd model with 12 dummy regressors
   F          = summary(lm(Y ~ X_dummy_1))$fstatistic
   F_dummy    = summary(lm(Y ~ X_dummy_1 + X_dummy_2 + X_dummy_3 + X_dummy_4 +
		X_dummy_5 + X_dummy_6 + X_dummy_7 + X_dummy_8 +
		X_dummy_9 + X_dummy_10 + X_dummy_11 + X_dummy_12))$fstatistic

   #-calculating p-values
   F_p_val_01_dummies_only[iter] = 1 - pf(F[1],       F[2],       F[3])
   F_p_val_12_dummies_only[iter] = 1 - pf(F_dummy[1], F_dummy[2], F_dummy[3])

   #-2 models, each with 1 regressor related to Y;
   #-1st model additionally with 1 dummy regressor,
   #-2nd model additionally with 12 dummy regressors
   F          = summary(lm(Y ~ X + X_dummy_1))$fstatistic
   F_dummy    = summary(lm(Y ~ X + X_dummy_1 + X_dummy_2 + X_dummy_3 + X_dummy_4 +
		X_dummy_5 + X_dummy_6 + X_dummy_7 + X_dummy_8 +
		X_dummy_9 + X_dummy_10 + X_dummy_11 + X_dummy_12))$fstatistic

   #-calculating p-values
   F_p_val_with_01_dummies[iter] = 1 - pf(F[1],       F[2],       F[3])
   F_p_val_with_12_dummies[iter] = 1 - pf(F_dummy[1], F_dummy[2], F_dummy[3])

}

sum(F_p_val_01_dummies_only<0.05)/no_iter
[1] 0.0484

sum(F_p_val_12_dummies_only<0.05)/no_iter
[1] 0.0508

sum(F_p_val_with_01_dummies<0.05)/no_iter
[1] 0.7289

sum(F_p_val_with_12_dummies<0.05)/no_iter
[1] 0.3587

I would not be surprised to observe it if the data were not normally distributed or if there were some correlations between the regressors. This is not the case here.

I do not understand why the dummy regressors do not explain enough variance not to lower sensitivity when one regressor is indeed related to $Y$. Do you know how to explain that?

$\endgroup$
  • $\begingroup$ Are you stating that dummy regressors do not relate to Y variable? $\endgroup$ – Aksakal Jun 11 '18 at 13:22
  • $\begingroup$ edited to: "I thought that the addition of covariates which are not related to the dependent variable Y does not decrease sensitivity. SUCH random/dummy regressors..." $\endgroup$ – Wiktor Jun 11 '18 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.