Permutation test on the intercept in MANLY(1997) framework?

Question

We assume to have the following regression model: $Y=β_0+β_1X_1+β_2X_2+ϵ$

I recall here the Manly procedure (from another post here :How to do permutation test on model coefficients when including an interaction term?):

$1.$ The Variable $Y$ is regressed on $X_1$ and $X_2$ together (using least squares) to obtain an estimate $b_2$ of $β_2$ and a value of the usual t-statistic, tref for testing $β_2=0$ for the real data. We hereafter refer to this as the reference value of $t$.

$2.$ The $Y$ values are permuted randomly to obtain permuted values $Y^*$.

$3.$ The $Y^*$ values are regressed on $X_1$ and $X_2$ (unpermuted) together to obtain an estimate $b_2^*$ of $β_2^*$ and a value of $t^∗$ for the permuted data.

$4.$ Steps 2-3 are repeated a large number of times, yielding a distribution of values of $t^*$ under permutation.

$5.$ The absolute value of the reference value $t_{ref}$ is placed in the distribution of absolute values of $t^∗$ obtained under permutation (for a two-tailed t-test). The probability is calculated as the proportion of values in this distribution greater than or equal, in absolute value, to the absolute value of tref (Hope, 1968)

In my case i have a regression model code with dummy variable and an associated model design matrix i.e: $Y=β_0X_0+β_1X_1+β_2X_2+ϵ$ where my $X_0$ is a deterministic a vector of 1: $X_0= [1,..,1]^t$.

I have an r code that implement the manly procedure, but i have a problem with testing $H_0: β_0= 0$.

The procedure work well for the "normal" covariates coefficients tests($β_1$ and $β_2$) but strange things appear when it come to testing $β_0$. However theoretically $X_0= [1,..,1]^t$ can be consider as a covariates and so the Manly procedure could be applied.

So my question is: Is it possible to perform such a Manly test on the intercept (in the literature the question was often eluded)? If it is the case, how to get out of this case?

Answer 1

I believe I came across the same problem. Performing a permutation test using the procedure Kjetil describes produces p-values that agree very well with the classical ANOVA. The one odd-ball statistic was always the intercept; it seems to be consistently off (too big).

I am performing my permutation tests on the Fuel Consumption Data from "Applied Linear Regression" by Sanford Weisberg 2nd edition (page 35).

I am performing my permutation test in Microsoft Excel using macros.

The model is just a simple linear model of the predictors; no squared terms and no interaction terms.

I noticed something quite peculiar: If I performed the regression using the 'Regression' option from the 'Data Analysis' menu, my permutation p-values matched the ANOVA p-values well, except for the intercept.

If I performed the regression using the 'LINEST' function in the worksheet, the intercept's p-value matched that from the ANOVA, but now the last regression coefficient (FUELC) was way off. An interesting note about LINEST: it reports coeffcients and their standard errors in reverse; the intercept is reported last. I wondered if the difference was caused by the algorithms calculating the statistics in a different order?

I had the idea that things weren't working out for the statistic that's calculated last. Exchanging the last two columns in my domain data and rerunning confirmed my suspicions: now FUELC had the correct p-Value, but the last variable (ROAD) was way off (again too high)!

Since permutation testing was always generating a p-value that was too high for the last variable, I thought perhaps adding a random predictor variable e[0,1] to the model would improve things.... And it did!

Now my permutation tests generate p-values that are extremely close to the usual ANOVA for all varaibles including the intercept! But ONLY when the regressions are performed with the LINEST function.

Unfortunately I don't have the expertise to explain why adding a random variable to my set of predictors worked, or why the permutation test Kjetil described doesn't work for the intercept. I am hoping my experiments will inspire somebody to explain.