How to determine net effect in OLS?

Question

Let's say I have the following model: y = int + x1 + x2 + x3 + x1*x2 + e

x1 and x2 are binary indicators while x3 is continous.

The estimation of this model is as follows:

       Coefficient  Std Error   p-Value
x1    -0.01818      0.05123      0.6234
x2    -0.05153      0.04238      0.1527
x1*x2  0.07750      0.03641      0.0256
x3     -0.03585     0.04245      0.5033

To get the net effect of x1 when x2 = 1, I would add the coefficients of x1 + x1*x2. However, what about the significance. x1*x2 is statistically significant, but x1 is not. Can I still add the coefficients and claim it to be significant or do I need to somehow get the significance of of x1+x1*x2?

Answer 1

Some simple matrix algebra on the covariance matrix of your estimates

Any quality stats package will report (somewhere) an estimated covariance matrix for your estimates. Eg. if your model is:

$$ y = b_0 + b_1 x_1 + b_2 x_2 + b_3 x_3 + b_4 x_1x_2 + \epsilon$$

Under the usual assumptions of linear regression your estimate of vector $\mathbf{b}$ is asymptotically normal. Your stats package can give you its estimate of the covariance matrix of $\mathbf{b}$:

\begin{align*} \hat{\operatorname{Var}}\left( \mathbf{b} \right) &= \hat{\Sigma} \end{align*}

(For reference, the standard errors for estimate $\hat{\mathbf{b}}$ are the square root of the diagonal elements of $\hat{\Sigma}$.) If you want to calculate the standard error for the sum $b_1 + b_4$, you can create a vector $\mathbf{r}$ such that $\mathbf{r}'\mathbf{b} = b_1 + b_4$. That is:

$$ \mathbf{r} = \begin{bmatrix} 0 \\ 1 \\ 0 \\ 0 \\ 1 \end{bmatrix} $$ And then compute:

$$\begin{align*}\sqrt{\mathrm{Var}\left( b_1 + b_4 \right)} &= \sqrt{\mathbf{r}'\Sigma \mathbf{r} }\\ &= \sqrt{\sum_{i}\sum_{j} r_i r_j \Sigma_{ij}} \end{align*}$$

Run another regression approach

This may be easier to program, but you have to be careful in figuring out the regression to run. Instead of estimating:

$$ y = b_0 + b_1 x_1 + b_2 x_2 + b_3 x_3 + b_4 \left( x_1x_2 \right) + \epsilon$$

You could instead run another linear regression:

$$ y = c_0 + c_1 (x_1 - x_1x_2) + c_2 x_2 + c_3 x_3 + c_4\left(x_1x_2\right) + \epsilon$$

That is, create a new variable $x_1 - x_1x_2$ and include that instead of $x_1$. You should find that $c_4 = b_1 + b_4$, and you can then use the standard error your stats package gives you for $c_4$.

Why this works:

Let matrix $A$ be defined as:

$$A = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0\\ 0 &0 & 1&0 & 0\\ 0 & 0 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 & 1 \end{bmatrix} \quad \quad A^{-1} = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0\\ 0 &0 & 1&0 & 0\\ 0 & 0 & 0 & 1 & 0 \\ 0 & -1 & 0 & 0 & 1 \end{bmatrix} $$

Let $\mathbf{c} = A \mathbf{b}$ hence: $$\mathbf{c} = \begin{bmatrix} b_0 \\ b_1 \\ b_2 \\ b_3 \\ b_1 + b_4 \end{bmatrix}$$.

If you transform the data by $Z = X A^{-1}$ the OLS estimator for $ y_i = z_0 + c_1 z_{i1} + c_2 z_{i2} + c_3 z_{i3} + c_4 z_{i4} + u_i$ is: $$\begin{align*} \hat{\mathbf{c}} &= (Z'Z)^{-1}(Z'\mathbf{y}) \\ &= ({A'}^{-1}X'XA^{-1})^{-1}({A'}^{-1}X'\mathbf{y})\\ &= A(X'X)^{-1}X'\mathbf{y} \\ &= A \hat{\mathbf{b}} \end{align*}$$

How do you construct $Z = X A^{-1}$? Since $X = \begin{bmatrix} x_0, & x_1, & x_2, & x_3, & x_1 x_2 \end{bmatrix}$ we have:

$$ \begin{bmatrix} x_0, & x_1, & x_2, & x_3, & x_4 \end{bmatrix} A^{-1} = \begin{bmatrix} x_0, & x_1 - x_1x_2, & x_2, & x_3, & x_1x_2 \end{bmatrix}$$

So if you run the regression:

$$ y = c_0 + c_1 (x_1 - x_1x_2) + c_2 x_2 + c_3 x_3 + c_4 \left(x_1x_2\right) + \epsilon$$ You'll find $c_0 = b_0$, $c_1 = b_1$, $c_2 = b_2$, $c_3 = b_3$, and $c_4 = b_1 + b_4$ and the standard error for $c_4$ is the standard error for $b_1 + b_4$.