7
$\begingroup$

Let the "ordinary-least-squares regression of $Y$ on $X$" be given by $$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i\text{.}$$ Suppose I run the following:

  • The OLS regression of $Y$ on $X$
  • The OLS regression of $Y$ on $Z$
  • The OLS regression of $X$ on $Z$

and in all three cases, their slope coefficients $\hat{\beta}_1 = 2$.

I am interested in the slope coefficient of $Y$ on $X + Z$. How does this compare to the value $2$ (i.e., less than, equal to, greater than, or impossible to know)?

Attempt. Let $\hat{\beta}_{Y, X}$ be $\hat{\beta}_1$ in the case of the OLS regression of $Y$ on $X$, and similarly for the other three cases. Then we know that $$\hat{\beta}_{Y, X} = \dfrac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sum(x_i - \bar{x})^2}$$ hence $$\dfrac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2} = \dfrac{\sum_{i=1}^{n}(z_i - \bar{z})(y_i - \bar{y})}{\sum_{i=1}^{n}(z_i - \bar{z})^2} = \dfrac{\sum_{i=1}^{n}(x_i - \bar{x})(z_i - \bar{z})}{\sum_{i=1}^{n}(z_i - \bar{z})^2} = 2\text{.}$$ We also know that regression coefficients are unaffected by centering, so without loss of generality, I assume that all variables are centered and that $\sum_{i=1}^{n}(x_i - \bar{x})^2 = \sum_{i=1}^{n}(z_i - \bar{z})^2 = 1$, leading to $$\sum_{i=1}^{n}x_iy_i = \sum_{i=1}^{n}x_iz_i = 2\text{.}$$ Thus, by linearity of the arithmetic mean, we have $$\hat{\beta}_{Y, X+Z} = \dfrac{\sum_{i=1}^{n}y_i(x_i + z_i)}{\sum_{i=1}^{n}(x_i + z_i)^2} = \dfrac{4}{\sum_{i=1}^{n}(x_i^2 + 2x_iz_i + z_i^2)}\text{.}$$ Here's where I'm stuck. Can we do something clever with the above quantity?

I am told the answer is that $\hat{\beta}_{Y, X+Z}< 2$.

$\endgroup$
13
  • 4
    $\begingroup$ consider that for simple regression the standardized beta is the same as Pearson's correlation $r$, which cannot be greater than 1. Standardization certainly affects betas. $\endgroup$ Commented Mar 7, 2022 at 14:25
  • 1
    $\begingroup$ Re your top comment, I don't think you can stipulate that all three have SS = 1, and get marginal slopes =2. Just x & z should be OK, but y would have to be larger. $\endgroup$ Commented Mar 7, 2022 at 14:31
  • 2
    $\begingroup$ Assuming $\newcommand{\Var}{\rm Var} r(X,Y)=0$, $\Var(X+Y)=\Var(X)+Var(Y)$, thus the range of $X+Y$ is larger, but the variance / range of $Y$ doesn't increase. So the slope must be lower. Your $X$ & $Y$ are correlated, so $\Var(X+Y)=\Var(X)+Var(Y)+2{\rm Cov}(X, Y)$. As a curiosity, note that if $r(X, Y) = -1$, the sum would be a constant. $\endgroup$ Commented Mar 7, 2022 at 15:52
  • 1
    $\begingroup$ @gung Narrowly conceived, the question asks only to compare this slope to $2.$ But it's natural and illuminating to interpret that as a challenge to find the set of all values the slope possibly could attain. Moreover, by far the more interesting part of the solution IMHO is the lower bound, because the upper bound is explicitly in the question and clearly related to its suppositions. The lower bound is not so simply related to those suppositions. There's also an interesting prior question concerning whether any vectors might satisfy all the given assumptions. $\endgroup$
    – whuber
    Commented Mar 8, 2022 at 18:46
  • 1
    $\begingroup$ @whuber, fair enough $\endgroup$ Commented Mar 8, 2022 at 19:17

2 Answers 2

11
$\begingroup$

First, it is possible for these conditions simultaneously to hold, as I will show.

Second, the regression of $Y$ on $X+Z$ must lie in the open interval $(10/9,2)$ and can attain any value in that interval.


Vector notation is particularly convenient here.

The given information tells us (in the order given in the question) that

  1. $Y = 2X + E$ where $E$ is orthogonal to $X.$
  2. $Y = 2Z + \beta W + F$ where $F$ is orthogonal to $Z$ and $W$ (and $\beta$ is at this point unknown).
  3. $X = 2Z + W$ where $W$ is a zero-sum vector orthogonal to $Z.$

Implicitly, not all of $X,Y,$ and $Z$ are zero, for otherwise there's nothing of interest: we would just be saying, three times over, that $2$ times the zero vector is zero; and then there would be no bounds on the regression of $Y$ against $X+Z.$ Consequently, all of these vectors must be nonzero.

To find $\beta,$ use $(2)$ to regress $Y$ on $X$ as

$$2 = \hat\beta_{Y;X} = \frac{Y\cdot X}{||X||^2} = \frac{4||Z||^2 + \beta||W||^2}{4||Z||^2 + ||W||^2}.$$

When $W\ne 0,$ the unique solution is

$$\beta = 2 + \frac{4||Z||^2}{||W||^2}.$$

(When $W=0$ the equation reads $2=1,$ which has no solutions.)

We may now compute the regression of $Y$ against $X+Z$ as

$$\hat\beta_{Y;X+Z} = \frac{Y\cdot(X+Z)}{||X+Z||^2} = \frac{(2Z+\beta W + F)\cdot(3Z + W)}{||3Z + W||^2} = \frac{10||Z||^2 + 2||W||^2}{9||Z||^2 + ||W||^2}.$$

The right hand fraction is the slope of a ray emanating from the origin and passing through some point in the interior of the segment connecting the points $(10,9)$ and $(2,1)$ in the plane (with the weights given by the relative values of $||Z||^2$ and $||W||^2$), making it obvious the bounds are $10/9$ and $2/1$ -- and they cannot be attained because both weights are nonzero, QED.


Here is a histogram of the regression coefficients for a thousand simulated configurations (in $\mathbb{R}^{10}$):

Figure

The red vertical lines mark the bounds. The simulation supports these by exhibiting a full range of values filling these bounds but never extending beyond them.

The R code shows how the foregoing analysis was implemented. (It includes a post-simulation check that all the given regression coefficients equal $2,$ as intended.)

z <- scale(c(1, rep(0,9))) 
z <- z / sqrt(length(z)-1) # Must be zero mean, unit norm

set.seed(17)
sim <- replicate(1e3, {
  w <- rexp(length(z))
  w <- residuals(lm(w ~ z))
  w2 <- sum(w*w)
  y <- 2 * z + (4/w2 + 2)*w
  x <- 2*z + w
  
  a <- coefficients(lm(y ~ x + 0))
  b <- coefficients(lm(y ~ z + 0))
  c. <- coefficients(lm(x ~ z + 0))
  d <- coefficients(lm(y ~ I(x+z) + 0))
  c(a,b,c.,d)
})
table(sim[1:3,]) # All 2

hist(sim[4,], xlim=c(10/9,2), breaks=seq(10/9, 2, by=1/18), freq=FALSE,
     col=gray(.9), xlab="Value",
     main=expression(hat(beta)[group("", list(Y, X+Z), "")]))
abline(v=c(10/9, 2), col="Red", lwd=2, lty=2)
$\endgroup$
6
$\begingroup$

Note: OP states there is likely an error in the original question. My answer assumes everything OP states is correct thus the following answer may be incorrect

Let's focus on the denominator of the final expression. Let

$$A = \sum_{i = 1}^n x_i^2 + 2x_i z_i + z_i^2$$

If we can show that $A > 2$ then we know that $\hat{\beta}_{Y, X+Z} = \frac{4}{A} < 2$.

Now you've assumed that $\sum_{i=1}^n x_i z_i = 2$. We also know that each $x_i^2$, $z_i^2$ $\geq 0$.

Therefore: \begin{align} A &= \sum_{i = 1}^n x_i^2 + 2x_i z_i + z_i^2 \\ &= \sum_{i = 1}^n x_i^2 + 2\sum_{i = 1}^nx_i z_i + \sum_{i = 1}^n z_i^2 \\ &\geq \sum_{i = 1}^n 0 + 2 \times 2 + \sum_{i = 1}^n 0 \end{align} (because $x_i^2, z_i^2 \geq 0$ and $\sum x_i z_i = 2$) $$ \geq 4$$ Now since $A > 4$ (and specifically, $A > 2$) we can conclude that $\hat{\beta}_{X, Y+X} < 2$

$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.