0
$\begingroup$

Here https://stats.stackexchange.com/a/313138 @whuber describes a beautiful solution to generating a correlated vector to an existing one. The thing i cant figure out is $SD()$ in following expression: $$X_{Y;\rho} = \rho\, \operatorname{SD}(Y^\perp)Y + \sqrt{1-\rho^2}\,\operatorname{SD}(Y)Y^\perp.$$ And also this sentence:

("$\operatorname{SD}$" stands for any calculation proportional to a standard deviation.)

The exact question is: what does standard deviation of two orthogonal vectors or something proportional to it have to do with finding suitable linear combination of them with intended $\theta$?

EDIT2: another way of asking, what is wrong with this equation?: $$X_{Y;\rho} = \rho\, Y \frac{1}{\begin{Vmatrix}Y\end{Vmatrix}} + \sqrt{1-\rho^2}\,Y^\perp \frac{1}{\begin{Vmatrix}Y^\perp\end{Vmatrix}}.$$

Or if it is supposed to be correct, how $\operatorname{SD}(Y^\perp)$ is related to $\frac{1}{\begin{Vmatrix}Y\end{Vmatrix}}$, and $\operatorname{SD}(Y)$ to $\frac{1}{\begin{Vmatrix}Y^\perp\end{Vmatrix}}$ ?

$\endgroup$
10
  • $\begingroup$ By multiplying each variate by the standard deviation of the other, the standard deviation of each term is the same and equal to the standard deviation of $X$. $\endgroup$
    – Xi'an
    Commented Mar 9, 2021 at 13:54
  • $\begingroup$ Could you elaborate more, or give a reference or title to follow? Why equalizing $sd$ is important for two bases and how is it going to be the $sd$ of $X$? $\endgroup$
    – Hooman
    Commented Mar 9, 2021 at 14:48
  • 2
    $\begingroup$ I think the original answer of @whuber is detailed enough. $\endgroup$
    – Xi'an
    Commented Mar 9, 2021 at 14:49
  • 2
    $\begingroup$ Re the edit: just calculate the correlation coefficient for the wrong equation to see why you don't necessarily come up with the desired value of $\rho.$ $\endgroup$
    – whuber
    Commented Mar 15, 2021 at 22:02
  • $\begingroup$ @whuber Thank you for your comment. I did a horrible mistake in the previous edit. I corrected it. i ran both codes. your solution generates the exact $\rho$ ($\rho = 0.2$), but mine(edit2) gives just an accurate approximation ($\rho = 0.1994477$). And still i cant figure out why. $\endgroup$
    – Hooman
    Commented Mar 18, 2021 at 15:18

1 Answer 1

0
$\begingroup$

Indeed, if you do not go through the computation yourself, you will not see why the equation has to be the way it has to be (and as the comments indicate, it will be seen as a trivial point). So let us dive into the trivial computations.

(Will alter the notation as I find the original cumbersome)


The goal

The goal is to generate a variable $Z$ that fulfills a correlation $\rho$ with a target variable $T$. The equation indicates that this variable $Z$ will be defined as follows:

$$ \boldsymbol{Z} = \rho \sigma_{\hat{\epsilon}} \boldsymbol{T} + \sqrt{1-\rho ^2} \sigma_T \boldsymbol{\hat{\epsilon}} $$

As a reminder, $\boldsymbol{\hat{\epsilon}}$ are the residuals from the regression of a placeholder variable $Z_\text{temp}$ on the target variable: ${Z_\text{temp}}_i = \alpha_0 + \alpha_1 T_i + \hat\epsilon_i$.

Let us stop for a second to see what the equation consists of. We can see that each term follows the structure correlation factor * scaling * vector. Hence, as mentioned originally, this equation is just showing a linear combination of the vectors $\boldsymbol{\hat{\epsilon}}$ and $T$. Now we want that the correlation of this linear combination and the original target are equal to a desired correlation, i.e. $\operatorname{Cor}(Z, T) = \rho$.

The computation

We know the correlation is defined as:

$$ \rho = \operatorname{Cor}(Z, T) = \frac{1}{\sigma_Z \sigma_T}\mathbb{E}[(Z - \mathbb{E}[Z])(T - \mathbb{E}[T])] $$

We see that to expand this equation we need to know the value of the mean and the variance of $Z$, namely $\mathbb{E}[Z]$ and $\sigma_Z^2$.

If we apply the expectation operator to the definition of $Z$, we can see that the mean is:

$$ \mathbb{E}[Z] = \rho \sigma_{\hat{\epsilon}} \mathbb{E}[T] + \sqrt{1-\rho^2}\sigma_T\mathbb{E}[\hat{\epsilon}] \\ \mathbb{E}[Z] = \rho \sigma_{\hat{\epsilon}} \mathbb{E}[T] \tag{By definition of residuals} \\ $$

Then, to get the variance we will apply the variance operator to the same starting equation:

$$ \operatorname{Var}(Z) = \operatorname{Var}(\rho \sigma_{\hat\epsilon}T) + \operatorname{Var}(\sqrt{1-\rho^2}\sigma_T \hat\epsilon) + 2 \operatorname{Cov}(\rho \sigma_{\hat\epsilon}T, \sqrt{1-\rho^2}\sigma_T \hat\epsilon) $$

By construction we know that the covariance of the residuals and the targets is zero, hence the third term can be omitted. Then:

$$ \operatorname{Var}(Z) = \rho^2 \sigma_{\hat\epsilon}^2\operatorname{Var}(T) + \sqrt{1-\rho^2}^2 \sigma_T^2\operatorname{Var}( \hat\epsilon) \\ \operatorname{Var}(Z) = (\rho^2) \sigma_{\hat\epsilon}^2 \sigma_T^2 + (1-\rho^2) \sigma_T^2 \sigma_{\hat\epsilon}^2 \\ \operatorname{Var}(Z) = \sigma_T^2 \sigma_{\hat\epsilon}^2 = \sigma_Z^2 $$

Note what happened. We could factorize the correlation factors since both terms are just a product of the same variances. This would not happen if we would divide each variance by the standard deviation as in your proposed alternative. We can also appreciate why the terms were $\rho$ and $\sqrt{1-\rho^2}$ and not, say, $\rho$ and $1-\rho$: when squared they have to sum to one. This is why Xi'an said "the standard deviation of each term is the same and equal to the standard deviation of [$Z$]". We scale each vector by the standard deviation of the other one so they have the same variance, then we use functions of the correlations as weights in the linear combination.

After this small detour for computing the mean and the variance of $Z$, we can now expand our correlation calculation:

$$ \operatorname{Cor}(Z,T) = \frac{\mathbb{E}[(\rho \sigma_\hat\epsilon T + \sqrt{1 - \rho^2} \sigma_T \hat\epsilon - \rho \sigma_{\hat\epsilon} \mathbb{E}[T])(T - \mathbb{E}[T])]}{\sigma_Z \sigma_T} \\ = \frac{\rho \sigma_{\hat\epsilon} (\mathbb{E}[T^2] - \mathbb{E}[T]^2 - \mathbb{E}[T]^2 + \mathbb{E}[T]^2) + \sqrt{1 - \rho^2}(\mathbb{E}[\hat\epsilon T] - \mathbb{E}[\hat\epsilon] \mathbb{E} [T]) }{\sigma_Z \sigma_T} \\ $$

Then we can see that the first parenthesis is the MOSSOM definition of variance, whereas the second is the definition for the covariance, which by construction is zero between the residuals and the target. Thus:

$$ \operatorname{Cor}(Z,T) = \rho \frac{ \sigma_{\hat\epsilon} \sigma_T^2} {\sigma_Z\sigma_T} $$

And here is where we can see why it works out to multiply each vector by the other's standard deviation, since we showed that $\sigma_Z^2 = \sigma_{\hat\epsilon}^2 \sigma_T^2$, we can replace $\sigma_Z$ by $\sigma_{\hat\epsilon} \sigma_T$, thus fulfilling our goal:

$$ \operatorname{Cor}(Z,T) = \rho \frac{ \sigma_{\hat\epsilon} \sigma_T^2} {\sigma_{\hat\epsilon} \sigma_T^2} = \rho $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.