Bounds for common correlation of uncorrelated random variables with another random variable

Question

Suppose we have a random variable $y$ and a collection $(x_1,\dots,x_n)$ of $n$ random variables that are all uncorrelated: $\operatorname{corr}(x_i,x_j) = 0$ $\forall i \ne j$ and that all have the same correlation with $y$: $\operatorname{corr}(y,x_i) = \rho$ $\forall i$. Obviously, they can't all have correlation $1$ or $-1$ with $y$ and be uncorrelated at the same time, so there must be bounds.

This question came up when I was solving a different problem, finding the correlation between $y$ and the linear model $\hat{y} = X(X^\intercal X)^{-1}X^\intercal y$, where $X = (x_1,\dots,x_n)$ and $y,x_i$ are now thought of as $m\ge n$ random samples from the random variables. I found that if the random variables have mean $0$ and variance $1$, the expected value is $\operatorname{corr}(y,\hat{y})=\rho\sqrt{n}$, there's also a more complicated formula for the general case (can post if interested, wanna keep this post short).

This lead me to conjecture that the bounds for $\rho$ are $\left[-\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}\right]$, now thinking about proving it for the general case.

Answer 1

In terms of $(x_1, x_2, \ldots, x_n, y)$ the correlation matrix is

$$\Sigma_n = \pmatrix{1 & 0 & \cdots & 0 & \rho\\ 0 & 1 & \cdots & 0 & \rho\\ \vdots & \vdots & \ddots & 0 & \rho\\ 0 & 0 & \cdots & 1 & \rho\\ \rho & \rho & \cdots & \rho & 1}.$$

Its principal proper minors are all identity matrices, which have positive determinants. The determinant of $\Sigma_n$ itself can be computed by first subtracting $\rho$ times each of the first $n$ rows from the last row (of rhos), thereby producing a new bottom row of

$$\pmatrix{0 & 0 & \cdots & 0 & 1 - n\rho^2}.$$

This row-reduction produces an upper triangular matrix, whence its determinant is the product of its diagonal entries, equal to

$$|\Sigma_n| = 1 \times 1 \times \cdots \times 1 \times (1 - n\rho^2) = 1 - n\rho^2.$$

This is non-negative if and only if

$$-\frac{1}{\sqrt{n}} \le \rho \le \frac{1}{\sqrt{n}}.\tag{*}$$

Sylvester's Criterion states this is equivalent to positive-semidefiniteness of $\Sigma_n.$

---

We don't really need to invoke Sylvester's Criterion. All such matrices $\Sigma_n$ do indeed arise as correlation matrices when $(*)$ holds; and otherwise, because $|\Sigma_n|$ becomes negative for any other values of $\rho,$ it would be impossible for $\Sigma_n$ to be a correlation matrix.

Take, for instance, $n+1$ uncorrelated zero-mean, unit-variance random variables $Z_1, Z_2, \ldots, Z_{n+1}.$ Let $\rho$ satisfy $(*)$ and set

$$\lambda = \pm \sqrt{1/\rho^2 - n}$$

to have the same sign as $\rho.$ Set $$X_i=\pm Z_i\text{ for }1\le i\le n$$ (according to the sign of $\rho$) and $$Y=(Z_1+Z_2 + \cdots + Z_n + \lambda Z_{n+1}).$$ Since

$$\operatorname{Cov}(X_i, Y) = \pm 1$$

and

$$\operatorname{Cov}(Y,Y) = n + \lambda^2,$$

the correlations between the $X_i$ and $Y$ all equal

$$\operatorname{Cor}(X_i, Y) = \frac{\pm 1}{\sqrt{n + \lambda^2}} = \rho$$

as intended.