Correlations with a linear combination means correlation with individual variables?

Question

I was having this conversation with someone, and I feel like I have an explanation, but would greatly appreciate if someone could verify whether what I'm thinking is just completely wrong.

He made the argument that if you know that a variable [a] is correlated with a linear combination of variables [C], then you know something about the correlations between [a] and the constituent variables of [C]. And vice versa.

I feel like this is wrong, but I don't have a great explanation of why. There are two reasonings I've come up with, but both feel like I just pulled them out of nowhere, and I could use some feedback:

1. Creating a linear combination of variables, variables which potentially are uncorrelated with each other, could be seen as equivalent to adding random noise to any one of those variables, which would mean that you can't really say anything about the correlations with some outside variables after the combination.
2. I had this explanation that is very vague in my head and involves vectors and conic sections, but I'm not sure how to make it make sense. Basically, if you know the correlations between [a] and the constituent variables of [C], you can construct a particular vector space within which the correlation of [a] and [C] must lie, but I don't think that works in the opposite direction if you start with a correlation between [a] and [C].

Any feedback would be greatly appreciated.

Answer 1

It would be helpful if your friend could specify exactly how you are supposed to go from knowing the correlation between a linear combination of variables to knowledge of the marginal / pairwise correlations. I have no idea what he is talking about.

I'm not sure about your proposed reasons, but a simple consideration would be multicollinearity. When predictor / explanatory variables are correlated with each other, the estimated coefficients, $\hat\beta_j$s, can vary wildly depending on which variables are included in the model or not. The full model, with all variables included, and the marginal model, with only one covariate included, are special cases of this. It is quite possible, for instance, for the estimated relationship to vary in sign (from a positive relationship to a negative one, or vice versa). You could also look up Simpson's Paradox as a simple example of this.

Multicollinearity is a common topic on CV, you can read more about it here by searching on the associated tag: multicollinearity. There is also a thread on Simpson's Paradox on CV (and for which I wrote an answer), which you can read here: basic-simpsons-paradox.