What's the difference between two correlated variables and two variables with common latent factor?

Question

Is it possible to distinguish two variables that are just correlated with each other but are not part of the same phenomenon, from two variables, that are part of the same phenomenon and share a common superior/higher latent factor? I mean a mathematical distinction and not a theoretical definition of a variable.

Let's say I measure the relationship between job quality and salary received, and I can assume that these are variables that measure other (separate) phenomena, different things; lets assume that they are correlated. On the other hand, I have the job quality measured by the employee's manager and job quality measured by self-report of the employee; let's assume that these two variables are also correlated with each other. Sooo... The first two measure separate phenomena, the next two measure one and the same phenomenon which is simply expressed or measured differently. The first two are simply just/only correlated, the next two have an higher factor or they are two variables expressing the same thing.

Do we have a mathematical, statistical or any other rule that can tell us (only by looking at the variables, their numerical values), do the two variables have an common latent factor OR are "just randomly correlated"? What's the mathematical difference between the factor items and correlated variables? PCA, CFA, SEM techniques can tell these things apart, or is it a completely theoretical problem? If these techniques can discriminate differences, what factors/indices should be considered?

Answer 1

I believe there are two things you need to mentally separate:

• Actual correlation, and

• Spurious correlation.

---

If two things are truly correlated, then either: one of them causes the other, or there is a latent factor causing both, or, under some conditions, they both cause another observed factor.

Either way, the saying "no correlation without causation" is definitive. If two things move together, it's because they have some causal relationship.

---

However, sometimes we see correlations in our data that are not true correlations. If your data is a random sample, this becomes a question of hypothesis testing: is the correlation I'm seeing likely real, or is it possible that it is a sampling artifact?

In this case you can use bootstrap procedures to compute the sampling error distribution of your correlation measure and see if it's significantly different from zero. Then it might be real.

If you don't have randomised data, you're out of luck. There's no statistical way to do hypothesis testing then, and you have to guess.

---

To bootstrap the sampling distribution of the correlation measure you're using, do the following:

Let's say your observed data is 25 pairs of values.

1. Draw, with replacement, a random sample of 25 pairs from the observed data. Compute the correlation measure for this random sample. Call the resulting correlation value the first bbootstrap replication of the correlation.

2. Do the first step many times to generate many bootstrap replications of the correlation.

3. Check if, say, 95 % of the bootstrap replications are greater than 0. Then you can say with 95 % confidence that the correlation is indeed positive.

Answer 2

Because latent factor is not an observed phenomenon (at least in the current analysis) but is a construct or essense, it could be postulated literally for any correlation observed between phenomena. Including your example "job quality vs salary received" (everything depends on your imagination - what might be such a factor).

Even in a linear regression between two phenomena, $X$ and $Y$, the prediction $\hat Y$ can be interpreted as a latent variable within the observed $X$ in this instance, which is the cause of correlation between $X$ and $Y$. Moreover, we could further assume that there is noise within $X$ besides the latent factor in it, and so to turn to an "errors-in-variables" approach in regression.

Is it a completely theoretical problem? Mostly, it is a theoretical. Please glance on the picture in the footnote 2 in this answer. A correlation between two variables could be explained: a) there is one common factor, b) there are two independent common factors, c) there are two common factors, correlated (we can allow for correlated factors in a Factor analysis). Note that case (c) becomes close to declaring each variable its own "latent factor", in which case "latent" becomes indistinguishable from the "observed". So there is more dialectics than math.

In a latent variable technique such as factor analysis we assume all the analyzed variables have to do with the latent factors, because the latents are extracted from their collection; but they are loaded by the latents by different amount, depending on the correlations.

(A zero correlation in a collection of two variables - a toy example - produces loadings .707 by PCA but .000 by Exploratory FA by principal axes method, on both variables. That says that the method of extraction of the latents is important: PCA in this example clearly spoils the job while FA is adequate.)

A good related question for you to check (nuances in terminology): What's the relationship between covariance, shared variance, and common variance?

We can, but we do not have to assume latent factor to account (for) a correlation. For example, if we want to compute a distance between two respondents (x and y) by some correlated features and we want our distance to incorporate/express, rather than ignore, the condition of the correlatedness, we might:

• perform PCA (or factor analysis), to extract one or few latents, and then compute euclidean distance in the space of these latent(s) only;
• don't extract the latents, but compute the distance in the frame of original variables, but in oblique axes (angles between them to reflect the correlations): $d_{xy} = \sqrt{(x-y)'R(x-y)}$ (Butler's distance, R is the correlation matrix).

These two are both reasonable, quite different solutions.