2
$\begingroup$

I am struggling to understand how statisticians define a Latent Variable.

Suppose the joint probability associated with a statistical model is:

$$ p(x, Z, \theta)$$

$x$ are the random variables, $Z$ are the latent variables and $\theta$ are the parameters of the model that need to be estimated.

Although I haven't fully understood this point yet, but usually we want to remove the Latent Variables $Z$ from the estimation process since we are not interested in them. So using the laws of probability, we can write:

$$L(\theta; X) = p(X | \theta) = \int p(X, Z | \theta) dZ$$

But this brings me to my question: In a given model, how do we know which variables are Latent and which variables are not Latent?

The most popular example that comes up in this regard is the Gaussian Mixture Model:

$$p(x, \pi_1, \pi_2, \mu_1, \sigma_1, \mu_2, \sigma_2 ) = \pi_1 \cdot N(x | \mu_1, \sigma_1) + \pi_2 \cdot N(x | \mu_2, \sigma_2)$$

As I understand, $\mu_1, \sigma_1, \mu_2, \sigma_2$ are the actual (i.e. Non-Latent) parameters we are interested in estimating (i.e. $\theta$) and $\pi_1, \pi_2$ are the Latent Variables (i.e. $Z$). But why exactly is it defined this way?

I can understand the logic that $\mu_1, \sigma_1, \mu_2, \sigma_2$ might be more important for the analysis compared to $\pi_1, \pi_2$.

But what I don't understand is why $\pi_1, \pi_2$ are considered as fundamentally unobservable/hidden/missing parameters whereas $\mu_1, \sigma_1, \mu_2, \sigma_2$ are not considered as unobservable/hidden/missing.

Technically, you don't observe any parameter - all you observe is data, and then using Maximum Likelihood Estimation, you create a formula to estimate parameters based on the data ... such that your parameter estimates were those that had the highest probability of generating the data you observed.

So to sum things up, in any estimation problem, all parameters being estimated are unobserved and hidden ... but why do we consider some of them as Latent and some of them as Non-Latent?

Previous posts consulted:

$\endgroup$
0

1 Answer 1

3
$\begingroup$

I would say the fact that you integrate them out is what makes them latent variables; they no longer appear in the likelihood either as parameters or as data.

You could, in principle, regard the $z_i$ as parameters (and perhaps call them $\zeta_i$), and obtain a likelihood $p(X; \zeta,\theta)$ that would be written the same as $P(X|Z;\theta)$. The problem with doing that is that the maximum likelihood estimators are not consistent. It's a technically valid choice, but a bad one.

Or, if you were a Bayesian, you wouldn't make any technical distinction between latent variables and parameters. You would still need to make a basically similar decision about using a hierarchical prior over the $z_i$ (which is more like treating them as a latent variable) or using something like independent priors (which is more like treating them as parameters). Again, it would probably be better for estimation to use the hierarchical prior.

$\endgroup$
3
  • $\begingroup$ thanks Thomas. Is there some relationship between latent variables and the resulting likelihood becoming intractable? $\endgroup$
    – stats_noob
    Nov 7, 2023 at 1:58
  • $\begingroup$ any chance you could show me/link me why the MLE wont be consistent if I were to consider the latent variables the other way around? $\endgroup$
    – stats_noob
    Nov 7, 2023 at 2:01
  • $\begingroup$ If you take a Gaussian mixture model and treat the cluster memberships as parameters you end up with $k$-means clustering; the difference between them is discussed here: stats.stackexchange.com/questions/489459/… and probably other questions as well $\endgroup$ Nov 7, 2023 at 2:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.