13
$\begingroup$

I have asked about this before and have really been struggling with identifying what makes a model parameter and what makes it a latent variable. So looking at various threads on this topic on this site, the main distinction seems to be:

Latent variables are not observed but have an associated probability distribution with them as they are variables and parameters are also not observed and have no distribution associated with them which I understand as that these are constants and have a fixed but unknown value that we are trying to find. Also, we can put priors on the parameters to represent our uncertainty about these parameters even though there is only one true value associated with them or at least that is what we assume. I hope I am correct so far?

Now, I have been looking at this example for Bayesian weighted linear regression from a journal paper and been really struggling to understand what is a parameter and what is a variable:

$$ y_i = \beta^T x_i + \epsilon_{y_i} $$

Here $x$ and $y$ are observed but only $y$ is treated as a variable i.e. has a distribution associated with it.

Now, the modelling assumptions are:

$$ y \sim N(\beta^Tx_i, \sigma^2/w_i) $$

So, the variance of $y$ is weighted.

There is also a prior distribution on $\beta$ and $w$, which are normal and gamma distributions respectively.

So, the full log likelihood is given by:

$$ \log p(y, w, \beta |x) = \Sigma \log P(y_i|w, \beta, x_i) + \log P(\beta) + \Sigma \log P(w_i) $$

Now, as I understand it both $\beta$ and $w$ are model parameters. However, in the paper they keep referring to them as latent variables. My reasoning is $\beta$ and $w$ are both part of the probability distribution for the variable $y$ and they are model parameters. However, the authors treat them as latent random variables. Is that correct? If so, what would be the model parameters?

The paper can be found here (http://www.jting.net/pubs/2007/ting-ICRA2007.pdf).

The paper is Automatic Outlier Detection: A Bayesian Approach by Ting et al.

$\endgroup$
  • 2
    $\begingroup$ It might help to list a cite to the paper (& maybe a link). Part of the problem is that what these are exactly differs b/t the Frequentist & Bayesian perspectives. From the Bayesian perspective, a parameter does have a distribution--it isn't just something added on to represent uncertainty. $\endgroup$ – gung Feb 24 '15 at 17:53
  • $\begingroup$ I thought it would be unfair as people would think that I expect them to read the paper without explaining things but I have put it now. $\endgroup$ – Luca Feb 24 '15 at 19:54
  • $\begingroup$ Why can't you put a prior on a latent variable? I'm a Bayesian novice, but it seems like you should be able to do that. $\endgroup$ – robin.datadrivers Feb 24 '15 at 20:20
  • $\begingroup$ I think one certainly can, of course and have to in the Bayesian setup. However, I am not sure why $w$ or $\beta$ are variables in this setting. To me they looks like parameters of the model. I am having trouble telling what is it that makes say $w$ a variable rather than a parameter in this setup. I am a novice as well, as you can clearly see... $\endgroup$ – Luca Feb 24 '15 at 20:21
  • 2
    $\begingroup$ Thanks, @Luca. It wouldn't be good if you were requiring people to read the paper, but having it there for context is nice. I think you've done this right. $\endgroup$ – gung Feb 24 '15 at 20:37
6
$\begingroup$

In the paper, and in general, (random) variables are everything which is drawn from a probability distribution. Latent (random) variables are the ones you don't directly observe ($y$ is observed, $\beta$ is not, but both are r.v). From a latent random variable you can get a posterior distribution, which is its probability distribution conditioned to the observed data.

On the other hand, a parameter is fixed, even if you don't know its value. Maximum Likelihood Estimation, for instance, gives you the most likely value of your parameter. But it gives you a point, not a full distribution, because fixed things do not have distributions! (You can put a distribution on how sure you are about this value, or in what range you thing this value is, but this is is not the same as the distribution of the value itself, which only exists if the value is actually a random variable)

In a Bayesian setting, you can have all of them. Here, parameters are things like the number of clusters; you give this value to the model, and the model considers it a fixed number. $y$ is a random variable because it is drawn from a distribution, and $\beta$ and $w$ are latent random variables because they are drawn from probability distributions as well. The fact that $y$ depends on $\beta$ and $w$ doesn't make them "parameters", it just makes $y$ dependent on two random variables.

In the paper they consider that $\beta$ and $w$ are random variables.

In this sentence:

These update equations need to be run iteratively until all parameters and the complete log likelihood converge to steady values

in theory they talk about the two parameters, not the ones that are random variables, since in EM this is what you do, optimizing over parameters.

$\endgroup$
  • $\begingroup$ The question was about latent variables. $\endgroup$ – Tim Mar 6 '15 at 10:35
  • $\begingroup$ fixed, I hope it is clearer now. $\endgroup$ – alberto Mar 6 '15 at 10:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.