Parameters vs latent variables - Cross Validated most recent 30 from stats.stackexchange.com 2019-09-17T02:49:09Z https://stats.stackexchange.com/feeds/question/139066 https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/139066 13 Parameters vs latent variables Luca https://stats.stackexchange.com/users/36540 2015-02-24T17:48:33Z 2019-03-18T15:09:27Z <p>I have asked about this before and have really been struggling with identifying what makes a model parameter and what makes it a latent variable. So looking at various threads on this topic on this site, the main distinction seems to be:</p> <p>Latent variables are not observed but have an associated probability distribution with them as they are variables and parameters are also not observed and have no distribution associated with them which I understand as that these are constants and have a fixed but unknown value that we are trying to find. Also, we can put priors on the parameters to represent our uncertainty about these parameters even though there is only one true value associated with them or at least that is what we assume. I hope I am correct so far?</p> <p>Now, I have been looking at this example for Bayesian weighted linear regression from a journal paper and been really struggling to understand what is a parameter and what is a variable:</p> <p>$$y_i = \beta^T x_i + \epsilon_{y_i}$$</p> <p>Here $x$ and $y$ are observed but only $y$ is treated as a variable i.e. has a distribution associated with it. </p> <p>Now, the modelling assumptions are:</p> <p>$$y \sim N(\beta^Tx_i, \sigma^2/w_i)$$</p> <p>So, the variance of $y$ is weighted. </p> <p>There is also a prior distribution on $\beta$ and $w$, which are normal and gamma distributions respectively. </p> <p>So, the full log likelihood is given by:</p> <p>$$\log p(y, w, \beta |x) = \Sigma \log P(y_i|w, \beta, x_i) + \log P(\beta) + \Sigma \log P(w_i)$$</p> <p>Now, as I understand it both $\beta$ and $w$ are model parameters. However, in the paper they keep referring to them as latent variables. My reasoning is $\beta$ and $w$ are both part of the probability distribution for the variable $y$ and they are model parameters. However, the authors treat them as latent random variables. Is that correct? If so, what would be the model parameters?</p> <p>The paper can be found here (<a href="http://www.jting.net/pubs/2007/ting-ICRA2007.pdf">http://www.jting.net/pubs/2007/ting-ICRA2007.pdf</a>). </p> <p>The paper is Automatic Outlier Detection: A Bayesian Approach by Ting et al.</p> https://stats.stackexchange.com/questions/139066/-/140626#140626 6 Answer by alberto for Parameters vs latent variables alberto https://stats.stackexchange.com/users/36312 2015-03-06T10:32:50Z 2019-03-18T15:09:27Z <p>In the paper, and in general, (random) <strong>variables</strong> are everything which is drawn from a probability distribution. <strong>Latent</strong> (random) <strong>variables</strong> are the ones you don't directly observe (<span class="math-container">$y$</span> is observed, <span class="math-container">$\beta$</span> is not, but both are r.v). From a latent random variable you can get a posterior distribution, which is its probability distribution conditioned to the observed data.</p> <p>On the other hand, a <strong>parameter</strong> is fixed, even if you don't know its value. Maximum Likelihood Estimation, for instance, gives you the most likely value of your parameter. But it gives you a point, not a full distribution, because fixed things do not have distributions! (You can put a distribution on how sure you are about this value, or in what range you thing this value is, but this is is not the same as the distribution of the value itself, which only exists if the value is actually a random variable)</p> <p>In a Bayesian setting, you can have all of them. Here, parameters are things like the number of clusters; you give this value to the model, and the model considers it a fixed number. <span class="math-container">$y$</span> is a random variable because it is drawn from a distribution, and <span class="math-container">$\beta$</span> and <span class="math-container">$w$</span> are latent random variables because they are drawn from probability distributions as well. The fact that <span class="math-container">$y$</span> depends on <span class="math-container">$\beta$</span> and <span class="math-container">$w$</span> doesn't make them "parameters", it just makes <span class="math-container">$y$</span> dependent on two random variables.</p> <p>In the paper they consider that <span class="math-container">$\beta$</span> and <span class="math-container">$w$</span> are random variables. </p> <p>In this sentence:</p> <blockquote> <p>These update equations need to be run iteratively until all parameters and the complete log likelihood converge to steady values</p> </blockquote> <p>in theory they talk about the two parameters, not the ones that are random variables, since in EM this is what you do, optimizing over parameters.</p>