8
$\begingroup$

In frequentist estimation, a large degree of emphasis in SEM/CFA modeling is placed on whether the model is identified, that is, whether each parameter can be uniquely estimated from the data. The typical idea is that, for p parameters, then you need at least p bits of information to identify the model.

Now for the Bayesians, the concept of identification is not so black and white. From my read of the literature (Palomo et al., 2011), Bayesian estimation will always produce a posterior distribution for each parameter, no matter their number. Rather, identification is defined as if Bayesian learning occurs: if the posterior is different from the prior (due to the influence of the data). An unidentified Bayesian model is one in which the prior and posterior are exactly the same, and nothing is learned from the data. Adding more parameters, it seems, reduces the ability to estimate each one, and after a certain point the model becomes unidentified.

Model identification is straightforward in a frequentist framework, but it is much more vague in the Bayesian paradigm and provokes a lot of questions.

How does adding more parameters reduce the overall information available to estimate the others? Does adding information in the priors increase the ability to estimate the other free parameters (i.e., makes them more identifiable/increased Bayesian learning)?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ There is nothing specific in your question as far as SEM/CFA go. You are right though that if a model is underidentified (in the "traditional", or "frequentist", sense) -- which usually means that some parameters are identified and others aren't -- then Bayesian learning will not contribute anything to the parameters that are underidentified, and they will retain their priors as their posteriors. In an MCMC implementation, you may not see any obvious convergence failures, as you may still have a nice convergent chain -- it will just converge right away to $N(0,10^6)$... $\endgroup$
    – StasK
    Commented Dec 14, 2015 at 15:01
  • $\begingroup$ You are right about the SEM/CF issue. However, SEM/CFA is often where the issue of identification comes up. In a regression model, I can fit more parameters than data points, and identification is usually not a concern. $\endgroup$
    – ATJ
    Commented Dec 17, 2015 at 23:02
  • $\begingroup$ Just for future reference: what you're describing by "an unidentified Bayesian model is one in which the prior and posterior are exactly the same, and nothing is learned from the data" is referred to as "posterior collapse", especially in the more recent ML literature. $\endgroup$
    – Durden
    Commented Jan 7 at 20:33

1 Answer 1

Reset to default
3
$\begingroup$

"An unidentified Bayesian model is one in which the prior and posterior are exactly the same, and nothing is learned from the data". While this is not a main concerns, the mentioned point stays valid in Bayesian setting, if from some $\Theta_1 \ne \Theta_2$, $p(x|\Theta_1)=p(x|\Theta_2)$ then the posterior distribution will not converge to a "prior independent" solution and the prior will lead a part of inference (even in the asymptotical limit of observations number) which may be an undersirable property.

For your to last question, there is a closely related question with answer on this site : Is there any reason to prefer a bayesian model with few variables?.

Moreover consider that adding an extra parameter $\theta_2$ through $p(\theta_1|\theta_2)$ and hyperprior on $\theta_2$, $p(\theta_1)$ writes: $$ p(\theta_1)=\int_R p(\theta_1|\theta_2)p(\theta_2)d\theta_2 $$ I do not know if there are some general results relating $var(\theta_1)$ and $var(\theta_2)$ but for example $ var(\theta_1) \ngtr var(\theta_1|\theta_2=\alpha)$ in general. So adding an hyperparameter to the model does not systematically result in a prior model of higher variance.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.