In a hierachical model, we have $$p(x_1, \dots, x_N, z_1, \dots, z_N, \beta) = p(\beta) \prod_{i=1}^N p(z_i | \beta) p(x_i | z_i) $$ In such models, we have $x_i \perp x_j | \beta, i \neq j$. However, in general, we do not have $x_i \perp x_j$, while in general we usually assume that data points $x_{1:N}$ are drawn i.i.d. from some model.

Is this a somewhat contradictory?

  • 1
    $\begingroup$ what does this formula mean in laymen terms? and what component indicates hierarchicity? $\endgroup$ – develarist Sep 10 '20 at 17:27
  • $\begingroup$ For a concrete example, let's say we are modeling the SAT score for a population of individuals. $x_i$ are the scores, $z_i$ are the skills. The scores can be seen as some random fluctuation of the skills. $\beta$ models the population of the skills. The formula represents generating process as i just mentioned. $\endgroup$ – David Sep 10 '20 at 20:32
  • $\begingroup$ hierarchy suggests seniority doesn't it? i don't see it being indicated $\endgroup$ – develarist Sep 10 '20 at 21:10

The resolution of this issue lies in recognition of the difference between conditional independence versus marginal independence. Conditional independence does not generally imply marginal independence, and indeed, in most cases it implies positive marginal correlation. This occurs because outcomes of one of the random variables in the set gives information on the underlying parameters (the conditioning parameters of the underlying distribution) and this in turn gives information on the other random variables in the set. If you would like a more detailed answer than this, I recommend you read O'Neill (2009). This paper sets out the relationship between conditional and marginal independence in classical and Bayesian models, and it discusses some of the implications of conditional independence. It also gives conditions for induced marginal positive correlation for conditionally IID variables.

In the hierarchical model you have given, the elements $x_1,...,x_N$ are conditionally independent given $\beta$. (You are wrong to assert that they are IID --- this holds only conditionally on the parameter $\beta$ and only assuming that $z_1,...,z_N$ are identically distributed conditional on $\beta$.) Each observed value $x_i$ gives information about the underlying value $z_i$ and through it gives information about $\beta$. Conditional on the underlying parameter value $\beta$ the observations are independent, but once you take that conditioning away they are no longer independent because they each give information on $\beta$, and so they indirectly give information on each other. Unless you are dealing with a pathological case, if you derive the joint density for $x_1,...x_N$ (without conditioning on $\mathbf{z}$ or $\beta$) you will find that there is dependence between the values, and most likely positive correlation. This is a standard result in Bayesian models and a standard outcome for conditionally IID values in general.

  • $\begingroup$ Thanks for the answer, which answer a lot of my questions. But if I am modeling some data with a Gaussian Mixture: $x_i$ is the data, $z_i$ is discrete cluster asssignment, which aligns with the model assumption. We often say statement like "assume data generated i.i.d. from mixture of Gaussian". However, in our prob model, as you said, $x_i$ will be correlated. To be more concrete, in posterior predictive distribution, $x_i$ will also be correlated. To me, it's like modeling i.i.d. data with non i.i.d. distribution. Would be great if you can point to me the breaking part in this logic. $\endgroup$ – David Sep 11 '20 at 4:14
  • $\begingroup$ The problem there is that when you say, "assume data generated IID from mixture of Gaussian", what you are actually modelling is IID conditional on that Gaussian distribution (i.e., conditional on its parameters). The shorthand statement covers the fact that this is actually a conditional IID model. (Note: If we actually modelled by using marginal IID assumption then modelling would be over very quickly --- we would just say, all values are marginally IID, therefore they cannot give any information on each other, so none have any predictive value for the others.) $\endgroup$ – Ben Sep 11 '20 at 4:17
  • $\begingroup$ This clarifies a lot. So to reiterate this, the usual IID assumptions are rarely marginal IID. Rather, always the case, they are IID conditional on some particular model. I'll go to that paper for more details. Thanks $\endgroup$ – David Sep 11 '20 at 5:08
  • 1
    $\begingroup$ Indeed, you would never bother with a marginal IID assumption, because then (by definition) you cannot learn about one observation from another (since they are independent). I recommend you read the paper linked in this answer, as it discusses all of this in quite a bit of detail. $\endgroup$ – Ben Sep 11 '20 at 5:28

Typically in hierarchical models, the variable $Z_i$ is taken to be some categorical variable that groups different observations, not a feature that varies between each and every individual. Taking the SAT example, one could reasonably take $Z_i$ to be the mean score of the school that student $i$ attended, since many students will attend the same school. Taking $Z_i$ as the individual skill level kinda defeats the purpose of using a hierarchical model, since this variable does not group your observations at all.

To make my point clearer, let us take the SAT example. We could use a model where students' scores are normally distributed around their school's mean score, that is:

$$X_i|Z_i\sim\mathcal N(Z_i,\sigma^2)$$

Then, to complete the hierarchical model, we say the schools' mean are normally distributed around a grand mean:

$$Z_i\sim\mathcal N(\mu,\alpha)$$

Now, one has to define priors for $\mu$, $\sigma^2$ and $\alpha$ and presto: the model is now fully specified.

The advantages of this approach include:

  • When we predict a student's score, this allows us to take into consideration the school they come from.
  • If a school has too few observations, this model automatically takes the distribution of the other schools' results into account when estimating the mean score for that school.
  • this approach promotes shrinkage of the school means $Z_i$ to the overall mean $\mu$

If you use $Z_i$ to represent individual characteristics, such as skill level, you lose these advantages. Such a model would be perfectly valid and might be technically a hierarchical model, but a rather atypical one. (Unless you are considering each student takes the test more than once, then it would be a usual hierarchical model, since the variable "students" groups the variable "test score")

Now, suppose students $i$ and $j$ go to the same school (that is, $Z_i=Z_j$). Their SAT scores are dependent: if we learn $i$ got a good score, the probability of $Z_i$ being a good school increases and the expected score for $j$ increases. The only way we can make their scores independent is conditioning in their school's mean score, which you denoted as $x_i\perp x_j|\beta$.

This model does not suppose observations are iid - the observations are neither independent now identically distributed, i they have different values of $Z$. There is no contradiction in this, not all statistical models need iid assumptions. Most do, but not all.

You can find more info about hierarchical models and their implementation in PyMC3 in this link Hope that was helpful!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.