My knowledge of probabilistic inference is severely limited, so coming from a Computer Science background I'm trying to understand what makes probabilistic inference difficult to implement in a parallel/distributed manner.
For MCMC at least I know there has been a lot of work in parallel and distributed implementations with recent work from the labs of Ryan Adams (FlyMC, Predictive Prefetching) and Erik Xing. The approaches also seem to differ in whether they take an asymptotically exact approach, or an exact approach.
Probabilistic programming languages like Stan use parallelism but only across chains (i.e. run one chain per process). For example in this (admittedly old) answer by one of the core devs of Stan he mentions:
There is no explicitly parallel code in Stan or rstan but neither is there any code that prevents the binary from being executed by several processes simultaneously.
So this is my naive question: What makes it difficult to do probabilistic inference in a distributed/parallel manner?
Would it be possible to do probabilist programming in a distributed setting for example, or are there core limitations in the nature of 'universal inference engines' that make them impossible/inefficient to distribute?