The biggest weakness of conjugate priors is that (in certain cases) they cannot achieve the following two properties simultaneously:
- Proper
- Vague
A conjugate prior is equivalent to adding virtual points to your dataset. This is what makes them computationally efficient. But this can also make it impossible to have a vague proper prior, because it can happen that if you choose the virtual points before seeing the data, then there will always be a dataset far away from those points, such that the virtual points will unduly influence the posterior. This happens because the posterior of an exponential family is a function of sufficient statistics, and sufficient statistics can have a breakdown point of 0, meaning it only takes a single outlier to exert arbitrary influence on the statistics.
To illustrate, suppose we want to estimate the mean parameter $m$ of a normal distribution. A conjugate prior for $m$ must be a normal distribution. It is impossible to choose a normal prior on $m$ that is both proper and vague. Consider a game where you have to pick a proper normal prior, and then I get to pick a dataset. No matter what prior you pick, I can pick a dataset (of any size) whose empirical mean is sufficiently far away from the mean of that prior, such that the posterior distribution of $m$ is unduly influenced by that prior. This happens because the sufficient statistic of $m$ is the arithmetic average, which has breakdown point 0. On the other hand, if you are allowed to use non-conjugate priors, then it is easy to choose a prior on $m$ that is both proper and vague (for example, a Cauchy distribution).
The paper "Bayesian robustness modelling using regularly varying distributions" gives the mathematical arguments behind these statements.