Justification for use of non-conjugate priors?

Google searches gives no results to this question and there is the opposite question in this site, which makes me think this has an intuitive response I am missing.

In most course notes and responses I find a great deal of advantages and conveniences for the use of conjugate priors. After all, we are dealing with subjective beliefs that hardly impose a specific form of complexity into its representation (and we have a great deal of flexibility of representation with the possibility of using mixtures of conjugate distributions). Why then use non-conjugate priors?

I usually see the example of replacing a gamma prior with a lognormal prior, but no rationale is given to why the pros outweigh the cons for doing so.

• Conjugate priors can make it easier to find posterior distributions, but with modern computational methods it is not necessary to use conjugate priors. The choice of an appropriate prior has to do with prior knowledge and opinion. Sometimes it is easier to translate that information into a non-conjugate prior. Commented May 24, 2020 at 5:18
• For some models there might not be conjugate priors. Also, conjugate priors sometimes lead to linear updating. You might want a prior that is somehow rejected if the data strongly disagrees with the prior, that might need heavier tails. Commented May 25, 2020 at 2:26
• @BruceET what examples are there where our internal beliefs are not able to be represented by conjugate-prior? Trying to grasp the concrete and specific shortcomings that conjugate priors could have.
– Kuku
Commented May 25, 2020 at 13:13
• Not complicated. Conjugate priors for binomial data are beta family. But for someone not familiar with shapes of beta distributions it may not seem natural to choose a beta prior, Commented May 25, 2020 at 17:31
• See this paper by Diaconis & Ylvisaker. I will see if I can write an answer along those lines. Commented May 26, 2020 at 22:44

To illustrate, suppose we want to estimate the mean parameter $$m$$ of a normal distribution. A conjugate prior for $$m$$ must be a normal distribution. It is impossible to choose a normal prior on $$m$$ that is both proper and vague. Consider a game where you have to pick a proper normal prior, and then I get to pick a dataset. No matter what prior you pick, I can pick a dataset (of any size) whose empirical mean is sufficiently far away from the mean of that prior, such that the posterior distribution of $$m$$ is unduly influenced by that prior. This happens because the sufficient statistic of $$m$$ is the arithmetic average, which has breakdown point 0. On the other hand, if you are allowed to use non-conjugate priors, then it is easy to choose a prior on $$m$$ that is both proper and vague (for example, a Cauchy distribution).