Justification for use of non-conjugate priors? Google searches gives no results to this question and there is the opposite question in this site, which makes me think this has an intuitive response I am missing.
In most course notes and responses I find a great deal of advantages and conveniences for the use of conjugate priors. After all, we are dealing with subjective beliefs that hardly impose a specific form of complexity into its representation (and we have a great deal of flexibility of representation with the possibility of using mixtures of conjugate distributions). Why then use non-conjugate priors?
I usually see the example of replacing a gamma prior with a lognormal prior, but no rationale is given to why the pros outweigh the cons for doing so.
 A: The biggest weakness of conjugate priors is that (in certain cases) they cannot achieve the following two properties simultaneously:


*

*Proper

*Vague


A conjugate prior is equivalent to adding virtual points to your dataset.  This is what makes them computationally efficient.  But this can also make it impossible to have a vague proper prior, because it can happen that if you choose the virtual points before seeing the data, then there will always be a dataset far away from those points, such that the virtual points will unduly influence the posterior.  This happens because the posterior of an exponential family is a function of sufficient statistics, and sufficient statistics can have a breakdown point of 0, meaning it only takes a single outlier to exert arbitrary influence on the statistics.
To illustrate, suppose we want to estimate the mean parameter $m$ of a normal distribution.  A conjugate prior for $m$ must be a normal distribution.  It is impossible to choose a normal prior on $m$ that is both proper and vague.  Consider a game where you have to pick a proper normal prior, and then I get to pick a dataset.  No matter what prior you pick, I can pick a dataset (of any size) whose empirical mean is sufficiently far away from the mean of that prior, such that the posterior distribution of $m$ is unduly influenced by that prior.  This happens because the sufficient statistic of $m$ is the arithmetic average, which has breakdown point 0.  On the other hand, if you are allowed to use non-conjugate priors, then it is easy to choose a prior on $m$ that is both proper and vague (for example, a Cauchy distribution).   
The paper "Bayesian robustness modelling using regularly varying distributions" gives the mathematical arguments behind these statements.
