conjugate prior: is ever the best choice?

Question

I'm reading about the conjugate prior of classic probability distributions (e.g. beta distribution for binomial distribution); it's explained just as "algebric trick" to have easier calculation in computing $$ Pr(x|\theta)=\frac{Pr(\theta|x)Pr(x)}{Pr(\theta)} $$ my questions are:

1. is conjugate prior the only possible choice for $Pr(\theta)$?
2. If not, conjugate is ever the best choice?
3. In parameter estimation methods (e.g. Maximum Likelihood Estimation), if I choice, for example, two different prior distribution $Pr^A(\theta)$ (e.g. the conjugate) and $Pr^B(\theta)$ (e.g. another distribution different from the conjugate) and compute MLE with both configurations, will I obtain the same solutions? If not, why? Parameter estimation methods should find best solution (e.g. MLE would maximize), so it should give me the best solution (i.e. the best value of $\theta$) independently from the shape of prior that I choice...Is right?

Answer 1

As you said, conjugate priors make things easier and an additional nice property is that when you refresh the model using the posterior as the new prior things are nice and consistent. For instance you can refresh a specific analysis with new data every month, substitute the prior with the old posterior see how the parameters of the posterior distribution evolve throughout the time which is nice.

That being said, it is not the only option and in many cases it's not the one having the best performance. Obviously there is no hard rule and it depends on the case. Different prior distributions yield different results, especially when don't have that many data

Answer 2

How do you define optimal?

Typically, the conjugate priors won't be the best choice, but they may be reasonable choices. Typically conjugate priors are chosen for computational efficiency. Jeffrey's priors can often be shown to have many favorable "optimal" properties.

In terms of MLE you can think of Bayesian priors as being similar to penalized likelihood functions in frequentist framework. So to answer your question 3: No, consider the lasso model (L1 penalized linear regression) vs ridge (L2 penalized linear regression) the MLE to these models is not the same.