How concerned should I be about the appropriateness of my prior?

Question

As I understand it, selecting a prior provides something of a starting point for your analysis. From there, the distribution is shaped by the observed data. Obviously, the more data you observe, the more discrepancy is possible between the prior and posterior distributions (especially if the selected prior is inappropriate). As a result, it would seem to make sense that, for some large n, the selection of a prior is essentially irrelevant because the observed data will overwhelm the prior. Is this, in fact, the case? If so, does this actually occur in practice (or does that value of n need to be so ridiculously large that the point is purely theoretical)?

The underlying problem that I’m facing is that if I have m data points and I’m concerned about the appropriateness of my prior, what are some tools at my disposal to determine if my concern is legitimate?

Note: I realize that this question is very theoretical and that a concrete answer isn’t really possible (I’m sure a lot of this depends on the types of distributions, how inappropriate the prior is, etc.), so I’m worried that this might violate the condition that questions must be “practical, answerable questions based on actual problems you face.” If this is the case, please let me know. I’m new to the site and don’t really have a firm grasp on the etiquette yet…

Answer 1

It's always possible to create a prior that will overwhelm your data, no matter how many observations you have. However, for any fixed prior, as the number of observations grows, the influence of the prior shrinks (except for the 0-mass case that Macro pointed out in his comment).

For some prior distributions there's a concept of "prior sample size": if your prior sample size is $n_p$ and you have $n$ observations, then the posterior is in some sense a weighted average of the prior and data, weighted with $n_p$ and $n$ respectively. The easiest place to see this is when the Beta distribution is used as a prior for the Binomial distribution, where the prior sample size is $\alpha+\beta$. If I use a $\operatorname{Beta}(4,1)$ prior, that's sort of like saying that I believe my prior information is as good as 5 observations, and I expect success 80% of the time. If I then observe 5 data points (say 3 successes, 2 failures) my posterior will then be $\operatorname{Beta}(7,3)$--now my posterior is worth 10 observations (5 prior + 5 data), with a mean of .7. The prior is still pretty strongly weighted here. But if I observe 500 observations then my prior is basically irrelevant, because my data sample size is 100 times as large as my prior sample size.

On the other hand, I could use a $\operatorname{Beta}(8000,2000)$ prior. In this case, even if I observed 5000 data points, my posterior is still mostly determined by my prior.

If you're in a case where it's easy to calculate this sort of "prior sample size" (which also includes common models such as Normal-Normal, InverseGamma-Normal, and Gamma-Poisson), then this can give you an idea of how influential your prior is relative to your data. Otherwise I try to err on the side of diffuse priors, on the basis that it's (usually) better to overestimate your posterior uncertainty than to underestimate it.