Priors that do not become irrelevant with large sample sizes

This may be a weird question. My colleagues and I are working on a medical estimation problem, where relevant prior knowledge regarding plausible values of some physiological parameters exists. In addition, these parameters can be estimated using time series data, which often have some tens of thousands of samples. What often happens is that, due to model imperfections, MAP estimation converges towards implausible solutions (e.g., very small parameter values). The priors are essentially ignored because they become irrelevant given the large amount of available (and informative) measurements.

Now I am well aware that one way (probably the preferred one) to solve this problem is to improve the time series model and try to fix its imperfections. This is proving to be really hard, however, for a number of reasons.

Another (partial) remedy might be to somehow "fix" the influence of the prior such that it does not become irrelevant. This could be done, e.g., by choosing the prior's covariance as a function of the sample size or simply assigning the prior and the likelihood fixed weights when determining final parameter estimates. That got me wondering: is this "a thing"? Is there a name for doing something like this, i.e., a "fixed-influence prior"? Surely much more knowledgeable people than me have thought about this problem.

• I agree with Dave that it is a feature. When your MAP nevertheless produces "implausible" solutions, I would advocate considering the likelihood rather than the prior as the culprit. Think local maxima etc. If you still want to tweak the prior, the fictitious sample interpretation of conjugate priors might be a starting point, see e.g. stats.stackexchange.com/questions/155059/… Sep 13, 2022 at 12:13
• Note also that MAP inference is only debatably Bayesian. Sep 13, 2022 at 12:30
• But I think this is a great question; for example sklearn Lasso function automatically multiplies the provided regularization coefficient by the sample size, somewhat like what you describe. Sep 13, 2022 at 12:32
• A prior which sets the probabilities or densities of impossible parameter values to zero will lead to a posterior distribution with those values remaining zero. If those parameter values are merely improbable and the evidence points towards them then you start to move into the realms of Cromwell or Sherlock Holmes. Sep 13, 2022 at 12:38
• Wouldn't this amount to taking the outputs of the model and basically just pushing them into the region of measurements that are considered plausible? If I know that human body temperature is always between 13 °C and 47 °C, and I have a model that says that the patient's temperature is almost certainly approximately 200 °C, then applying a really strong prior that forces the estimate down to 47 °C isn't necessarily going to give me reasonable results. Sep 13, 2022 at 21:07

A prior distribution is your guess (hopefully a good guess, but still a guess).

Then you observe data and see what really happens.

When you have enough observations that contradict your original guess, it is reasonable to change your mind.

What you’re observing strikes me as a feature, not a bug, of Bayesian inference.

• In principle, I agree - the problem only arises in the context of significant model mismatch, I believe? In that case, while not completely useless, an ML estimate may be significantly biased (as it is in our case), and I would like my prior knowledge to not be completely overruled by that bias. Sep 13, 2022 at 12:10
• I do not mean to be stubborn, but I would take the same stance in this case as in my comment above - if the model is well-specified, then bias should be smallish in large samples (which is where the prior is overruled). If bias is still large in large samples, this, to me, points to misspecification, think omitted variable bias, so that it may be more promising to reconsider the specification of the model rather than to tweak the prior. Sep 13, 2022 at 14:31
• If you think maximum likelihood/MAP/modes of posterior distributions are risky, then apply a suitable loss function - you may end up with the mean of the posterior distribution as a point value (though that has its own issues, as it may be impossible) Sep 13, 2022 at 21:13
• (+1) I think part of the problem here is that the specified prior does not match OP's stated beliefs. If the beliefs are so strong that one is prepared to discount the posterior after an analysis involving tens of thousands of data points, then it implies that OP's actual prior beliefs are MUCH stronger than specified in the model prior.
– mkt
Sep 14, 2022 at 10:02
• @DikranMarsupial That is precisely correct. In particular, it is proving very challenging to find a sufficiently good noise model that captures all relevant ways in which the regression model is misspecified, leading to biased parameter estimates. Sep 14, 2022 at 13:15

I do agree with the previous answers, but if you really want to "fix" the influence of the prior, here are some ideas.

• If your prior is based on historical data, you can use a power prior [1] to control the relative influence of your prior on the posterior obtained with new data.

• Alternatively, you can also consider weighing the likelihood (power scaling) so that the relative influence of your prior is increased. However, if, for example, you have a Gaussian model, this would be equivalent to increasing the standard deviation of the Gaussian. So in the end, maybe you do need to change your model.

You may also be interesting in reading [2] which uses power scaling as a way to diagnose prior sensitivity.

[1] Ibrahim, J. G., Chen, M. H., Gwon, Y., & Chen, F. (2015). The power prior: Theory and applications. Statistics in Medicine, 34(28), 3724–3749. https://doi.org/10.1002/sim.6728

[2] Kallioinen, N., Paananen, T., Bürkner, P.-C., & Vehtari, A. (2021). Detecting and diagnosing prior and likelihood sensitivity with power-scaling. https://arxiv.org/abs/2107.14054v1

Have you considered that your expectation is simply wrong, perhaps because of publication bias?

Alternatively, if you are so confident in your beliefs that you're willing to discount the posterior after an analysis of tens of thousands of data points, it seems to me that your specified prior does not truly reflect the strength of your belief. You should probably specify a much stronger prior - if it's strong enough, the posterior wouldn't be dominated by the data.