Weighted likelihood in Bayes' rule

Question

This question is about whether we can treat a weighted likelihood as a likelihood inside Bayes' rule.

Let $\theta$ be a parameter and let $x$ be some observed random variable. Given a prior over $\theta$ and a likelihood function of observing $x$ under $\theta$ (both satisfying certain regularity), we may evaluate the posterior probability density over $\theta$ evaluated at $\theta_0$. Bayes' rule may be expressed as $$p(\theta_0 \mid x) \propto p(x \mid \theta_0) p(\theta_0),$$ where $\propto$ hides a certain normalising constant $p(x) = \int p(x \mid \theta) p(\theta) \, d\theta$. The normalising constant does not depend on $\theta_0$, and so for some purposes can be ignored. For example, calculation of the MAP $\theta^\ast = \text{argmax}_{\theta_0} p(\theta_0 \mid x)$ does not require knowledge of the normalising constant.

Let $w_i>0$ denote some weight and define the weighted likelihood $q_i(x_i \mid \theta_0) = p(x_i \mid\theta_0)^{w_i}$. Define the "weighted posterior" $$ q(\theta_0 \mid x) \propto \prod_{i=1}^n q_i(x_i \mid \theta_0)p(\theta_0).$$

Does the weighted posterior have an interpretation as a true posterior, after appropriate redefinition of the likelihood? Can we also ignore the normalising constant for the purposes of MAP estimation? What about for other purposes e.g. sampling from the "posterior"?

Answer 1

While this is beyond statistics, exponential weights can be added to both the likelihood and the prior to introduce (or fix) certain types of biases. $$ P(H|D) \propto P(D|H)^\beta P(H)^\alpha $$ where for $0 < \beta < 1$ the likelihood $P(D|H)$ is flatter than in "original" Bayesian inference, while for $\beta > 1$ it is sharper. These models are common in behavioral sciences to explain irrational deviations from the idealized Bayesian decision-maker. There is a great paper by Matsumori et al. (2018) putting this into context. Here is a figure of theirs explaining the effect of the exponential biases $\alpha$ and $\beta$:

Related is the idea of power priors for historical data ("old likelihood").

It is also related to the idea of tempered likelihoods and cold posteriors in Bayesian neural networks.

Answer 2

Here's my very personal take on it.

1. I have seen this proposal of weighting the likelihood several times over the years. It's a fairly natural way to extend Bayesian inference, and it comes up from time to time.

2. I personally hate it. The words are strong, but they match my opinion.

3. To me, it comes down to why we do statistics. We are trying to provide a justified analysis of a dataset. Bayesian analysis has its flaws and limits, but it also has a clear reasoning:

   • We propose a probabilistic model of the data, with a latent unknown parameter
   • We apply Bayes' theorem and compute the posterior

   This is a logically consistent (but criticizable) approach to statistics.

4. Contrast this to a weighted-likelihood method: what the hell is the underlying justification for that proposal?

5. Contrast this to something like robust statistical methods (a la Huber, Ronchetti): here we have a clear reason for down-weighting the likelihood in some cases: a frequentist analysis shows that it improves the properties of the estimator.

6. However, note that the resulting procedure is something that processes the data and gives an estimator. If that is all we care about, then sure, what the hell, let's do it. But such rudderless data analysis is, in my opinion, bad and worse than following a principled approach to statistics which focuses on why we do things. In the specific case of these approaches, I think they are worse than robust statistics.