2
$\begingroup$

I am trying to understand what is and what is not considered Bayesian inference. Let say I am to estimate a parameter or a vector of parameters say $\theta$ and I have data on some features of the distribution say the first two moments $\hat{m_1}$ and $\hat{m_2}$. I think it is OK to use Bayes theorem to derive a sort of a posterior as follows

$p(\theta|\hat{m_1},\hat{m_2})\propto p(\hat{m_1},\hat{m_2}|\theta)p(\theta)$

where $p(\hat{m_1},\hat{m_2}|\theta)$ is the joint distribution of $\hat{m_1}$ and $\hat{m_1}$ conditional on $\theta$. Which one of these is not Bayesian inference?

  1. use the actual distribution of $p(\hat{m_1},\hat{m_2})$ in the above formula
  2. use the Asymptotic distribution of $p(\hat{m_1},\hat{m_2})$ in the above formula
  3. How about the following when I have a frequentist estimate of $\theta$ and its distribution?

$p(\theta|\hat{\theta})\propto p(\hat{\theta}|\theta)p(\theta)$

Addendum: Here is an example to clarify the question: Suppose I want to learn about the parameters of the distribution of income given my data and let's assume it follows a parametric distribution with parameter $\theta$ . Often the data on income distribution is available only in the form of some summary statistics (e.g. deciles or Lorenz ordinates). for simplicity let say only 2 summary statistics of $\hat{m_1}$ and $\hat{m_2}$ are available. It is possible to obtain either the actual or more often asymptotic distribution of these summary statistics as a function of unknown $\theta$. Let's denote this with $p(\hat{m_1},\hat{m_2}|\theta)$. (1) and (2) above asks whether this can be used like a likelihood and proceed with the Bayesian inference. I personally think this is a legitimate Bayesian analysis and I have seen papers doing such things.

Now if the above is legitimate Bayesian inference and the objective of Bayesian analysis is to incorporate prior information then I can use the above logic and easily make any frequentist estimator Bayesian without MCMC or any high power methods. I can calculate $\hat{\theta}$ (the frequentist estimate of $\theta$) and the frequentist analysis often gives me the (asymptotic) distribution of $\hat{\theta}$ as a function of $\theta$ which I denote with $p(\hat{\theta}|\theta)$. I can combine this (often a normal distribution) with prior $p(\theta)$ and do a simple Bayesian analysis. This does not seem right but what is wrong with it or what is that I am missing?

Addendum2: The latter may not as simple as I thought since the variance of the asymptotic normal distribution could depend on $\theta$ in a complicated way.

$\endgroup$
4
  • 1
    $\begingroup$ I believe this thread answers your question stats.stackexchange.com/questions/167051/who-are-the-bayesians doesn't it? TL;DR it is not about Bayes theorem. $\endgroup$
    – Tim
    May 10, 2022 at 12:55
  • $\begingroup$ Thanks for the link but it doesn't exactly answer my question. That link is about who can be called Bayesian whereas my question is whether the examples I gave make sense (or are interesting) in conventional Bayesian analysis. $\endgroup$
    – user41838
    May 10, 2022 at 16:18
  • $\begingroup$ Ok, but then how you want to use $p(\hat{m_1},\hat{m_2})$ and what it is? It is never used in your first equation. $\endgroup$
    – Tim
    May 10, 2022 at 17:26
  • $\begingroup$ $p(\hat{m_1},\hat{m_2}|\theta)$ is the joint distribution of $\hat{m_1}$ and $\hat{m_1}$ conditional on $\theta$ assuming this is available in (1) or its asymptotic distribution in (2). My own feeling is that (1) can lead to a reasonable Bayesian inference, (2) is still reasonable but it may be called approximate Bayesian inference. (3) it is not interesting Bayesian analysis, $\endgroup$
    – user41838
    May 11, 2022 at 1:41

1 Answer 1

0
$\begingroup$

Let me split the answer into two parts.

Moments in the likelihood considered as regular data

First, I'll make it more general, and instead of considering

$$ p(\theta|\hat{m_1},\hat{m_2})\propto p(\hat{m_1},\hat{m_2}|\theta)\,p(\theta) $$

I'll consider the general case

$$ p(\theta|X)\propto p(X|\theta) \,p(\theta) $$

where $X$ can be any data. If you can define the likelihood function in terms of the moments $\hat{m_1},\hat{m_2}$ as the data (same as data about $n$ i.i.d. Bernoulli trials can be simplified to considering it as a binomial random variable for the total number of successes), then it is the same. In many cases, however, it might be hard to come up with such a likelihood function.

  1. use the actual distribution of $p(\hat{m_1},\hat{m_2})$ in the above formula

This question is confusing. Are you talking about the likelihood $p(\hat{m_1},\hat{m_2}|\theta)$ or the unconditional distribution $p(\hat{m_1},\hat{m_2})$?

If it is the latter, you cannot put unconditional distribution in the place of the conditional distribution, they are different things. It is hard to discuss it as a general case because it is not clear what you mean here and how exactly it refers to the initial problem. Say that you are interested only in the mode of the posterior (maximum a posterior estimation), so you are just maximizing the posterior probability, $p(\hat{m_1},\hat{m_2})$ cannot be maximized with regard of $\theta$ because it is not a function of $\theta$.

If you meant conditional distribution then lucky you! Usually, when deciding on a statistical model, you need to somehow come up with what the likelihood function should be for the problem. If you know it, you don't have to make this step.

  1. use the Asymptotic distribution of $p(\hat{m_1},\hat{m_2})$ in the above formula

Same as above. Unusually we don't "know" the likelihood function, but make some assumptions.

  1. How about the following when I have a frequentist estimate of $\theta$ and its distribution?

First, in the frequentist approach, you would not have the estimate for the distribution of the parameter, because the parameter is not considered a random variable. You would have the point estimate. But answering the question, deciding on priors based on in-data statistics is called empirical Bayesian approach, it is controversial, but sometimes used.

Can we use moments in the likelihood function?

In some cases, it might be hard to come up with a likelihood function, but instead, the distribution can be characterized in other terms. In such cases, we can use likelihood-free inference and things like approximate Bayesian computation (ABC) (see also other threads for ). Your problem with using only the moments rather than the data might fall into this category. In ABC, the likelihood is replaced with a distance function that serves as a proxy for it.

Finally, "what is Bayesian inference?" is a variant of the Who Are The Bayesians? question that already got some nice answers. As you can learn from the linked thread, Bayesian statistics is not about using the Bayes theorem, but is more general than this. That is how ABC falls into this category, though it does not exactly use the Bayes theorem, just takes from the general idea behind it.

$\endgroup$
6
  • $\begingroup$ Thanks for the attempt to answer but I haven't fully got my answer yet. I have added some examples\explanations to clarify my question. $\endgroup$
    – user41838
    May 12, 2022 at 14:53
  • $\begingroup$ @user41838 my answer addresses exactly what you mentioned in the edit. The answer is abstract because you asked an abstract question. If you have a specific problem in mind, ask a specific question. $\endgroup$
    – Tim
    May 12, 2022 at 17:31
  • $\begingroup$ I believe you have answered number (1) and (2) but not (3). I still don't know from your answer that (3) is a legitimate Bayesian analysis. You say "First, in the frequentist approach, ..... estimate for the distribution of the parameter, because the parameter is not considered a random variable." I did not claim we get distribution of $\theta$ from a frequentist analysis, I said we can get distribution of $\hat{\theta}$. I don't also see the relevance of "answering the question, deciding on priors .... is called empirical Bayesian approach, it is controversial, but sometimes used". $\endgroup$
    – user41838
    May 13, 2022 at 0:18
  • $\begingroup$ @user41838 (3) is answered, if you create the priors based on estimates from the data, it's empirical Bayesian approach. $\endgroup$
    – Tim
    May 13, 2022 at 5:49
  • $\begingroup$ In which part of (3) i.e. $p(\theta|\hat{\theta})\propto p(\hat{\theta}|\theta)p(\theta)$ there is a prior that depends on data? The frequentist distribution of $\hat{\theta}$ i.e. $p(\hat{\theta}|\theta)$ is used as likelihood and prior $p(\theta)$ does not depend on data. $\endgroup$
    – user41838
    May 13, 2022 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.