35 votes
Accepted

Posterior distribution and MCMC

If this was not a clear conflict of interest, I would suggest you invest more time on the topic of MCMC algorithm and read a whole book rather than a few (6?) articles that can only provide a partial ...
Xi'an's user avatar
  • 106k
29 votes

Why is it necessary to sample from the posterior distribution if we already KNOW the posterior distribution?

This question has likely been considered already on this forum. When you state that you "have the posterior distribution", what exactly do you mean? "Having" an available$-$in the ...
Xi'an's user avatar
  • 106k
24 votes

Alternatives to Bayesian statistics when distributions are unknown

If the distribution of the data is unknown, the Bayesian way of handling this uncertainty is to put a prior on it. There exists a huge literature on Bayesian non-parametrics, including the ...
Xi'an's user avatar
  • 106k
23 votes
Accepted

Why is the normalisation constant in Bayesian not a marginal distribution

$p(D)$ is a constant with respect to the variable $\theta$, not with respect to the variable $D$. Think of $D$ as being some data given in the problem and $\theta$ as the parameter to be estimated ...
Eric Perkerson's user avatar
22 votes

Effective Sample Size for posterior inference from MCMC sampling

The question you are asking is different from "convergence diagnostics". Lets say you have run all convergence diagnostics(choose your favorite(s)), and now are ready to start sampling from the ...
Greenparker's user avatar
  • 15.6k
20 votes

What is the difference between posterior and posterior predictive distribution?

They refer to distributions of two different things. The posterior distribution refers to the distribution of the parameter, while the predictive posterior distribution (PPD) refers to the ...
CPT's user avatar
  • 461
19 votes
Accepted

What is/are the implicit priors in frequentist statistics?

In frequentist decision theory, there exist complete class results that characterise admissible procedures as Bayes procedures or as limits of Bayes procedures. For instance, Stein necessary and ...
Xi'an's user avatar
  • 106k
18 votes
Accepted

Alternatives to Bayesian statistics when distributions are unknown

From what you’re saying is that you want something Bayesian, but you can’t define the likelihood. For such cases there is approximate Bayesian computation (see abc), where in place of likelihood you ...
Tim's user avatar
  • 138k
15 votes
Accepted

Why does thinning work in Bayesian inference?

Thinning has nothing to do with Bayesian inference, but everything to do with computer-based pseudo-random simulation. The whole point in generating a Markov chain $(\theta_t)$ via MCMC algorithms is ...
Xi'an's user avatar
  • 106k
14 votes

What is/are the implicit priors in frequentist statistics?

@Xi'an's answer is more complete. But since you also asked for a pithy take-away, here's one. (The concepts I mention are not exactly the same as the admissibility setting above.) Frequentists often (...
civilstat's user avatar
  • 3,814
13 votes

Multivariate normal posterior

With the distributions on our random vectors: $\mathbf x_i | \mathbf \mu \sim N(\mu , \mathbf \Sigma)$ $\mathbf \mu \sim N(\mathbf \mu_0, \mathbf \Sigma_0)$ By Bayes's rule the posterior ...
conjectures's user avatar
  • 4,226
13 votes

Posterior distribution and MCMC

How can you "draw samples from the posterior distribution" without first knowing the properties of said distribution? In Bayesian analysis we usually know that the posterior distribution is ...
Ben's user avatar
  • 125k
13 votes
Accepted

Are MCMC based methods appropriate when Maximum a-posteriori estimation is available?

No need to use MCMC in this case: Markov Chain Monte-Carlo (MCMC) is a method used to generate values from a distribution. It produces a Markov chain of auto-correlated values with stationary ...
Ben's user avatar
  • 125k
13 votes
Accepted

Posterior Predictive Distribution as Expectation of Likelihood

$\newcommand{\y}{\mathbf y}$We have $$ E_{\theta|\y}\left[f(\theta)\right] = \int f(\theta) p(\theta | \y)\,\text d\theta $$ just by definition of expectation (and you could cite the LOTUS as well), ...
jld's user avatar
  • 20.2k
13 votes
Accepted

In Bayesian inference, why are some terms dropped from the posterior predictive?

This is based on the assumption that $x$ is conditionally independent of $D$, given $\theta$. This is a reasonable assumption in many cases, because all it says is that the training and testing data ($...
Ruben van Bergen's user avatar
13 votes

Why is the normalisation constant in Bayesian not a marginal distribution

The normalising constant in the posterior is the marginal density of the sample in the Bayesian model. When writing the posterior density as $$p(\theta |D) = \frac{\overbrace{p(D|\theta)}^\text{...
Xi'an's user avatar
  • 106k
13 votes
Accepted

Uniform posterior on bounded space vs unbounded space

It is not possible to have a flat (uniform) probability distribution on an unbounded space, so in particular it's not possible to have a flat posterior distribution. If you had a uniform probability ...
Thomas Lumley's user avatar
13 votes
Accepted

Posterior distribution is impossible depending on which prior hyperparameters are used?

Let's work through the steps. To begin with we have \begin{equation} p(\delta|\beta)\,p(\beta|\alpha)\,p(\alpha) , \end{equation} where \begin{align} p(\delta|\beta) &= \textsf{Bernoulli}(\delta|\...
mef's user avatar
  • 3,226
13 votes

Can a posterior probability be larger than 1 when more samples are available?

As a rule, the probability of an event is never greater than one (the pdf can take any arbitrarily high value (nonnegative), but the probability measure of any event, i.e. any member of the $\sigma$-...
frank's user avatar
  • 10.8k
12 votes

Can we always pull a joint posterior apart?

No, it is not. In order for that to be true, $A$ and $B$ should be conditionally independent given $\theta$.
gunes's user avatar
  • 57.3k
11 votes

Why is it necessary to sample from the posterior distribution if we already KNOW the posterior distribution?

Yes you might have an analytical posterior distribution. But the core of Bayesian analysis is to marginalize over the posterior distribution of parameters so that you get a better prediction result ...
Karlsson Yu's user avatar
11 votes
Accepted

Posterior computation for Laplace distribution

The likelihood for $n$ iid observations looks like: $ f(x_1,...x_n|\lambda,\mu) \propto \frac{1}{\lambda^n} exp(-\frac{1}{\lambda}\sum_{i=1}^n|x_i-\mu|)$ Hence a conjugate prior for $\lambda$ with $\...
conjectures's user avatar
  • 4,226
11 votes
Accepted

MAP estimation as regularisation of MLE

Maximum likelihood method aims at finding model parameters that best match some data: $$ \theta_{ML}=\mathrm{argmax}_\theta \,p(x|y,\theta) $$ Maximum likelihood does not use any prior knowledge ...
Jan Kukacka's user avatar
  • 11.4k
10 votes

Where is wrong with my formulation of estimating the probability of a biased coin?

I think part of the problem is that your notation is, to put it nicely, getting in your way. The symbol $\theta$ is usually used to denote the unknown probability of a head, but here you've written $p(...
Sycorax's user avatar
  • 91k
10 votes

Are MCMC based methods appropriate when Maximum a-posteriori estimation is available?

It is unclear to me what you call an analytical posterior $\pi(\theta)$ and hence why this analyticity should preclude one from using MCMC. Even for a posterior distribution that is available in ...
Xi'an's user avatar
  • 106k
10 votes
Accepted

Under what circumstances can an improper prior be used in bayesian analysis?

The textbook definition is that an improper prior is a prior probability density that is not integrable, i.e. it does not have a finite integral. A CDF cannot be higher than 1 hence the name "...
PaulG's user avatar
  • 1,267
10 votes
Accepted

Proving that a function is always increasing

By the law of iterative expectations and conditional independence, for any $n \in \mathbb{N}$ and $u_i \in \{0, 1\}$, $i = 1, \ldots, n$, we have \begin{align} & P(X_1 = u_1, \ldots X_n = u_n) = ...
Zhanxiong's user avatar
  • 18.8k
9 votes
Accepted

Bayesian inference: numerically sampling from the posterior predictive

If you can simulate values from $P(x_{\text{new}}|\theta)$, you can simply use your $N$ samples from posterior predictive and generate $x_{\text{new},i}$ for each posterior sample from this model to ...
lbelzile's user avatar
  • 484
9 votes

Use of prior and posterior predictive distributions?

Some uses of the posterior predictive: Simulating future data based on your model assumptions and data observed to this point. This is useful for predictions, forecasting, etc. Model checking via ...
Sean Easter's user avatar
  • 8,814
9 votes
Accepted

Why do independent priors for two random variables not result in an independent joint posterior distribution?

When you have an independent prior on $X$ and $Y$, then the posterior might not factor into $X$ and $Y$ pieces just because the likelihood doesn't factor into $X$ and $Y$ pieces. It's easy to see ...
Taylor's user avatar
  • 20.6k

Only top scored, non community-wiki answers of a minimum length are eligible