Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now
24

This paper by Christian (Xi'an) Robert and George Casella provides a nice summary of the history of MCMC. From the paper (emphasis is mine). What can be reasonably seen as the first MCMC algorithm is what we now call the Metropolis algorithm, published by Metropolis et al. (1953). It emanates from the same group of scientists who produced the Monte Carlo ...


11

The excellent answer by knrumsey gives some history on the progression of important academic work in MCMC. One other aspect worth examining is the development of software to facilitate MCMC by the ordinary user. Statistical methods are often used mostly by specialists until they are implemented in software that allows the ordinary user to implement them ...


10

We often don’t care about the likelihood, just the value for which the likelihood is maximized. When you use the likelihood function to find a maximum likelihood estimator, you get the same point giving the maximum whether you include constants out front or not. Sure, that maximum value will be different, but that is not our concern. So let’s make it ...


7

I don't quite know your specific use case though I can provide you with a bunch of resources so that you can make your own decision: Michael Betancourt's case study on sparse regressions: https://betanalpha.github.io/assets/case_studies/bayes_sparse_regression.html That is a particularly great resource because he shows the effect of different priors on the ...


6

It is a marginal distribution. It is called a density, because it refers to continuous random variable, so it has a probability density function. It is called "marginal" because we "marginalize", i.e. integrate out the other variables that it is conditional on $$ p_X(x) = \int_y \; p_{X|Y}(x|y) \; p_Y(y) \; dy $$ In case of Bayes theorem $$ P(A|X) = \...


6

The sample space of an experiment is the set of all possible outcomes of that experiment. For example, if you toss a die two times, the sample space of this experiment would be $$ \Omega = \{ (1, 1), (1, 2), (1, 3), ..., (1, 6), (2, 1), (2, 2), ..., (6, 1), (6, 2), ... (6, 6) \} $$ In the example with guessing the numbers, the experiment is to choose 4 ...


6

You can check the following paper by Sarah Van Erp et al (2019), who discuss different shrinkage priors (below you can see table from their paper). Those priors vary greatly in their shapes, and so amount of shrinkage they provide. Besides discussing pros and cons of those priors, the authors describe a simulation study, where they compare performance of ...


5

As I explained earlier this week in my introductory lecture to Monte Carlo methods, the founding principle of such numerical methods is the Law of Large Numbers, or the stabilisation of empirical frequencies to their expectations. Markov chain Monte Carlo algorithms are a special case of methods implementing the Monte Carlo principle, in that Markov chains ...


5

The first question is whether you really want significance testing. I'm not sure you do, but, if you do, then, sure, the results of Bayesian analysis might give more information, if you supply more information to them by choosing an informative prior. But TANSTAAFL (there ain't no such thing as a free lunch). What if your prior is wrong? From what I have ...


5

A divergent transition in Stan tells you that the region of the posterior distribution around that divergent transition is geometrically difficult to explore. For example here is a quote from the manual: The primary cause of divergent transitions in Euclidean HMC (other than bugs in the code) is highly varying posterior curvature, for which small step ...


4

Yes, the likelihood is the likelihood. You sometimes see likelihood defined only up to a multiplicative constant (as Fisher did) but that doesn't harm either of those applications if you are consistent in how you deal with it. Unfortunately, by asking a yes-or-no question to which the answer is "yes" there's not much more to say. If the answer had been no, ...


4

Frequentist view 👀 In one sense, we can think of both regularizations as "shrinking the weights"; L2 minimizes the Euclidean norm of the weights, while L1 minimizes the Manhattan norm. Following this line of thinking, we can reason that the equipotentials of L1 and L2 are spherical and diamond-shaped respectively, so L1 is more likely to lead to sparse ...


4

I would recommend leaving words like "significance" out of such descriptions. Be specific instead. For your example sentence (evidently intended for a Discussion section), try something like: "People given decaffeinated coffee and told truthfully that it was decaffeinated experienced a reduction of X (95% credible interval, Y-Z) in caffeine withdrawal ...


4

The whole point of doing Bayesian Inference, is that you stipulate your data was generated by a model with unknown parameters. An example of this would be "the samples were all drawn independently from the same normal distribution, but we don't know the mean or variance of that distribution" We then try to use the data to come up with beliefs about the ...


4

You can do either you can reject the impossible points and simply stay at the current location: https://xianblog.wordpress.com/2011/07/05/bounded-target-support/ Do not generate a new proposal if you get an impossible move. You just want to stay where you are rather than making an impossible move. Or you can make the whole thing into a ring and simply ...


3

Let's say you updated your prior with incoming data as you said, and obtained a posterior. Let's call it as $f_P(p)$. And call the number of heads in the next $n$ flips as $X$. Then, $P(X=k|p)$ is in binomial form. In order to get a final expression for $P(X=k)$, you need to integrate wrt posterior: $$P(X=k)=\int_{p\in\mathcal{P}}P(X=k|p)f_P(p)dp={n\choose k}...


3

Given $d_1$, the Bayes theorem reads, $$P(x\ |\ d_1)=\frac{P(d_1\ |\ x)P(x)}{P(d_1)}$$ now you get $d_2$, in which case we can write $$P(x\ |\ d_1, d_2)=\frac{P(d_1, d_2\ |\ x)P(x)}{P(d_1, d_2)}$$ taking into account that $P(d_1, d_2\ |\ x) = P(d_2 \ | \ d_1, \ x)P(d_1 \ | \ x)$ and that $P(d_1, d_2) = P(d_2 \ | \ d_1)P(d_1)$, by substituing we obtain ...


2

It is useful to point out that Andrew Stuart has a webpage, it includes a link titled: "Lecture Notes: ICM2014" (.PDF) which downloads his lecture: "Uncertainty Quantification in Bayesian Inversion". His webpage also lists a link to one of his papers: "The Bayesian Approach To Inverse Problems" (July 2 2015), by Masoumeh Dashti and Andrew M. Stuart. What ...


2

Your problem is studied in the field of Bayesian experimental design. The question that experimental design faces is, which measure should I make in order to maximize the information gain? Or put in another way, which measure of $E$ is going to allow me to discriminate the most between different values of the parameter $X$? A possible way to measure the ...


2

There may be something to the idea that computing the posterior in one cell should affect the computation for the next cell. But then, why not re-do the computation for the first cell? Also, disregarded in the statement of this problem is that results for one tower might affect results for nearby towers more than for distant ones. My guess is that you are ...


2

Normal distribution, along with binomial distribution, are the two most popular examples of distributions used for introducing Bayesian statistics. You can found them described in any handbook on Bayesian statistics, for example the two references mentioned by Kevin Murphy in the refereed paper [Bis06] C. Bishop. Pattern recognition and machine ...


2

First, to have a posterior distribution for $\theta$, $\theta$ must be (modeled as) a random variable. For the likelihood function that is not necessary. So this is deeper than the comment (by @gazza89) saying The likelihood is a pdf, it's just normalised w.r.t all possible data outcomes, and the posterior is a pdf, but it's normalised w.r.t all ...


2

If the threshold is zero and you integrate $\psi$ out of the posterior, the second model becomes a probit model. Gibbs samplers actually estimate the $\psi$ parameters, but that is not a good idea (if you can avoid it) for Hamiltonian Monte Carlo samplers such as Stan. A Stan program with a probit log-likelihood would be data { int<lower = 0> N; ...


2

Whether or not your approach is legitimate depends in large part about how you describe your approach when publishing or presenting your results. If you are completely open about your approach and your process then the reader is able to judge your approach for themselves. I state this because statistics so often involves subjective choices that have no clear ...


2

The literature you linked to and other references employ both a Bayesian and a non-Bayesian approach to solving the problem. In the lecture slides you can see that on slide 10 they start with a likelihood ratio test and this is not Bayesian. But on slide 25 (for example 2) they have switched to a Bayesian analysis. So the theory itself does not specify ...


1

You want something called a Posterior Predictive Distribution. Your Bayesian model is a generative model. In essence, this means that if you knew the probability of the coin landing on heads (call it $\theta$), then you could generate predictions from your likelihood. Using the posterior to draw multiple $\theta$ and then passing them to the likelihood ...


1

Let $\Theta$ be the RV for the probability of heads. Let`s denote the posterior with $f(\theta|X)$. You now want $P(X_1=x_1, ..., X_n=x_n)$. where $X_i$ is the RV for outcome of the $i-th$ trial and $x_i \in \{0,1\}$. Then $P(X_1=x_1, ..., X_n=x_n) = \int_\Theta \Pi_{i=1}^n\theta^{x_i}(1-\theta)^{1-x_i} f(\theta|X)d\theta$ This means that you "take a ...


1

If the question is about simulation, rather than Bayesian statistics, since the mcmc tag is (unnecessarily) used, the generation of more variates than needed is quite common in random variate generations and goes under different names, like auxiliary variable models, latent variable models, demarginalisation, &tc. For instance, the generic accept-reject ...


1

Let's first distinguish the prior $\pi(p)$ from the likelihood function $f(x|p)$ for clarity. Let the marginal likelihood (equivalently evidence, normalizing constant) $$f(x) = \int f(x|p) \pi(p)dp.$$ Before we do any drawing, let's consider what the model tells us. $$p\sim\text{Beta}(\alpha,\beta)\\ y \sim \text{Binom}(n, p)$$ What this says is, we posit ...


1

You sum, or integrate, over the joint distribution, in this case of whether the your dog is hungry or not, and whether you feed your dog, or not. Since $P(A, B) = P(A \vert B) P(B)$, you find that $$P(H) = P(H \vert F)P(F) + P(H\vert NF)P(NF) = \frac{1}{10}\frac{6}{10} + \frac{7}{10}\frac{4}{10} = \frac{17}{50}.$$ By total probability we also have ...


Only top voted, non community-wiki answers of a minimum length are eligible