31

I am a Bayesian, but I find these kinds of criticisms against "frequentists" to be overstated and unfair. Both Bayesians and classical statisticians accept all the same mathematical results to be true, so there is really no dispute here about the properties of the various estimators. Even if you are a Bayesian, it is clearly true that the sample mean is no ...


26

This paper by Christian (Xi'an) Robert and George Casella provides a nice summary of the history of MCMC. From the paper (emphasis is mine). What can be reasonably seen as the first MCMC algorithm is what we now call the Metropolis algorithm, published by Metropolis et al. (1953). It emanates from the same group of scientists who produced the Monte Carlo ...


23

On the face of it your assumptions are inconsistent in that you think more people will default than have smartphones but you also think all defaulters have smartphones. Part of the problem is that some of your assumptions are for users of your app and part for the whole population and you treat these as being for the same group If instead you just ...


21

No, $\bar x$ has its own sampling distribution. Take, for example, the variances of $\bar x$ and $x_i$, in which the former is always lower ($\leq$) than the latter, which means $\bar x$ is not sampled from $p_\theta(x)$.


21

Good examples so far but consider $$X_i \sim Bernoulli(.5)$$ In that case the distribution of the data will only have support on 0 and 1. But the sample mean will have an ever decreasing probability of taking a value of 0 or 1 as the sample size gets larger and larger. That alone should show that the mean isn't being sampled from the original distribution....


20

One precise formulation of Bayes' Theorem is the following, taken verbatim from Schervish's Theory of Statistics (1995). The conditional distribution of $\Theta$ given $X=x$ is called the posterior distribution of $\Theta$. The next theorem shows us how to calculate the posterior distribution of a parameter in the case in which there is a measure $\nu$ ...


19

The Oxford English Dictionary defines "conjugate" as an adjective meaning "joined together, esp. in a pair, coupled; connected, related." It's not a huge stretch to imagine that a conjugate prior has a special and strong connection to its posterior. It's used in a similar sense in chemistry (conjugate acid/base; conjugate solution), ...


17

This is a good question. I'm going to use a simple example to illustrate my approach. Suppose I am working with someone who needs to provide me priors on the mean and the variance for a gaussian likelihood. Something like $$ y \sim \mathcal{N}(\mu, \sigma^2) $$ The question is: "What are this person's priors on $\mu$ and $\sigma^2$?" For the mean I ...


16

Overview quick remarks The model with three points does make a better fit. The fit with three points is only slightly better. The model with only one point is not very bad. The difference in loocv score may indicate that the model with more points is a significant/probable/likely improvement, but the effect size is only small. Even if the three points model ...


15

$p(D)$ is a constant with respect to the variable $\theta$, not with respect to the variable $D$. Think of $D$ as being some data given in the problem and $\theta$ as the parameter to be estimated from the data. In this example, $\theta$ is variable because we do not know the value of the parameter to be estimated, but the data $D$ is fixed. $p(D)$ gives the ...


14

I'd recommend looking at this answer from @Paul for guidance on so-called "random effects" and hierarchical models. In particular, this quote is on point: Random effects are estimated with partial pooling, while fixed effects are not. Partial pooling means that, if you have few data points in a group, the group's effect estimate will be based ...


13

No, it is only valid in cases as the Cauchy distribution, the means of samples of the Cauchy follow the same Cauchy dstribution.


13

It is not possible to have a flat (uniform) probability distribution on an unbounded space, so in particular it's not possible to have a flat posterior distribution. If you had a uniform probability density on the entire real line, you would need a function $f(x)$ that integrated to 1 (to be a probability density) but was constant. That's not possible: any ...


12

The excellent answer by knrumsey gives some history on the progression of important academic work in MCMC. One other aspect worth examining is the development of software to facilitate MCMC by the ordinary user. Statistical methods are often used mostly by specialists until they are implemented in software that allows the ordinary user to implement them ...


12

You have an inconsistent set of assumptions. Like saying 80 % of the population of the world like soccer. 100 % of all the people, who like soccer, like also tennis, which implies at least 80 % of the population like tennis. But then you say, 50 % of the population like tennis...! In order to derive $P(A)$, you could specify first $P(A\vert B^c)$ and then ...


12

Thinning has nothing to do with Bayesian inference, but everything to do with computer-based pseudo-random simulation. The whole point in generating a Markov chain $(\theta_t)$ via MCMC algorithms is to achieve more easily simulations from the posterior distribution, $\pi(\cdot)$. However, the penalty for doing so is creating correlation between the ...


12

As an even more pathological example, consider a sample from the distribution which is uniform on the union of $[0,1]$ and $[3,4]$. As the sample size increases, the mean will tend to 2 which isn't even in the support of the distribution. Another similar example is the uniform distribution on the boundary of the unit sphere (in any number of dimensions)


11

We often don’t care about the likelihood, just the value for which the likelihood is maximized. When you use the likelihood function to find a maximum likelihood estimator, you get the same point giving the maximum whether you include constants out front or not. Sure, that maximum value will be different, but that is not our concern. So let’s make it ...


11

It's worth noting that there is nothing that prevents Frequentist analysis from saying "Conditional on none of your data being censored, $\hat \mu$ is equal to $\bar x$ and will be unbiased. Conditional on some of your data being censored, the MLE estimator $\hat \mu$ is no longer equal to $\bar x$ and has some bias". Of course, marginalizing over ...


10

...example where, with an improper prior, the Bayesian estimator equals the maximum likelihood estimator... There is a fundamental issue with this property, namely that it depends on the parameterisation of the sampling model. Indeed, if $$\hat\theta^\text{MAP}=\arg\max_\theta L(\theta|x)\pi(\theta)=\arg\max_\theta L(\theta|x)=\hat\theta^\text{MLE}\tag{1}$...


10

It is a machine learning algorithm, it doesn't have to belong to either of those categories. Frequentist and Bayesian statistics is the distinction based on how probabilities are interpreted. Machine Learning algorithms are about finding patterns in the data and making predictions based on the learned patterns. Frequentist vs Bayesian is about inference ...


9

A divergent transition in Stan tells you that the region of the posterior distribution around that divergent transition is geometrically difficult to explore. For example here is a quote from the manual: The primary cause of divergent transitions in Euclidean HMC (other than bugs in the code) is highly varying posterior curvature, for which small step ...


9

The question has no clear answer because the empirical Bayes formulation does not & cannot specify how the hyperparameter is estimated. Take the simplest Normal mean estimation problem. When$$X\sim\mathcal N_p(\theta,I_p)\qquad\qquad\theta\sim\mathcal N_p(0,\sigma^2 I_p)$$the Bayes estimator of $\theta$ is$$\delta^\pi(x)=\frac{\sigma^2}{1+\sigma^2}x$...


9

I think this is exaggerated language. Both frequentist and Bayesian have their merits, and statisticians routinely rely on both types in their work. To answer your questions: We can still consider $X \sim N(\mu, 1)$. However, we are not observing $X$, but $X' = \min(100, X)$, which is another Random Variable. 2,3. Now, the authors are saying $\mathbb{E}(...


9

No. Suppose you have $X_1, X_2 \sim N(0,1)$. Then, $$ \bar{X} = \dfrac{X_1 + X_2}{2} \sim N\left(0, \dfrac{1}{2} \right)\,. $$ But $N(0,1) \ne N(0, 1/2)$.


9

The normalising constant in the posterior is the marginal density of the sample in the Bayesian model. When writing the posterior density as $$p(\theta |D) = \frac{\overbrace{p(D|\theta)}^\text{likelihood }\overbrace{p(\theta)}^\text{ prior}}{\underbrace{\int p(D|\theta)p(\theta)\,\text{d}\theta}_\text{marginal}}$$ [which unfortunately uses the same symbol $...


9

The posterior is prior$\,\times\,$likelihood$\,\times\,$constant; the uniform density is simply a constant and gets absorbed in the other constant term. Take as an explicit example the prior $\mathrm{uniform}(0,1)$; then, since the prior pdf is $f(\theta) = 1$, prior$\,\times\,$likelihood = 1$\,\times\,$likelihood = likelihood.


8

Frequentist view 👀 In one sense, we can think of both regularizations as "shrinking the weights"; L2 minimizes the Euclidean norm of the weights, while L1 minimizes the Manhattan norm. Following this line of thinking, we can reason that the equipotentials of L1 and L2 are spherical and diamond-shaped respectively, so L1 is more likely to lead to sparse ...


8

I don't quite know your specific use case though I can provide you with a bunch of resources so that you can make your own decision: Michael Betancourt's case study on sparse regressions: https://betanalpha.github.io/assets/case_studies/bayes_sparse_regression.html That is a particularly great resource because he shows the effect of different priors on the ...


8

The prior which is uniform over $[m,\infty)$ is improper, but informative: it contains the information that the value is at least $m$.


Only top voted, non community-wiki answers of a minimum length are eligible