The Stack Overflow podcast is back! Listen to an interview with our new CEO.
41

I believe that you and your colleague are correct. Statistics.com has the correct line of thinking, but makes a simple mistake. Out of the 90 "OK" claims, we expect 20% of them to be incorrectly classified as fraud, not 80%. 20% of 90 is 18, leading to 9 correctly identified claims and 18 incorrect claims, with a ratio of 1/3, exactly what Bayes' rule yields....


24

This paper by Christian (Xi'an) Robert and George Casella provides a nice summary of the history of MCMC. From the paper (emphasis is mine). What can be reasonably seen as the first MCMC algorithm is what we now call the Metropolis algorithm, published by Metropolis et al. (1953). It emanates from the same group of scientists who produced the Monte Carlo ...


18

Penalized regression estimators such as LASSO and ridge are said to correspond to Bayesian estimators with certain priors. Yes, that is correct. Whenever we have an optimisation problem involving maximisation of the log-likelihood function plus a penalty function on the parameters, this is mathematically equivalent to posterior maximisation where the ...


18

Your question can also be stated as: "$X$ is dependent on $a$ and $b$. And $a$ and $b$ are independent. Does this imply that $a$ and $b$ are conditionally independent given $X$?" The answer is no. We just need a counter-example to show it isn't the case. Suppose $X = a + b$. Then, once we know $X$'s value, $a$ and $b$ are dependent (information about one ...


15

With flat priors, the posterior is identical to the likelihood up to a constant. Thus MLE (estimated with an optimizer) should be identical to the MAP (maximum a posteriori value = multivariate mode of the posterior, estimated with MCMC). If you don't get the same value, you have a problem with your sampler or optimiser. For complex models, it is very ...


15

Some possible generic explanations for this perceived discrepancy, assuming of course there is no issue with code or likelihood definition or MCMC implementation or number of MCMC iterations or convergence of the likelihood maximiser (thanks, Jacob Socolar): in large dimensions $N$, the posterior does not concentrate on the maximum but something of a ...


13

This is based on the assumption that $x$ is conditionally independent of $D$, given $\theta$. This is a reasonable assumption in many cases, because all it says is that the training and testing data ($D$ and $x$, respectively) are independently generated from the same set of unknown parameters $\theta$. Given this independence assumption, $p(x|\theta,D)=p(x|\...


12

I beg to disagree with the answer given by pche8701: the main reason a flat prior is introduced (in an improper setting) as $f(\theta)\propto c$ or $f(\theta)\propto 1$ which is equivalent but more rigorous is that (i) any constant $c$ leads to the same posterior distribution and (ii) there is no principled way to choose a value for the constant $c$ since a ...


12

The p-value is the probability of seeing what you saw or something more extreme if the null hypothesis was true. The p-value is not the probability that the null hypothesis is true. So yes, interpreting a p-value as the probability that the null hypothesis is true is akin to the prosecutor's fallacy. If you want that probability, you need to assume a ...


11

You are correct. The solution that the website posted is based on a misreading of the problem in that 80% of the nonfraudulent claims are classified as fraudulent instead of the given 20%.


11

With prior $\mathsf{Unif}(0,1) \equiv \mathsf{Beta}(\alpha_0=1,\beta_0 =1)$ and likelihood $\mathsf{Binom}(n, \theta)$ showing $x$ successes in $n$ trials, the posterior distribution is $\mathsf{Beta}(\alpha_n=1 + x,\; \beta_n = 1 + n - x).$ (This is easily seen by multiplying the kernels of the prior and likelihood to get the kernel of the posterior.) ...


11

The excellent answer by knrumsey gives some history on the progression of important academic work in MCMC. One other aspect worth examining is the development of software to facilitate MCMC by the ordinary user. Statistical methods are often used mostly by specialists until they are implemented in software that allows the ordinary user to implement them ...


10

This is a term that is specifically from empirical Bayes (EB), in fact the concept that it refers to does not exist in true Bayesian inference. The original term was "borrowing strength", which was coined by John Tukey back in the 1960s and popularized further by Bradley Efron and Carl Morris in a series of statistical articles on Stein's paradox and ...


10

So long as the confidence interval is treated as random (i.e., looked at from the perspective of treating the data as a set of random variables that we have not seen yet) then we can indeed make useful probability statements about it. Specifically, suppose you have a confidence interval at level $1-\alpha$ for the parameter $\theta$, and the interval has ...


10

This is called Laplace's smoothing, or Laplace's rule of succession, as Pierre-Simon Laplace used it for estimating the probability the sun rises again tomorrow: "We thus find that an event having occurred a number of times, the probability that it will happen again the next time is equal to this number increased by the unit, divided by the same number ...


10

Logistic regression, Bayesian or not, is a model defined in terms of Bernoulli distribution. The distribution is parametrized by "probability of success" $p$ with mean $p$ and variance $p(1-p)$, i.e. the variance directly follows from the mean. So there is no "separate" variance term, this is what the quote seems to say.


10

A few things: The BF gives you evidence in favor of a hypothesis, while a frequentist hypothesis test gives you evidence against a (null) hypothesis. So it's kind of "apples to oranges." These two procedures, despite the difference in interpretations, may lead to different decisions. For example, a BF might reject while a frequentist hypothesis test doesn'...


10

No, it doesn't: Under the assumption that $a \ \bot \ b$, the right-hand-side of your last equation is: $$p(x|a,b) \cdot p(a) \cdot p(b) = p(x,a,b) \overset{a,b}{\propto} p(a,b|x).$$ Thus, you are effectively asking whether or not: $$a \ \bot \ b \quad \quad \implies \quad \quad p(a|x) \cdot p(b|x) \propto p(a,b|x).$$ That is, you are asking whether ...


10

In a rather narrow sense, you are correct (1): if $$\pi(\theta|x) \propto L(\theta|x)$$ then $$\underbrace{\arg \max_\theta \pi(\theta|x)}_\text{MAP estimate} = \underbrace{\arg \max_\theta L(\theta|x)}_\text{ML estimate}$$ However, this coincidence is not of considerable interest as: [Re. (1) and (3):] It is not invariant by repameterisation, i.e., the ...


10

We often don’t care about the likelihood, just the value for which the likelihood is maximized. When you use the likelihood function to find a maximum likelihood estimator, you get the same point giving the maximum whether you include constants out front or not. Sure, that maximum value will be different, but that is not our concern. So let’s make it ...


9

The conditional density kernels are: $$\begin{equation} \begin{aligned} f(x|y) &\propto \exp(-|x|-a \cdot |x-y|), \\[6pt] f(y|x) &\propto \exp(-|y|-a \cdot |x-y|). \\[6pt] \end{aligned} \end{equation}$$ The difficulty here is to derive the actual densities that go with these kernels, which takes a bit of algebra. Integration over the full range of ...


9

It's because $x$ is assumed to be independent of $D$ given $\theta$. In other words, all data is assumed to be i.i.d. from a normal distribution with parameters $\theta$. Once $\theta$ is taken into account using information from $D$, there is no more information that $D$ gives us about a new data point $x$. Therefore $p(x|\theta, D) = p(x|\theta)$.


9

Consider the situation in which you have $n = 20$ observations of a binary (2-coutcome) process. Often the two possible outcomes on each trial are called Success and Failure. Frequentist confidence interval. Suppose you observe $x = 15$ successes in the $n = 20$ trials. View the number $X$ of Successes as a random variable $X \sim \mathsf{Binom}(n=20; p),$ ...


8

Sorry for being confusing! The joint posterior distribution on $(\xi_1,\xi_2)$ is $$\pi(\xi_1,\xi_2|x)\propto \exp\{-(x-\xi_1)^2/2\}\pi_1(2\xi_1)\pi_2(2\xi_2)$$ Therefore the marginal posterior on $\xi_2$ is given by the marginal of the above, up to a constant, that is, $$\pi(\xi_2|x)\propto \int\exp\{-(x-\xi_1)^2/2\}\pi_1(2\xi_1)\pi_2(2\xi_2)\,\text{d}\xi_1$...


8

If you fit two models, one with logit and one with cloglog, you should report the results of both, and also carry out some type of model comparison technique and report the results of that. As for the models, this is a great situation in which to use Bayesian multilevel models (see this paper [PDF] by Gelman). We can pool information among groups to inform ...


8

Quoting verbatim from my book It is always possible to reduce an exponential family to a standard and minimal form of dimension $m$, and this dimension $m$ does not depend on the chosen parameterisation (Brown, 1986, pp. 13-16). (See Exercise 3.20 for an example of a non-regular exponential family.) Natural exponential families can also be ...


8

This is an case where the parameters are non-identifiable in your model. As you point out, contributions from the individual non-identifiable parameters in these ratios cannot be distinguished using the data. When using Bayesian analysis with a non-identifiable model, specification of a prior for all the individual non-identifiable parameters will still ...


8

Disclaimer: although there is nothing to complain about Ben's answer (!), except maybe that the normalising constant of the conditional is not of direct use, here is what I wrote while being off-line, so I may as well post it! The full conditional of $X$ given $Y$ has a density that is proportional to \begin{align} f(x|y) &\propto \exp\{ -|x|-a|y-x|\}\...


8

There are a couple of issues here: In classical statistics all the distributions used are implicitly conditional on $\theta$, which is considered to be an "unknown constant". In Bayesian analysis there is no such thing as an unknown constant (anything unknown is treated as a random variable) and we instead use explicit conditioning statements for all ...


8

Personally, I would venture a few guesses: (1) Bayesian statistics saw a huge uptick in popularity in the last couple decades. Part of this was due to advancements in MCMC and improvements in computational resources. Bayesian statistics went from being theoretically really nice but only applicable to toy problems to an approach that could be more ...


Only top voted, non community-wiki answers of a minimum length are eligible