1
$\begingroup$

I was exposed to bayes in college, but it's been some time. I'm trying to confirm some ideas about Bayesian vs. Frequentist updates, but I can't seem to get the right jargon to search Google. The following is a simple example of what I'm looking for. Obviously, I'd appreciate confirmation of the assumptions, but technical names for anything I describe would be even more helpful (so I know how to find "next steps").

Given a binomial process that generates A or B. After 100 draws, we have 36 Bs.

Assumption 1: A Frequentist and Bayesian process conclude the same p(B)

If assumption 1 is wrong (or depends on a prior not sufficiently specified), let's assume that that both agree on p(B) and draw the next value.

Assumption 2: Bayesian processes can be used (perhaps trivially) to update this belief

Assumption 3: The Frequentist and Bayesian would conclude the same p(B) after the draw

Following from Assumption 3,

Assumption 4: Since p(B) for the frequentist is well-specified, we can back into a specific formula for the Likelihood Ratio used by the Bayesian

Assumption 5: The formula for Likelihood Ration includes the sample size

For example, what terminology would I use to find this formula?

$\endgroup$
  • $\begingroup$ "Bayesians" do not conclude a point estimate of $p(B)$, one of the hallmarks of Bayesian analysis is obtaining entire distributions. $\endgroup$ – Andris Birkmanis Apr 6 '17 at 22:52
  • $\begingroup$ @AndrisBirkmanis what about MAP estimators? Surely those are point estimates. $\endgroup$ – Taylor Apr 6 '17 at 23:21
  • $\begingroup$ They are; there is a POV that they are not representative of Bayesian approach: en.wikipedia.org/wiki/Maximum_a_posteriori_estimation#Criticism. $\endgroup$ – Andris Birkmanis Apr 7 '17 at 0:30
  • $\begingroup$ @AndrisBirkmanis, I appreciate that you're making an important technical distinction. In this intentionally stylized case, does it lead to any material differences? Absent any knowledge except the 100 draws, is there any any wager in which the two approaches differ? $\endgroup$ – claytond Apr 7 '17 at 1:22
  • $\begingroup$ I believe Dave Harris gave an excellent answer. $\endgroup$ – Andris Birkmanis Apr 7 '17 at 1:59
3
$\begingroup$

Lets clean up some notation issues first. Let the true value of the parameter be denoted as $\theta$ and a point estimator of $\theta$ be denoted as $\hat{\theta}$. $\pi(k=K)$ denotes the probability that $k=K$. The Frequentist process begins with designing the experiment and creating a null hypothesis. The null can matter if a decision about the parameter needs to be made rather than just an inference that results in an action.

To talk about comparing the methods, you really need to state a null hypothesis first because probabilities are always with reference to the null. To make matters simple, let us assert that it is important to us for some reason that $\theta\le{.5}$, which implies that half or less of the tosses will result in an A as the number of tosses goes to infinity. For comparative purposes, we will use the same hypothesis for the Bayesian method.

The Frequentist is assuming the coin tosses are "identical," though what that precisely means is a bit of a problem. Fundamentally, the process of tossing the coin is subject to the same forces. The Bayesian does not necessarily make that strong assumption. It is possible that a con man or magician tossed it sometimes, deterministically, and a child tossed it later in a manner similar to "randomness." The Bayesian makes the weaker assumption the model you assert is a fair representation of the events in nature.

The Bayesian needs to specify a prior distribution. Since nothing is known about the true location of $\theta$ for the purposes of our exposition, we will use the uniform distribution as our prior distribution. It is important to note that although the uniform distribution grants equal weight to all possible values of $\theta$ the expected value of theta is $$\int_0^1\theta\mathrm{d}\theta=\frac{1}{2}.$$

This bias in the expectation will never go away, though its importance will diminish to zero as the sample size goes to infinity. You asserted that

Given a binomial process that generates A or B. After 100 draws, we have 36 Bs.

The minium variance unbiased estimator is $\hat{\theta}=.64$. The posterior density function will be $$199697662796638231933820787675(1-\theta)^{36} \theta^{64}.$$

If the Bayesian needed a point estimator for some reason, it would be based upon the cost of being wrong in making the estimate. Because of this, there are an infinite number of points. Nonetheless, there are three very common cost functions, the quadratic, the absolute linear lost and the all-or-nothing cost functions. The three points are the mean, median and mode of the posterior density function and not of the data. The respective value of the Bayesian point estimators will be .637255, .638155 and .64 respectively.

The frequentist rejects the null at p<.002855. The Bayesian has no null. The probability that $\theta\le{.5}$ is .00253869 under the Bayesian model.

For the Frequentist, if the process were repeated the concern is with the null and although there is no updating of the estimator, there is a process of reviewing how frequently the null is rejected. For the Bayesian the location of the estimator is updated with each sampling.

You are probably looking for terms like conjugate prior, Polya distribution, beta-binomial, Bernouli trials, Bayesian updating and maximum likelihood.

$\endgroup$
  • $\begingroup$ Thanks! StackExchange actually listed a beta-binomial question among the "Related" questions and (a Google search later) this model has some properties that align well with the direction I'm headed. $\endgroup$ – claytond Apr 7 '17 at 1:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.