-6
$\begingroup$

In Step 1, we have a prior. Using bayes rule we construct the posterior.

In step 2 of some iterated bayesian procedure, the prior becomes the posterior from step one and use bayes rule to calculate the new posterior.

By induction we can do this forever and construct new posteriors based on priors.

I heard this is how Bayesian anything works.

This sounds like any optimization algorithim including Newton-raphson which uses previous information to predict the next step and updates some value. I'm not sure why Frequentists are okay with optimization which is converging to a solution using prior information but are uncomfortable with priors and posteriors. Why?

Does frequentist optimization not update values to converge at a solution?

Improve this question
$\endgroup$
7
  • 4
    $\begingroup$ This is a confusing question for me. Why do you think optimization is inherently Bayesian? You seem to be confusing, e.g., the math associated with the Brouwer Fixed Point Theorem with the math associated with the definition of conditional probability, which is what underlies Bayesian statistics. $\endgroup$
    – jbowman
    Commented Feb 16, 2020 at 4:29
  • $\begingroup$ Because when programming the optimization algorithm such as newton Rahson. You fix an initial starting point and then take derivatives and update that starting value until you converge at a solution. Priors are like that initial starting point and the posterior is the convergent result of the algorithm. $\endgroup$
    – user271077
    Commented Feb 16, 2020 at 4:31
  • 7
    $\begingroup$ That is an analogy, not an identity. $\endgroup$
    – jbowman
    Commented Feb 16, 2020 at 4:32
  • 2
    $\begingroup$ Optimization does not have a prior and a posterior. $\endgroup$
    – jbowman
    Commented Feb 16, 2020 at 4:35
  • 2
    $\begingroup$ No, Bayesian statistics is about optimally updating your prior "beliefs", which are encoded as a probability distribution. The "true distribution" of our beliefs changes as our knowledge changes, and our knowledge changes as we observe data. The "true distribution" of the data is encoded in the likelihood function, and there is some presumption that we get that right. You can certainly have prior beliefs over models, though - see Bayesian Model Averaging for some reviews of this. $\endgroup$
    – jbowman
    Commented Feb 16, 2020 at 4:40

3 Answers 3

Reset to default
4
$\begingroup$

I'm not sure I really understand your analogy, but it seems like your analogy doesn't fully capture either the objections or the philosophy of Bayesianism.

If I were going to criticize Bayesianism, I'd point to the fact that there are no guarantees on the results. Frequentists enjoy frequency properties of their estimators. They can say, at least in theory, that 19 out of every 20 95% confidence intervals they construct will contain the true estimated. That is a very nice thing to be able to say about your methods. Because bayesians necessarily don't view probability in terms of frequency, they can't make those sorts of claims.

Another popular, but ultimately bad, objection is that priors are completely subjective. If you and I have different priors, then how are we to decide which model or analysis is right? At least frequentists can claim their methods are unbiased.

People's objections to Bayesianism have nothing to do with the iterative procedure of conditioning. Without sounding like I am being harsh, I think your characterization shows a great lack of understanding of the Bayesian philosophy.

Improve this answer
$\endgroup$
6
  • $\begingroup$ What about basing priors based on preexisting scientific knowledge and deferring to the literature ? I totally believe that we shouldn’t rely on old knowledge in the literature which can be flawed and the paradox of russel comes in when if you rely on person A for the truth then who does person Arely on? There is not set of all sets. Statistics should be able to make discoveries independently of other fields and should be able to challenge old beliefs and not have to rely on other fields to make their models. Because relying on prior knowledge gets you into a circular logic. $\endgroup$
    – user271077
    Commented Feb 16, 2020 at 4:46
  • $\begingroup$ What about empirical bayes optimization method? Do frequentists object to empirical bayes ? $\endgroup$
    – user271077
    Commented Feb 16, 2020 at 4:47
  • 1
    $\begingroup$ Knowledge changes. I think if you are basing priors on studies done 50 year ago, then that maybe isn't defensible. But priors from last years experiments may be easier to defend. $\endgroup$
    – Demetri Pananos
    Commented Feb 16, 2020 at 4:48
  • 1
    $\begingroup$ I'm not an expert on Empirical Bayes, but to me it sounds like using existing data to create a prior and then proceed in the usual way. I don't see why we should do that instead of using weakly informative priors from the get go. $\endgroup$
    – Demetri Pananos
    Commented Feb 16, 2020 at 4:49
  • $\begingroup$ Is using data to create the prior the idea behind machine learning? $\endgroup$
    – user271077
    Commented Feb 16, 2020 at 4:51
2
$\begingroup$

Why are frequentists uncomfortable with bayesian statistics when "optimization" algorithms used in frequentist statistics is bayesian?

This is a loaded question. The premise that iterative frequentist methods are like the updating of prior information is false.

(In addition the reference to statisticians as 'frequentists' makes it sound like a religion or nationality. There are no frequentist people, but there are people that use frequentist methods.)

In Bayesian statistics we compute a posterior probability distribution for a parameter that is to be estimated (and in order to do so we need a prior probability distribution).

This type of computation, of a probability distribution for the parameter that is to be estimated, does not happen with frequentist methods. The iterative procedures in frequentist methods that you talk about (e.g. iterative re-weighted least squares, or optimization methods to find a MLE, the solution to an equation) are an entirely different thing than this computation of a posterior probability distribution. These iterative procedures require a starting point but it is different from a prior distribution.

Why is a prior distribution problematic and this starting point not?

  • The iterative procedures converge to a point/solution that is purely dependent on the data and completely independent on the starting point. There is no influence of the starting point (or only limited, for instance if the computer makes mistakes because of a bad starting point, but in principle the algorithm should lead to a final point/solution independent from the starting point).

  • On the other hand, with a Bayesian method the end result will be dependent on the data and also the prior information. This is different from the iteration in some algorithm to compute/solve an equation. With Bayes rule your solution becomes dependent on the prior. So a lot effort is needed to select a good prior for the analysis and many books and articles are written about it (it might be half if not more of all the literature about Bayesian methods).

A criticism here might be that the solution of a frequentist method is not purely dependent on the data but also on the model which is part of the analysis (although this a detour from your thinking about the iterative algorithms).

Frequentist methods and Bayesian methods are all containing information outside the data (which makes the methods subjective). The models, assumptions, decisions about analysis, etc. The thing is that with Bayesian methods you have this prior which can be of large influence (if it provides more information than the data).

But if we focus on your idea about the iterative algorithms... those have nothing to do with the prior information and subjective input into the statistical analysis.

Improve this answer
$\endgroup$
1
$\begingroup$

Bayesian: The rationale behind using a bayesian framework is not only the Bayes update rule or the availability of (subjective)prior if any exists, but is mainly due to marginalization and conditioning(of unknown on the known), which drive the modeling process in a Bayesian framework.

The unknown parameters are treated as random variables and are jointly modeled with the known data--any uncertainty associated with unknown parameters is taken into account over here; the posterior distribution of unknown parameters conditioned on known data is determined, and wherever required the distribution of parameters are acquired by marginalizing appropriately.

In addition, with hierarchical modeling, we can incorporate uncertainty associated with hyperparameters, too.

Frequentists: On the other hand, Frequentists model the unknown parameters(treating as just unknown values and not random variables) conditioned on the known data; although frequentists condition the unknown parameters on known data they do not consider any uncertainty associated with the unknown parameters while modeling, unlike Bayesians, which is achieved by considering the distribution of the parameters while modeling implying unknown parameters as Random Variables rather than just unknown.

This difference of whether or not to consider the uncertainty of parameters(distribution vs just value) lies at the root cause of dissension between the two ideologies.

Conclusion: It is marginalization that empowers the Bayesian framework to what is today, and not Bayes rule, availability of prior, which as you mentioned in the question have made their way conceptually into some areas of Mathematics like optimization.

Hope this helps!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.