3
$\begingroup$

I am reading a paper on the differences between bayesian outlook and frequentist outlook. The exact pic from the paper is:

enter image description here

I have read a decent amount about what likelihood is and how it is not a probability because it is defined as a function of parameters and hence does not integrate to 1. I know about conditional probabilities, or so I think. It would seem to me that in Bayes' formula $P(D|F)$ (probability that given data (assumed iid) is observed given parameter F) should be the product of probabilities independent data points being observed given the parameter $F$ which is exactly how the likelihood in frequentist approach is defined. Why is it said that $P(D|F)$ proportional to likelihood and not equal to it?

Edit 1:

This is from a separate source.

enter image description here

As you can see this also claims the same point that $P(D/F)$ is proportional to Likelihood calculated by multiplying probabilities as a function of a $\theta$. I cannot square with the language used here. I am completely clear on likelihood not being a pdf in $\theta$. I just don't understand why is $P(D/F)$ not equal to probabilities of individual samples multiplied together?

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ They are equal. Which trivially implies that they are proportional, but that's beside the point. Not sure why the author chose to word it that way. $\endgroup$
    – jcz
    Nov 2, 2018 at 14:54
  • 1
    $\begingroup$ The big difference between frequentist and Bayesian approaches is whether $F$ is treated as a parameter or a random variable. To the frequentist, $F$ is simply a number (although we don't know what the number is). The Bayesian though is willing to invoke the machinery of probability to model her own subjective uncertainty over $F$. $\endgroup$
    – Matthew Gunn
    Nov 2, 2018 at 15:28
  • $\begingroup$ I agree with the other responses. Conceptually, the sampling density and the likelihood function are equal. But in practice when you're actually doing inference (maximizing the likelihood function, using the Metropolis-Hastings algorithm to sample from the posterior distribution, etc), you often only need to work with the sampling density or the likelihood function up to a normalizing constant. Suffice it to say, the document you are citing is generally imprecise in its use of language. For example, $P(D_i\,|\,F)$ is not "the probability of $D_i$ given $F$." So take it with a grain of salt. $\endgroup$
    – jcz
    Nov 4, 2018 at 21:10
  • $\begingroup$ @bamts - Thank you for your valuable responses. How is $P(D_i/F)$ not "the probability of $D_i$ given $F$"? $\endgroup$
    – figs_and_nuts
    Nov 5, 2018 at 7:49
  • 1
    $\begingroup$ $D_i$ is a continuous random variable, and in the text $P(D_i\,|\,F)$ is defined to be the density function of $D_i$. So that function returns density values, not probabilities. To get probabilities of $D_i$ taking on certain values, you'd have to integrate $P(D_i\,|\,F)$, not evaluate it. $\endgroup$
    – jcz
    Nov 5, 2018 at 15:15

1 Answer 1

Reset to default
3
$\begingroup$

You can check other questions tagged as for more details, but basically likelihood function $L$ is a probability mass function, or probability density function, $f$ evaluated on some data $X$ and parametrized by $\theta$:

$$ L(\theta | X) = \prod_i f_\theta (X_i) $$

The definition is the same in both frequentist and Bayesian settings, but with the difference that Bayesians treat $\theta$ as random variable while frequentists treat $\theta$ as an unknown parameter, where likelihood function is maximized to find the "most likely" value of it.

My wild guess is that what the author means is that is you just maximize over function, then it doesn't matter if the function integrates to unity or not, so you can omit the normalizing constant from $f$ and simplify it. In Bayesian setting likelihood is a conditional probability distribution, but if you use MCMC same thing happens since the algorithms also don't care about normalizing constants.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Yeah, ignoring normalizing constants because they aren't computationally necessary comes up in both Bayesian and frequentist contexts and is a somewhat orthogonal issue to the whole Bayesian and frequentist distinction. $\endgroup$
    – Matthew Gunn
    Nov 2, 2018 at 15:43
  • $\begingroup$ Please see the edit i made. I understand the points you raise in your answer. I just dont know why is it written that evidence in the equation is proportional to individual probabilities multiplied together and not equal to it? $\endgroup$
    – figs_and_nuts
    Nov 4, 2018 at 10:17
  • $\begingroup$ @MiloMinderbinder as I said in my answer and others said in the comments, they are basically the same and there's no reason for using proportionality symbol. The only difference is in interpreting $\theta$, but I can't see how using proportionality symbol would have anything to do with this. If you want to know authors intentions, email them. $\endgroup$
    – Tim ♦
    Nov 4, 2018 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.