A confusion about Bayes's theorem - Cross Validated most recent 30 from stats.stackexchange.com 2022-01-25T23:13:03Z https://stats.stackexchange.com/feeds/question/375006 https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/375006 2 A confusion about Bayes's theorem MiloMinderbinder https://stats.stackexchange.com/users/46478 2018-11-02T14:50:15Z 2018-11-04T10:40:39Z <p>I am reading <a href="http://conference.scipy.org/proceedings/scipy2014/pdfs/vanderplas.pdf" rel="nofollow noreferrer">a paper</a> on the differences between bayesian outlook and frequentist outlook. The exact pic from the paper is:</p> <p><a href="https://i.stack.imgur.com/9Avfr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9Avfr.png" alt="enter image description here"></a></p> <p>I have read a decent amount about what likelihood is and how it is not a probability because it is defined as a function of parameters and hence does not integrate to 1. I know about conditional probabilities, or so I think. It would seem to me that in Bayes' formula <span class="math-container">$P(D|F)$</span> (probability that given data (assumed iid) is observed given parameter F) should be the product of probabilities independent data points being observed given the parameter <span class="math-container">$F$</span> which is exactly how the likelihood in frequentist approach is defined. Why is it said that <span class="math-container">$P(D|F)$</span> proportional to likelihood and not equal to it?</p> <p>Edit 1:</p> <p>This is from a <a href="http://www.stat.cmu.edu/~brian/463-663/week09/Chapter%2003.pdf" rel="nofollow noreferrer">separate source</a>. </p> <p><a href="https://i.stack.imgur.com/DjJ20.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DjJ20.png" alt="enter image description here"></a></p> <p>As you can see this also claims the same point that <span class="math-container">$P(D/F)$</span> is proportional to Likelihood calculated by multiplying probabilities as a function of a <span class="math-container">$\theta$</span>. I cannot square with the language used here. I am completely clear on likelihood not being a pdf in <span class="math-container">$\theta$</span>. I just don't understand why is <span class="math-container">$P(D/F)$</span> not equal to probabilities of individual samples multiplied together?</p> https://stats.stackexchange.com/questions/375006/-/375011#375011 3 Answer by Tim for A confusion about Bayes's theorem Tim https://stats.stackexchange.com/users/35989 2018-11-02T15:23:05Z 2018-11-04T10:40:39Z <p>You can check other questions tagged as <a href="/questions/tagged/likelihood" class="post-tag" title="show questions tagged &#39;likelihood&#39;" rel="tag">likelihood</a> for more details, but basically likelihood function <span class="math-container">$L$</span> is a probability mass function, or probability density function, <span class="math-container">$f$</span> evaluated on some data <span class="math-container">$X$</span> and parametrized by <span class="math-container">$\theta$</span>:</p> <p><span class="math-container">$$L(\theta | X) = \prod_i f_\theta (X_i)$$</span></p> <p>The definition <strong>is the same</strong> in both frequentist and Bayesian settings, but with the difference that <a href="https://stats.stackexchange.com/questions/224037/wikipedia-entry-on-likelihood-seems-ambiguous">Bayesians treat <span class="math-container">$\theta$</span> as random variable</a> while frequentists treat <span class="math-container">$\theta$</span> as an unknown parameter, where <a href="https://stats.stackexchange.com/questions/112451/maximum-likelihood-estimation-mle-in-layman-terms">likelihood function is maximized</a> to find the "most likely" value of it.</p> <p>My wild guess is that what the author means is that is you just maximize over function, then it doesn't matter if the function integrates to unity or not, so you can omit the normalizing constant from <span class="math-container">$f$</span> and simplify it. In Bayesian setting likelihood is a conditional probability distribution, <em>but</em> if you use MCMC <a href="https://stats.stackexchange.com/questions/129666/why-normalizing-factor-is-required-in-bayes-theorem">same thing happens</a> since the algorithms also don't care about normalizing constants.</p>