In chapter 3 of his excellent book ("Generative and discriminative classifiers: Naive Bayes and logistic regression") , Tom Mitchell says that, when learning classifiers based on Bayes rule, one can estimate the parameters of $P(X|Y)$ and $P(Y)$ by maximum likelihood. Example: for discrete $Y$ and $X$, estimator for $P(X=x_i|Y=y_j)$ would be the proportion in the dataset of $X$ having value $x_i$ when $Y$ has value $y_j$. Estimator for $P(Y=y_j)$ would be the proportion of $Y$ having value $y_j$.

Thing that I don't understand is "what is the difference between 1) estimating $P(X|Y)$ and $P(Y)$ parameters as stated above, make the product and divide by $P(X)$ and 2) estimating directly the parameters of $P(Y|X)$? For example, estimator for $P(Y=y_j|X=x_i)$ would be the proportion of $Y$ having value $y_j$ when $X$ has value $x_i$. The result is the same, isn't it?

Am I missing something?


closed as unclear what you're asking by Tim Mar 28 at 21:19

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ You ask about difference between estimating "these" parameters, but it's not clear from the context what do you mean by "these". Also notice that $P(X)$ is not dataset, but distribution. $\endgroup$ – Tim Mar 28 at 6:41
  • $\begingroup$ sorry if it was not clear, I edited my question. Hope it's more clear now. $\endgroup$ – Patrick Mar 28 at 12:11
  • $\begingroup$ With your edits you seem to be asking me questions rather then clarifying your initial question. You started with asking simple question about Bayes theorem. Now you ask about Bayesian statistics & comparing it to MLE. Your initial question was already answered. If you have new questions, start new question. $\endgroup$ – Tim Mar 28 at 21:24
  • 1
    $\begingroup$ You're right @Tim, I lost my train of thought a little. I will start a new question after making sure I can formulate it the right way! Thanks again for your help. $\endgroup$ – Patrick Mar 28 at 22:18
  • $\begingroup$ @Tim new question is here: stats.stackexchange.com/questions/400185/… $\endgroup$ – Patrick Mar 29 at 18:48

You are right, no matter if you estimate $P(Y|X) = \frac{P(X|Y)\,P(Y)}{P(X)}$ using Bayes theorem, or estimate $P(Y|X)$ directly from the data, the result would be the same. If it is not the same, then something is wrong (you made mistake, it is a problem of computer numerical precision etc.). We use Bayes theorem in cases when we can't calculate the probability directly, or for other reasons, doing this indirectly has advantages. For example, you can check this thread, where it is described in detail how Bayes theorem is applied to medical data. In such case we can conduct a case-control study and learn how likely medical test is to discover the condition $P(+|C)$, we also may know what is the prevalence of the condition $P(C)$, but if the condition is rare, it would be insufficient to sample huge group of test takers, just to see how many of them has the condition $P(C|+)$.

For example, say that prevalence of the disease is $P(C)=0.0005$. Let's assume that probability of taking the medical test for the disease is independent of having this disease, so we can expect same proportion of infected people among those who took the test as in the general population. This means that among test-takers, you would need to sample $2000$ people to expect just a single person affected by the disease. What follows, overall size of sample needed to study $P(C|+)$ would need to be huge. On another hand, you could simply take a group of patients who have this condition (e.g. at the hospital) and controls who don't, and test them using your test. This would enable you to calculate $P(+|C)$, and then use Bayes theorem to get the "reverse" conditional probability. (Notice that this is a simplified example, I do not claim that this is a realistic example of how medical trials are conducted.)

  • $\begingroup$ thanks @Tim I'll check this thread right now $\endgroup$ – Patrick Mar 28 at 12:54
  • $\begingroup$ Very interesting thread @Tim! Unfortunately I still don't understand why I would estimate the parameters using the "long way" (see my comment in answer to @dnqxt). Suppose I have a relatively small sample of patients for which I have the information about having cancer or not and having tested positive or not on the test. Why estimate $P(C|+)$ by calculating $\frac{P(+|C)P(C)}{P(+|C)P(C)+P(+|\overline{C})P(\overline{C})}$ when I can estimate it directly by calculating $P(C|+)$ on my sample ? $\endgroup$ – Patrick Mar 28 at 13:44
  • $\begingroup$ @Patrick simple answer is that if you have this data, then you don't need this procedure. Bayes theorem is a way of getting the "reverse" conditional probabilities if you need them. $\endgroup$ – Tim Mar 28 at 15:20
  • $\begingroup$ I thought that Bayes could be useful if you have small datasets and/or prevalence of condition is rare. Isn't that true? $\endgroup$ – Patrick Mar 28 at 16:07
  • $\begingroup$ @Patrick Bayes theorem has nothing to do with size of datasets. Bayes theorem is about probabilities. Size of dataset matters if estimating the probabilities. As said before, if you have sufficient data, then you estimate the probabilities directly. $\endgroup$ – Tim Mar 28 at 16:19

As other people have pointed out, in your particular example, whether you calculate $P(Y | X)$ directly from your data or you use Bayes Theorem it makes no difference. Now you posed the following question

I still don't understand why I would estimate the parameters using the "long way" (see my comment in answer to @dnqxt). Suppose I have a relatively small sample of patients for which I have the information about having cancer or not and having tested positive or not on the test. Why estimate $P(C|+)$ by calculating $P(+|C)P(C)P(+|C)P(C)+P(+|\bar{C})P(\bar{C})$ when I can estimate it directly by calculating P(C|+) on my sample?

The answer is, you wouldn't use Bayes Theorem for this problem because there is no need. The canonical problem where Bayes Theorem shows up is when you aren't given the actual data, but you are given estimates of the probabilities. Usually it's stated like "Suppose the prevalence of this disease in the population is $p_1$, and the test has a sensitivity of $p_2$ and a specificity of $p_3$ etc. You test positive. What's the probability you actually have the disease. Without any other information, the only way to proceed is with Bayes Theorem

The other place that Bayes theorem shows up is hinted at in the title of that book chapter, namely in Naive Bayes, where you have some categorical outcome $Y$ and you want to correctly classify which group it belongs to based on some arbitrary feature vector $X = (X_1, \dots, X_n)$. If the features are all binary or categorical, then with enough data you again don't need to use Bayes theorem because you can estimate $P(Y|X)$ directly (at least in theory). But when you start throwing in continuous features things get more difficult, and one way to proceed is by using Bayes theorem to write it as

$P(Y | X_1, \dots, X_n) \propto P(X_1, \dots, X_n | Y)P(Y)$,

where $\propto$ denotes proportionality. You then make some assumptions about $P(X_1, \dots, X_n | Y)$ and $P(Y)$ to make the calculation simpler (the "naive" part of naive Bayes) and voila!, you have a naive bayes classifier. See https://en.wikipedia.org/wiki/Naive_Bayes_classifier for more info

  • $\begingroup$ Thanks @Bayesic for this explanation. Ok for the continuous features but if the features are all binary or categorical and you don't have much data (and/or prevalence of condition is very rare), would it be a use-case for Bayes? $\endgroup$ – Patrick Mar 28 at 16:03
  • $\begingroup$ That's really a judgement call as it depends on your actual data, the total number of combinations of features, and also (in the case of a binary outcome) whether you're more interested in correctly classifying events (as in fraud detection), or correctly classifying non-events (as in spam e-mail filtering). With a small number of features / feature categories, I don't think you can get better than just predicting based on the estimated P(Y | X) directly. Not sure if that helps $\endgroup$ – Bayesic Mar 28 at 16:18

Consider the following simple data set consisting of a list of pairs representing (X,Y) tuples:

$(0,0),(0,0), (0,1), (1,0), (1,1).$

Then the estimate of $P(X=0|Y=1) = 1/2$, which is different from the estimate of $P(Y=1|X=0) = 1/3$

  • $\begingroup$ thanks @dnqxt for your help. My point is that you can estimate $P(Y=1|X=0)$ directly by counting or you can estimate it indirectly with the Bayes formula: $\frac{P(X=0|Y=1) * P(Y=1)}{P(X=0|Y=0) * P(Y=0)+P(X=0|Y=1) * P(Y=1)}$. The two will give 1/3. Why would you estimate it using the long way? $\endgroup$ – Patrick Mar 28 at 12:27
  • $\begingroup$ @Patrick -- You're welcome. I just addressed the part of the question which is no longer there :) $\endgroup$ – dnqxt Mar 28 at 17:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.