3
$\begingroup$

Since I have a discriminative mindset hardly can I intuit the so-called parameters needed to specify a distribution in a generative Bayesian Network. I'd like to borrow an example from this blog. If we have two features X: "buy" and Y: 'win' to decide if an email is a spam.

In the scenario of a discriminative model:
It only needs three parameters: two for the two features($X$ and $Y$) and one as a bias:
$P(Z=1|F)=\frac{1}{1+exp(\beta_0 + \beta_1X + \beta_2 Y)}$ and $P(Z=0|F)=\frac{exp(\beta_0 + \beta_1X + \beta_2 Y)}{1+exp(\beta_0 +\beta_1X + \beta_2 Y)}$.

But in the scenario of a generative model, for instance a simple Naïve Bayes model(as that in the aformentioned blog):

80% of the emails are spam:
75% of them have the word “buy”, 40% of them have the word “win”

20% of emails are not spam:
12% of them have the word “buy”, 7% of them have the word “win”

enter image description here

The 2 features are independent given their parent(a common cause trail from X to Y). I try to transfer the example in the blog as a coditional independence representation as follows:

| z0   | z1      |      
|:-----|--------:|   
| 0.2  | 0.8     |  

| Z    | P(X0|Z) | P(X1|Z) |  
|:-----|--------:|--------:|
| z0   | 0.88    | 0.12    |
| z1   | 0.25    | 0.75    |

| Z    | P(Y0|Z) | P(Y1|Z) |  
|:-----|--------:|--------:|
| z0   | 0.93    | 0.07    |
| z1   | 0.60    | 0.40    |

Then I learned that there should be five parameters for the distribution as there are five rows in the above tables and that from $Pr(X,Y,Z)=Pr(X,Y|Z)Pr(Z)=Pr(X|Z)Pr(Y|Z)Pr(Z)$ I can get 2*3-1.

I know how to calculate how many parameters should be needed in that simple generative model but don't understand why there should be 5 parameters as I understand concretely why there should only be 3 parameters in the discrimative model.

Could anyone please show me where the parameters are used in this Bayes Model? As concrete as everyone can comprehend that in the discriminative model two for two features and one for the bias?

$\endgroup$

2 Answers 2

2
$\begingroup$

In discriminative model, when you give an $x$, I need $\beta_0,\beta_1,\beta_2$ to compute a probability for you, so three parameters, concretely as you say. For the Bayes network above, given $x,y,z$, I need to know the entries in the tables to compute the probability. But the table entries are related. For example, for $P(z)$, we just need to know one of the entries, e.g. $P(z_0)$. That's one parameter. For the conditionals, $P(x|z=z_0)$ and $P(x|z=z_1)$, again just one value at each row is needed to fill out the table. So, one parameter for $P(x|z=z_0)$, one for $P(x|z=z_1)$, and one for each $P(y|z=z_i)$, totalling upto 5 parameters. Luckily, in this scenario, it is equal to the number of lines. But, if you didn't have binary variables, at each row, we'd need more parameters.

$\endgroup$
2
  • $\begingroup$ Can I say that at least five values(or parameters as it can change) are needed for forming the network? 20%, 75%, 40%, 12% and 7%? $\endgroup$ Jan 20, 2019 at 14:05
  • $\begingroup$ Exactly! And, I think you've misplaced 0.88 and 0.12. $\endgroup$
    – gunes
    Jan 20, 2019 at 14:14
0
$\begingroup$

Actually the logistic regression and the Naive Bayesian Network are closely related.

If we converse the two arrows in the following graph we get the Bayesian Network for the discriminative model.

enter image description here

Since these two(Naive Bayes and Logistic Regression) function for different tasks, that's the NB with all given parameters can be responsible for all queries but for the LR we just need to infer $P(Z|X,Y)$. Both X and Y are given in any particular related tasks.

Knowing the difference of their responsibilities we can get to know easily the difference of their requirements.

To construct a NB we need to weight all the possibilities of the combinations by the joint probabilities. Independences can reduce much of the possible unwarranted combinations but there are still a lot of possibilities we should consider. Since the global joint distribution depends on the local conditional distributions, we only need to set all the conditional probabilities in all of the conditional probability tables(CPT).

Probabilistic parameters are encoded in a set of tables, one for each variable, in the form of local conditional distributions of a variable given its parents. Using the independence statements encoded in the network, the joint distribution is uniquely determined by these local conditional distributions.

Source: Bayesian Network Classifiers

Then we can just check how many numbers we should fill in the conditional probability tables. For the first one we need 1(since the conditional distributin should be valid the sum of the two should be 1); and the second table needs 2 parameters one for each row and the same for the third table. It is not only how many rows the CPTs have but also how many parameters each row needs that determine the necessary number of parameters. Such parameters can be human manipunated or learned from data using the global maximum likelihood.

EDIT:
That is a pretty easy question, and it would be very obvious had I differentiated these three concepts: variables, events and parameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.