1
$\begingroup$

This question is an extended version of this one.

As you can see here, two distributions are equal, I need to compute the parameters a,b,c,d and e. Could you show me a way to do that?


Assume a two-class problem with equal a priori class probabilities and Gaussian class-conditional densities as follows:

$$p(x\mid w_1) = {\cal N}\left(\begin{bmatrix} 0 \\ 0 \end{bmatrix},\begin{bmatrix} a & c \\ c & b \end{bmatrix}\right)\quad\text{and}\quad p(x\mid w_2) = {\cal N}\left(\begin{bmatrix} d \\ e \end{bmatrix},\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}\right)$$

where $ab-c^2=1$.

$\endgroup$
  • 2
    $\begingroup$ Which distributions do you say are equal? $p(x\mid w_1)$ equals $p(x\mid w_2)$ exactly when $a = b = 1$ and $c = d = e = 0$. There must be something else that you have been told but are not revealing to us. $\endgroup$ – Dilip Sarwate Nov 12 '11 at 22:33
  • $\begingroup$ @DilipSarwate $p(x\mid w_1)$ and $p(x\mid w_2)$ are equal. I thought the same way but $ab-c^2=1$ equation makes me start thinking that i am wrong. $\endgroup$ – user7345 Nov 12 '11 at 23:05
  • $\begingroup$ You really should post the entire question in full. Perhaps, you are on the wrong track as your question does not make much sense. Why do you say: "As you can see, two distributions are equal.."? I see no such thing and it is not at all obvious why you would make a statement like that. $\endgroup$ – varty Nov 13 '11 at 1:40
  • $\begingroup$ @varty i made a typo at the question. I am going to fix that and write down all the question. $\endgroup$ – user7345 Nov 13 '11 at 1:41
  • $\begingroup$ Hint: Write the actual density functions that you are expressing as by $p(x\mid w_1)$ and $p(x\mid w_2)$. Remember that each is a joint density function of two random variables, and so both joint densities will be functions of two real variables, say $x_1$ and $x_2$. If you do this correctly, you will have written things that have a common factor $1/2\pi$ and contain exponential functions whose arguments include quadratic expressions in $x_1$ and $x_2$. The boundary of the decision region is the set of points $(x_1,x_2)$ where the two densities that you have written are equal in value. $\endgroup$ – Dilip Sarwate Nov 14 '11 at 0:31
1
$\begingroup$

Disclaimer: The answer below responded to the original version of the OP's question, which was quite different in nature and less specific than the current version.


$p(x∣w_1)$ and $p(x∣w2)$ are equal.

OK, this is going to take a lot longer to answer.

In some statistical applications, a statistician (or a machine, since you included machine learning as a tag) needs to decide which of two hypotheses is true: $H_1 \colon w = w_1$ and $H_2 \colon w = w_2$. It is known that $$P(w = w_1) = P(w = w_2) = \frac{1}{2}.$$ This is what the equal a priori probabilities that you keep referring to means.

Here is a simple method: Always decide that $w = w_1$, and so hypothesis $H_1$ is always the true hypothesis. When in fact $H_1$ is true, your decision is perfectly correct; when in fact $H_2$ is true your decision is perfectly wrong, and thus you have a $50\%$ chance of making an error. More sophisticated methods use a coin toss or a call to a random number generator to decide, but unfortunately still have a $50\%$ chance of making an an error; the same as the simpler mulish insistence that $H_1$ is always true.

To get better performance, i.e., smaller error probability), the statistician might observe a random variable whose distribution depends on the value of $w$. If $w = w_1$, the distribution is $p(x\mid w_1)$; if $w = w_2$, the distribution is $p(x\mid w_2)$. For example, if $w = w_1$, $x$ is a normal random variable with mean $100$ and variance $1$, while if $w = w_2$, $x$ is a standard normal random variable with mean $0$ and variance $1$. So if the statistician observes that $x$ has value $101.2$, it is highly likely that $w = w_1$ and thus very likely that $H_1$ is true because a standard normal random variable is quite unlikely to have large value. On the other hand, if $x$ has small value (say between $-4$ and $+4$), then it is quite likely that $H_2$ is true and $w = w_2$. But notice that all this depends critically on the distributions $p(x\mid w_1)$ and $p(x\mid w_2)$ being different. If the distributions are the same, then observing $x$ is of no help in deciding between $H_1$ and $H_2$. Thus when you claim that

$p(x∣w_1)$ and $p(x∣w_2)$ are equal

you are effectively insisting that observing $x$ is useless as far as deciding between $H_1$ and $H_2$ is concerned.

So, how are these distributions known in the first place? The client might provide them to the statistician based on the knowledge of how the client's apparatus works. Your professor, like Professor Indiana Jones in the movie Raiders of the Lost Ark, might be making them up as he goes along (Remember that $99\frac{44}{100}\%$ of all statistics are made up!). In the context of machine learning, there may be training samples provided: Here are $200$ observations of $x$ when $H_1$ is true, and here are $200$ more when $H_2$ is true. (In your particular problem, $x$ is a bivariate normal random variable with independent (standard normal) components when $H_1$ is true and correlated normal components when $H_2$ is true, and so each sample would be a a pair of numbers). The machine estimates $p(x\mid w_1)$ from the first set of observations and $p(x\mid w_2)$ from the second set, and uses these estimates when making decisions when the real work comes along.

In summary, your claim that $p(x\mid w_1) = p(x\mid w_2)$ means that $x$ is totally useless in distinguishing the two cases. For your particular distribution, equality holds (if you nevertheless contiunue to insist on equality) exactly when $a=b=1$ and $c=d=e=0$ (in which case $ab-c^2 = 1$ as desired). There is no way of solving for $a,b,c,d,e$, or saying what values of $a,b,c,d,e$ make sense in your problem based on the information that you have provided. You need to be given these by your professor, or you need to be given training data so that you can estimate these parameters, or you should emulate Professor Jones and make up some numbers (subject to the constraints that $ab - c^2 = 1$ and $a, b > 0$) and solve the problem using these.

$\endgroup$
  • $\begingroup$ @onatm You have edited and changed the problem statement while I was writing the above answer, and some of my comments are no longer applicable. $\endgroup$ – Dilip Sarwate Nov 13 '11 at 2:07
  • $\begingroup$ thank you for your great answer, it is really informative even i gave you a poor explanation about my problem. $\endgroup$ – user7345 Nov 13 '11 at 2:47
  • $\begingroup$ I don't see how this addresses the question (at least very directly), but it's very possible that this is due to the edit to the question that happened around the same time. The real question appears to be how to find the Bayes optimal decision boundary between the two classes. $\endgroup$ – cardinal Nov 13 '11 at 16:00
  • $\begingroup$ @cardinal I was responding to the original question: "Two distributions are equal, I need to compute the parameters a,b,c,d and e. Could you show me a way to do that?" which seemed to show that the OP did not even understand the basic notion that the two distributions needed to be different if $x$ was going to be of any help in distinguishing between the two classes. He continued to insist that the two distributions were identical. If you feel that my answer is inappropriate, please feel free to edit it or even delete it entirely. $\endgroup$ – Dilip Sarwate Nov 13 '11 at 16:08
  • $\begingroup$ @DilipSarwate: Oh, dear. No, I would certainly never "delete" the content of another user's post! Rather, the effort you put into trying to understand the OP's question (and respond to it!) in its original (rather muddled) form is quite Herculean. Rather, I will edit to add a disclaimer at the top so that it isn't inadvertently downvoted! $\endgroup$ – cardinal Nov 13 '11 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy