0
$\begingroup$

Generally, known that Bayes Classifier is optimal for the probability of error. But when I did some experiments:

First Case: I have 2 classes data and their covariance matrices correlated in this case probability of Bayes classifier is %5 and Naive Bayes Classifier's probability of error %16

So, Bayes_Pe < Naive_Bayes_Pe which is expected.

But

Second Case: Again I have 2 classes data and their covariance matrices uncorrelated in this case probability of error of Bayes classifier increase to %16 and Probability of error of Naive Bayes Classifier is stay same again %16.

What is the reason that in the second case Bayes Classifier's Pe is increase and Naive stay stable??

My Code:

...

clear
format compact
close all

randn('seed',0)
% Definition of mu's and Sigma
% Mean vectors and covariance matrix
m1=[0 2]'; m2=[0 0]';  S1=[4 1.8; 1.8 1]; S2= [4 1.2; 1.2 1];  %S1=[4 0; 0 1]; S2= [4 0; 0 1];

% Number of data points
n_points_per_class=5000;

% (i) Data point generation
X=[mvnrnd(m1',S1,n_points_per_class); mvnrnd(m2',S2,n_points_per_class)]';
label=[ones(1,n_points_per_class) 2*ones(1,n_points_per_class)];
[l,p]=size(X);
%Plot the data set
figure; plot(X(1,label==1),X(2,label==1),'.b',X(1,label==2),X(2,label==2),'.r'); axis equal


% (ii) Bayes classification of X
% Estimation of a priori probabilities
P1=n_points_per_class/p;
P2=P1;
% Estimation of pdf's for each data point(Bayes)
for i=1:p
    p1(i)=(1/(2*pi*sqrt(det(S1))))*exp(-(X(:,i)-m1)'*inv(S1)*(X(:,i)-m1)/2);
    p2(i)=(1/(2*pi*sqrt(det(S2))))*exp(-(X(:,i)-m2)'*inv(S2)*(X(:,i)-m2)/2);
end
% Classification of the data points
for i=1:p
    if(P1*p1(i)>P2*p2(i))
        class(i)=1;
    else
        class(i)=2;
    end
end

% (iii) Error probability estimation
Pe=0; %Probability of error
for i=1:p
    if(class(i)~=label(i))
        Pe=Pe+1;
    end
end
Pe=Pe/p

%Plot the data set
figure; plot(X(1,class==1),X(2,class==1),'.b',X(1,class==2),X(2,class==2),'.r'); axis equal

%(iv) 
% Estimation of pdf's for each data point(Naive Part)

for i=1:p
    p1_naive(i)=( (1/(sqrt(2*pi*S1(1,1))))*exp(-(X(1,i)-m1(1))'*(X(1,i)-m1(1))/(2*S1(1,1))) ) * ( (1/(sqrt(2*pi*S1(2,2))))*exp(-(X(2,i)-m1(2))'*(X(2,i)-m1(2))/(2*S1(2,2))) );
    p2_naive(i)=( (1/(sqrt(2*pi*S2(1,1))))*exp(-(X(1,i)-m2(1))'*(X(1,i)-m2(1))/(2*S2(1,1))) ) * ( (1/(sqrt(2*pi*S2(2,2))))*exp(-(X(2,i)-m2(2))'*(X(2,i)-m2(2))/(2*S2(2,2))) );
end
% Classification of the data points
for i=1:p
    if(P1*p1_naive(i)>P2*p2_naive(i))
        class_naive(i)=1;
    else
        class_naive(i)=2;
    end
end

% (v) Error probability estimation
Pe_naive=0; %Probability of error
for i=1:p
    if(class_naive(i)~=label(i))
        Pe_naive=Pe_naive+1;
    end
end
Pe_naive=Pe_naive/p

%Plot the data set
figure; plot(X(1,class_naive==1),X(2,class_naive==1),'.b',X(1,class_naive==2),X(2,class_naive==2),'.r'); axis equal

...

$\endgroup$
5
  • $\begingroup$ I'm not that familiar with Bayes classification, but this question is probably unanswerable without a reproducible example. Can you post your code, ideally with data? $\endgroup$ Mar 30 '20 at 15:57
  • $\begingroup$ Also, you may be confusing the word "optimal" for the word "best". The former is probably true in a very restrictive context. The latter is usually very hard to know without testing within a cross-validation framework. en.wikipedia.org/wiki/No_free_lunch_theorem $\endgroup$ Mar 30 '20 at 16:00
  • 1
    $\begingroup$ @RGregStacey code added. $\endgroup$ Mar 30 '20 at 16:06
  • $\begingroup$ Not a full answer but a general comment. I wouldn't be surprised when one classifier outperforms another... $\endgroup$ Mar 30 '20 at 16:32
  • $\begingroup$ @RGregStacey Yes, I am looking for the exact reason for this, otherwise one classifier can of course be superior to another ... $\endgroup$ Mar 30 '20 at 16:37
1
$\begingroup$

Sorry, but I really do not like MATLAB, so I didn't go carefully through your code. Forgive me if I missed something about your question.

Optimal Bayes classifier is a theoretical concept, it says that if you knew the conditional distribution $p(Y=C|X)$, then the choice that has lowest probability of misclassification is when you choose the class with highest probability

$$ \underset{C}{\operatorname{arg\,max}} \; p(y=C|X) $$

Naive Bayes classifier is a classification algorithm, that uses the estimated marginal probabilities, naively assuming independence, to calculate probability distribution and use it for classification

$$ p(y, x_1, x_2, \dots, x_k) \approx p(y) \prod_{i=1}^k p(x_i | y) $$

You seem to be asking why using joint distribution in Bayes theorem, vs using the above approximation of joint distribution would give different results. In case where the features are not independent and you assume independence (Naive Bayes), the estimate of joint distribution would be wrong, so the result would be inferior. If the features are independent, then by definition of independence $p(x_1, x_2) = p(x_1) \, p(x_2)$, so both methods give exactly the same results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.