# Bayes Classification vs Naive Bayes Classification

Generally, known that Bayes Classifier is optimal for the probability of error. But when I did some experiments:

First Case: I have 2 classes data and their covariance matrices correlated in this case probability of Bayes classifier is %5 and Naive Bayes Classifier's probability of error %16

So, Bayes_Pe < Naive_Bayes_Pe which is expected.

But

Second Case: Again I have 2 classes data and their covariance matrices uncorrelated in this case probability of error of Bayes classifier increase to %16 and Probability of error of Naive Bayes Classifier is stay same again %16.

What is the reason that in the second case Bayes Classifier's Pe is increase and Naive stay stable??

My Code:

...

clear
format compact
close all

randn('seed',0)
% Definition of mu's and Sigma
% Mean vectors and covariance matrix
m1=[0 2]'; m2=[0 0]';  S1=[4 1.8; 1.8 1]; S2= [4 1.2; 1.2 1];  %S1=[4 0; 0 1]; S2= [4 0; 0 1];

% Number of data points
n_points_per_class=5000;

% (i) Data point generation
X=[mvnrnd(m1',S1,n_points_per_class); mvnrnd(m2',S2,n_points_per_class)]';
label=[ones(1,n_points_per_class) 2*ones(1,n_points_per_class)];
[l,p]=size(X);
%Plot the data set
figure; plot(X(1,label==1),X(2,label==1),'.b',X(1,label==2),X(2,label==2),'.r'); axis equal

% (ii) Bayes classification of X
% Estimation of a priori probabilities
P1=n_points_per_class/p;
P2=P1;
% Estimation of pdf's for each data point(Bayes)
for i=1:p
p1(i)=(1/(2*pi*sqrt(det(S1))))*exp(-(X(:,i)-m1)'*inv(S1)*(X(:,i)-m1)/2);
p2(i)=(1/(2*pi*sqrt(det(S2))))*exp(-(X(:,i)-m2)'*inv(S2)*(X(:,i)-m2)/2);
end
% Classification of the data points
for i=1:p
if(P1*p1(i)>P2*p2(i))
class(i)=1;
else
class(i)=2;
end
end

% (iii) Error probability estimation
Pe=0; %Probability of error
for i=1:p
if(class(i)~=label(i))
Pe=Pe+1;
end
end
Pe=Pe/p

%Plot the data set
figure; plot(X(1,class==1),X(2,class==1),'.b',X(1,class==2),X(2,class==2),'.r'); axis equal

%(iv)
% Estimation of pdf's for each data point(Naive Part)

for i=1:p
p1_naive(i)=( (1/(sqrt(2*pi*S1(1,1))))*exp(-(X(1,i)-m1(1))'*(X(1,i)-m1(1))/(2*S1(1,1))) ) * ( (1/(sqrt(2*pi*S1(2,2))))*exp(-(X(2,i)-m1(2))'*(X(2,i)-m1(2))/(2*S1(2,2))) );
p2_naive(i)=( (1/(sqrt(2*pi*S2(1,1))))*exp(-(X(1,i)-m2(1))'*(X(1,i)-m2(1))/(2*S2(1,1))) ) * ( (1/(sqrt(2*pi*S2(2,2))))*exp(-(X(2,i)-m2(2))'*(X(2,i)-m2(2))/(2*S2(2,2))) );
end
% Classification of the data points
for i=1:p
if(P1*p1_naive(i)>P2*p2_naive(i))
class_naive(i)=1;
else
class_naive(i)=2;
end
end

% (v) Error probability estimation
Pe_naive=0; %Probability of error
for i=1:p
if(class_naive(i)~=label(i))
Pe_naive=Pe_naive+1;
end
end
Pe_naive=Pe_naive/p

%Plot the data set
figure; plot(X(1,class_naive==1),X(2,class_naive==1),'.b',X(1,class_naive==2),X(2,class_naive==2),'.r'); axis equal


...

• I'm not that familiar with Bayes classification, but this question is probably unanswerable without a reproducible example. Can you post your code, ideally with data? Mar 30, 2020 at 15:57
• Also, you may be confusing the word "optimal" for the word "best". The former is probably true in a very restrictive context. The latter is usually very hard to know without testing within a cross-validation framework. en.wikipedia.org/wiki/No_free_lunch_theorem Mar 30, 2020 at 16:00
• @RGregStacey code added. Mar 30, 2020 at 16:06
• Not a full answer but a general comment. I wouldn't be surprised when one classifier outperforms another... Mar 30, 2020 at 16:32
• @RGregStacey Yes, I am looking for the exact reason for this, otherwise one classifier can of course be superior to another ... Mar 30, 2020 at 16:37

Optimal Bayes classifier is a theoretical concept, it says that if you knew the conditional distribution $$p(Y=C|X)$$, then the choice that has lowest probability of misclassification is when you choose the class with highest probability
$$\underset{C}{\operatorname{arg\,max}} \; p(y=C|X)$$
$$p(y, x_1, x_2, \dots, x_k) \approx p(y) \prod_{i=1}^k p(x_i | y)$$
You seem to be asking why using joint distribution in Bayes theorem, vs using the above approximation of joint distribution would give different results. In case where the features are not independent and you assume independence (Naive Bayes), the estimate of joint distribution would be wrong, so the result would be inferior. If the features are independent, then by definition of independence $$p(x_1, x_2) = p(x_1) \, p(x_2)$$, so both methods give exactly the same results.