# Implication of marginal independence of features for classification

This question is a follow-up to my earlier question on naive Bayes (NB) classification. The example we're considering is that of spam classification, in which an email is classified as spam ($$S \in \{0, 1\}$$) depending on whether it contains the words "buy" ($$B \in \{0, 1\}$$) and "cheap" ($$C \in \{0, 1\}$$).

In my original question, I had made an error by assuming that the NB assumption not only implied conditional independence of $$B$$ and $$C$$ given $$S$$ but also marginal independence. I'm trying now to understand what the implications are of assuming marginal independence between $$B$$ and $$C$$ (not mathematically but conceptually).

Does assuming $$P(B,C) = P(B)P(C)$$ imply that the distributions of $$P(B,C|S=0)$$ and $$P(B,C|S=1)$$ are the same? Does this therefore imply that neither $$B$$ nor $$C$$ are good features with which to separate out $$S=0$$ from $$S=1$$?

Edit: Let me provide a clarifying example. Let's assume that rather than having discrete features like the presence of the word "buy" or "cheap", I have some continuous features $$B$$ and $$C$$ (e.g., the normalized frequency of the occurrence of the each of the words within an email) whose true joint distribution $$P(B,C)= \sum_{S\in\{0,1\}}P(B,C|S)P(S)$$ is illustrated below. (This is the distribution with which the data was truly generated.) \begin{align} P(B,C|S=0) &= \mathcal{N}\left(\begin{bmatrix}1\\7\end{bmatrix}, \begin{bmatrix}0.25&0.3\\0.3&1\end{bmatrix}\right)\\ P(B,C|S=1) &= \mathcal{N}\left(\begin{bmatrix}4\\3\end{bmatrix}, \begin{bmatrix}0.5&-0.4\\-0.4&0.8\end{bmatrix}\right)\\ \end{align} Thus, the upper left Gaussian corresponds to $$S=0$$ and the bottom right to $$S=1$$. The class priors are $$P(S=0)=0.3$$ and $$P(S=1)=0.7$$. The marginal distributions for $$B$$ and $$C$$ are given in the upper left and bottom right subplots, respectively.

If we assume that $$P(B,C|S)=P(B|S)P(C|S)$$ (the NB assumption), then $$P(B, C) = \sum_SP(B|S)P(C|S)P(S)$$ looks like:

We have effectively zeroed out correlations between $$B$$ and $$C$$ within each class $$S$$ by computing the product of the two marginal distributions $$P(B|S)$$ and $$P(C|S)$$ of $$P(B,C|S)$$. If we were to fit a Gaussian model to the data from each class and we assumed that the covariance within a class was diagonal, this is what we would get. Note that the marginal distributions $$P(B)$$ and $$P(C)$$ (not conditioned on $$S$$) are still the same, which is expected since each class-conditional distribution is bivariate Gaussian and we're still averaging them using the same $$P(S)$$ as before.

The NB independence assumption is class-conditional, which means that we break the dependence between $$B$$ and $$C$$ within each class. This is a reasonable assumption to make since the two classes are easily distinguished by the values of $$B$$ and $$C$$; i.e. knowing the correlation does not buy us much. If, instead, the means were equal, and, moreover, the correlations were the same in magnitude but opposite in sign for the two classes, then, assuming class-conditional independence would mean assuming the generating distributions for both classes are the same (since the product of the marginals would be the same). This would prevent us from being able to distinguish $$S=0$$ from $$S=1$$, and therefore, in that situation, the NB assumption would not be a good choice.

If we were instead to assume marginal independence between $$B$$ and $$C$$, the joint distribution $$P(B,C)=P(B)P(C)$$ would look like:

This is where I am confused. If I had made this assumption instead, how does this tie back into classifying whether $$S=0$$ or $$S=1$$? From the data generating model (first figure), $$B$$ and $$C$$ clearly follow different distributions for each class, but here, I feel like I've lost any notion of which class the features belong to because the distributions are mixed. Another way of asking this is, if this were the data generating distribution, what do $$P(B,C|S=0)$$ and $$P(B,C|S=1)$$ look like? Or, to rephrase my original question, if the assumption that $$P(B,C)=P(B)P(C)$$ were reasonable, what does this imply about $$P(B,C|S)$$ and the ability to distinguish $$S=0$$ and $$S=1$$ using $$B$$ and $$C$$?

MATLAB code for generating the above figures is given below.

clear; close all

%% Correlation (figure 1)

% grid
x1 = 0:.01:6; x2 = 0:.01:10;
[X1,X2] = meshgrid(x1,x2);
X = [X1(:) X2(:)];

% means and covariances
mu1 = [1 7];
sigma1 = [0.25 0.3; 0.3 1];
mu2 = [4 3];
sigma2 = [0.5 -0.4; -0.4 0.8];

% class priors
ps = [0.3 0.7];

% calculate joint distribution by marginalizing out s
y = ps(1)*mvnpdf(X,mu1,sigma1) + ps(2)*mvnpdf(X,mu2,sigma2);
y = reshape(y,length(x2),length(x1));

% plot
figure; subplot(2, 2, 3); contourf(x1, x2, y); caxis([0, 0.25]); xlabel('B'); ylabel('C'); title('P(B,C)')
subplot(2, 2, 1); plot(x1, sum(y, 1)/100); xlim([0 6]); ylabel('P(B)')
subplot(2, 2, 4); plot(sum(y, 2)/100, x2); xlabel('P(C)')

%% Marginal independence (figure 3)

% marginalize joint distribution
y1 = sum(y, 1)/100;
y2 = sum(y, 2)/100;

% compute joint distribution from product of marginals
[Y1,Y2] = meshgrid(y1,y2);
Y = Y1.*Y2;

% plot
figure; subplot(2, 2, 3); contourf(x1, x2, Y); caxis([0, 0.25]); xlabel('B'); ylabel('C'); title('P(B,C)')
subplot(2, 2, 1); plot(x1, y1); xlim([0 6]); ylabel('P(B)')
subplot(2, 2, 4); plot(y2, x2); xlabel('P(C)')

%% Class-conditional independence (figure 2)

% modify covariances
sigma1 = [0.25 0; 0 1];
sigma2 = [0.5 0; 0 0.8];

% calculate joint distribution by marginalizing out s
y = ps(1)*mvnpdf(X,mu1,sigma1) + ps(2)*mvnpdf(X,mu2,sigma2);
y = reshape(y,length(x2),length(x1));

% plot
figure; subplot(2, 2, 3); contourf(x1, x2, y); caxis([0, 0.25]); xlabel('B'); ylabel('C'); title('P(B,C)')
subplot(2, 2, 1); plot(x1, sum(y, 1)/100); xlim([0 6]); ylabel('P(B)')
subplot(2, 2, 4); plot(sum(y, 2)/100, x2); xlabel('P(C)')


The problem in your last setting is that you don't specify $$S$$. In a (supervised) classification problem, $$S$$ is given. If $$S$$ is not given, you don't know what it is, and it can basically be anything. In the last example you have four (more or less) high density areas, which I call UL, UR, LL, LR (upper left, upper right, lower left, lower right). Now it could be that you have $$S=1$$ in UL and UR, in which case you in fact have class-conditional independence, but it could also be that you have $$S=1$$ in UL and LR, in which case there is dependence within classes, or $$S=1$$ just in LR, in which case the $$S=0$$ class has within-class dependence.
In fact the distribution of $$B$$ and $$C$$ doesn't imply anything about $$S$$ (so the answer to your question is "no"); even in your earlier examples both clearly visible "clusters" could have instances of both $$S=0$$ and $$S=1$$, despite the fact that intuitively it looks most likely that they correspond to the classes. Obviously here in fact you generated them artificially knowing the true classes, so you created this correspondence by actually specifying $$S$$ (you could however, as weird as it seems, have chosen $$S$$ so that conditionally on $$S=1$$ you have a mixture of two disjointed half normals, and for $$S=0$$ you take the other halves so that they give your two normals setup when put together). In theory there's nothing stopping you from doing that. In reality more often than not chances are that for such data there will be correspondence, however this is not a necessity and there are counterexamples (for example your two classes may be two different species of a genus, but on the observations that you have, male and female individuals may be clearly separated (in which you in this example are not interested) whereas the species may not be. Also, as in your last example, there may be classes that consist of separated subclasses, without any guarantee how these are located with respect to each other.
• Thanks so much for your response! Could you please clarify why if the data were generated using the mixture of normals in the first example, then assuming that $P(B,C)=P(B)P(C)$ is not helpful when trying to classifying whether $S=0$ or $S=1$? Is it because $P(B,C|S=0)$ is now also a mixture of normals whose modes overlap with that of $P(B,C|S=1)$? Jan 27 '20 at 16:20
• Or is there no relationship at all between $(B,C)$ and $S$? If so, why is this true, because don't we know that in the data-generating distribution, there was a relationship between them? Jan 27 '20 at 16:20
• Hmmm. Not quite sure whether I can say something other than what I've already said. In general the distribution of $B$ and $C$ does not constrain $S$, so if you know the distribution of $B$ and $C$ but nothing about $S$, $S$ can have any relationship (or none at all) with $B$ and $C$ and their joint distribution. Obviously if you assume you know the data generating distribution including $S$, you also know how $S$ is related to $B$ and $C$, however if you only know the distribution of $(B,C)$ it doesn't imply anything about $S$. Jan 28 '20 at 17:29
• I'm not saying there is no relationship, I'm saying it isn't clear what the relationship is, and it can be anything, unless you explicitly define $S$ (as you did in your presentation of the first example, or rather, you explicitly assumed to know the conditional distribution of $(B,C)$ given $S$). Obviously in practice this is about supervised classification, meaning that you at least have some instances of $B, C,$ and $S$, which gives you some information to start with. However, the plain fact that $B$ and $C$ are marginally independent doesn't allow you to say anything safe about $S$. Jan 28 '20 at 17:32