I have 10 features for 3 classes and all of them have a distribution similar to this (green, blue and orange is for class 0,1,2):

Feature 1 distribution

Feature 2 distribution

My result when apply Gaussian

I tried with NBGausian and LogisticRegression however it looks like that this algorithm is not good enough for them. Do you have any suggested algorithms for those?

  • $\begingroup$ Welcome to Cross Validated! From the two plots you posted, your categories appear to have considerable overlap on the features. If that is the case, then no machine learning method can reliably separate them. // The graphs make it look like there are many more instances of the green category than the other two. Is that correct? If so, do you wind up classifying most of the observations as green? $\endgroup$
    – Dave
    May 26 at 3:26
  • $\begingroup$ I uploaded my result when I used GaussianNB. The features overlap but we could see that their mean value are different. When I do a statistic test, they all showed significant differences between those classes. $\endgroup$ May 26 at 6:11

1 Answer 1


You are seeing the difference between statistical significance and practical significance.

I am with you that the distributions are different. They sure seem have different means. However, the distributions are not all that different. It should be hard to consistently classify the group. You might be able to do such classification occasionally and achieve an okay accuracy, but a high accuracy based on these two features seems impossible.

If you have this kind of overlap on the rest of your features, then you should not expect to score high accuracy values if you just use those features. While the distributions are statistically significantly different, the practical significance is not so great to result in high classification accuracy.

Where you might be able to make some progress is by considering multivariate distributions through interaction terms between the original features. For instance, consider the example below.

N <- 1000
X0 <- MASS::mvrnorm(
  c(0, 0),
    1, 0.9, 
    0.9, 1
    ), 2, 2)
X1 <- MASS::mvrnorm(
  c(0, 0),
    1, -0.9, 
    -0.9, 1
  ), 2, 2)
X <- rbind(X0, X1)
y <- c(
  rep(0, N), rep(1, N)
d <- data.frame(
  x1 = X[, 1],
  x2 = X[, 2],
  y = as.factor(y)
ggplot(d, aes(x = x1, y = x2, col = y)) +
L1 <- glm(y ~ X[, 1] + X[, 2], family = binomial)
L2 <- glm(y ~ X[, 1] + X[, 2] + X[, 1]:X[, 2], family = binomial)
r1 <- pROC::roc(y, predict(L1))$auc
r2 <- pROC::roc(y, predict(L2))$auc
print(r1) # I get 0.5248, so barely above chance
print(r2) # I get 0.9577, which is quite strong performance

enter image description here

In this example, the marginal distributions of the features are identical for the two categories (y), so predictions based on just the raw features cannot be accurate. However, looking at the plot, the joint distribution of the features are clearly different for the two categories. Considering the interaction leads to much stronger performance. A number of machine learning models consider these kinds of interactions internally without you having to explicitly program them, among them being random forests and neural networks.

A useful visualization for you might be something like a t-SNE plot. While t-SNE can fail to separate categories when they truly are separable and introduce separability when the categories are not, such a plot is suggestive of whether or not even considering interactions and joint distributions of the features will allow for accurate classifications by any model. It might be that, even if the (even marginal) distributions of the features are different, the categories simply are not that separable on the given features, and strong performance using these features is impossible.

Note the issues with accuracy as a performance metric, especially in a situation with imbalanced classes like you appear to have.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.