Features differ between classes Good evening everyone.
Regarding the topic related to Sparse Clustering (for example K-Means).
For example, in "Witten DM, Tibshirani R. A framework for feature selection in clustering" the authors say
"In Figure 2, we apply this method to a simple example with 6 equally-sized classes, where n = 120, p = 2000, and 200 features differ between classes"
What does it mean that 200 features differ between classes? How does this translate to R / Python / Matlab?
REF:
Witten DM, Tibshirani R. A framework for feature selection in clustering. J Am Stat Assoc. 2010 Jun 1; 105 (490): 713-726. doi: 10.1198 / jasa.2010.tm09415. PMID: 20811510; PMCID: PMC2930825.
LINK TO REF:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2930825/
 A: It means that these features (jointly, not necessarily marginally) have some distribution for class A that differs from the distribution for class B.
It could be that the marginal means all differ between the classes.
It could be that the marginal variances differ between the classes.
It could be that one marginal mean is different and the difference is in variance for the rest of the features.
It could be that the marginal distributions are identical but the correlation between the features is different in class A than in class B (e.g., independent in class A but correlated in class B).
Let's give some examples, all of which I simulated in R.
First, we set up the simulation.
library(ggplot2)
library(MASS)
set.seed(2022)
N <- 1000

NOW let's simulate and plot three independent variables, two of which differ in means between the classes.
# Three variables, two of which differ in means between the classes
#
cA  <- rep("A", N)
cB  <- rep("B", N)
F1A <- rnorm(N, 0, 1) # N(0, 1) on F0 class A
F1B <- rnorm(N, 4, 1) # N(4, 1) on F0 class B
F2A <- rnorm(N, 0, 1) # N(0, 1) on F1 class A
F2B <- rnorm(N, 4, 1) # N(4, 1) on F1 class B
F3A <- rnorm(N, 0, 1) # N(0, 1) on F2 class A
F3B <- rnorm(N, 0, 1) # N(0, 1) on F2 class B
d <- data.frame(
  group = rep(c(cA, cB), 3),
  value = c(F1A, F1B, F2A, F2B, F3A, F3B),
  feature = rep(c("Feature_1", "Feature_2", "Feature_3"), rep(2*N, 3))
)
ggplot(d, aes(x = value, fill = group)) +
  geom_density(alpha = 0.5) +
  facet_grid(~feature) +
  theme_bw()


NEXT, let's simulate and plot three independent variables, one of which differs in means between classes and one of which differs in variances.

FINALLY, let's simulate and plot three variables, all with equal margins between the classes but with a differnt correlation structure on features 1 and 2 between the classes.
# Three variables, all with equal margins between the 
# classes but with a differnt correlation structure 
# on features 1 and 2 between the classes
#
cA  <- rep("A", N)
cB  <- rep("B", N)
F_A <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.9, 0.9, 1), 2, 2))
F_B <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, -0.9, -0.9, 1), 2, 2))
#
# We have positive correlation in A and negative correlation in B,
# but the margins all are N(0, 1)
#
F1A <- F_A[, 1]
F1B <- F_B[, 1]
F2A <- F_A[, 2]
F2B <- F_B[, 2]
F3A <- rnorm(N, 0, 1) # N(0, 1) on F2 class A
F3B <- rnorm(N, 0, 1) # N(0, 1) on F2 class B
d <- data.frame(
  group = rep(c(cA, cB), 3),
  value = c(F1A, F1B, F2A, F2B, F3A, F3B),
  feature = rep(c("Feature_1", "Feature_2", "Feature_3"), rep(2*N, 3))
)
ggplot(d, aes(x = value, fill = group)) +
  geom_density(alpha = 0.5) +
  facet_grid(~feature) +
  theme_bw()


THE marginal distributions of all three features are the same! However, plotting the values of feature $1$ and feature $2$ in a scatterplot reveals there to be a difference.

dxy <- data.frame(
  group = rep(c(cA, cB), 2),
  Feature_1 = c(F1A, F1B),
  Feature_2 = c(F2A, F2B)
)
ggplot(dxy, aes(x = Feature_1, y = Feature_2, col = group)) +
  geom_point() +
  theme_bw()


CLARIFICATION
In my answer, $X$ has three components (features), $x_1$, $x_2$, and $x_3$, as in writing a regression like $y=x_1+x_2+x_3$. In the first two cases, $x_1$ has a different distribution for class $A$ than for class $B$, and ditto for $x_2$. In the final case, the marginal distributions are the same in classes $A$ and $B$, but the joint distribution of $(x_1,x_2)$ has positive correlation for class $A$, while the joint distribution of $(x_1,x_2)$ has negative correlation for class $B$.
