As part of my thesis, we wanted to investigate the ability of using z-scores as indicators for the homogeneity of clusters (or speaking more generally of partitions of a vector). After some modelling I believe this is not entirely possible and I would like to get some opinions on this.

Suppose I have some 2D data with one predictor c and my cluster assignment cl, which splits c into 5 groups of exactly 20 elements. There is little variation among the elements of c in every cluster and every cluster contains a distinct set of elements of c(i.e. cluster 1 contains mostly ones, cluster 2 mostly twos, a.s.o.). I then want to compute each indivdual cluster's z-score based on c as follows:

# library(data.table)
# dat<-as.data.table(dat)
meanCl<-dat[,mean(c),by=cl][,V1,]
mean0<-sapply(1:1000,function(x){
              mean(sample(dat[,c,],20))}
sd0<-sd(mean0)
mean0<-mean(mean0)
z<-(meanCl-mean0)/sd0

... i.e. I compute the mean of c for every subgroup in cl. I go on to compute the mean of 1000 clusters containing 20 random elements from the distribution of c. I then calculate every clusters' z-score by subtracting its mean from the mean of the random clusters' means and divide by the standard deviation in those means to get:

[1] -4.96401933 -4.05057547 -0.03142247  2.89159789  5.99730703

So even though all clusters are distinctly characterised in their internal distribution of c, the z-score for cluster 3 is basically 0, as its mean tends towards the mean of the sample distribution of c.

Typing this out, it makes sense to me to not be able to use a cluster's z-score as a predictor for its homogeneity, however, as intially stated, I would like some ideas on this.

As part of my thesis, we wanted to investigate the ability of using z-scores as indicators for the homogeneity of clusters (or speaking more generally of partitions of a vector). After some modelling I believe this is not entirely possible and I would like to get some opinions on this.

Suppose I have some 2D data with one predictor c and my cluster assignment cl, which splits c into 5 groups of exactly 20 elements. There is little variation among the elements of c in every cluster and every cluster contains a distinct set of elements of c(i.e. cluster 1 contains mostly ones, cluster 2 mostly twos, a.s.o.). I then want to compute each indivdual cluster's z-score based on c as follows:

# library(data.table)
# dat<-as.data.table(dat)
meanCl<-dat[,mean(c),by=cl][,V1,]
mean0<-sapply(1:1000,function(x){
              mean(sample(dat[,c,],20,replace=T))}
sd0<-sd(mean0)
mean0<-mean(mean0)
z<-(meanCl-mean0)/sd0

... i.e. I compute the mean of c for every subgroup in cl. I go on to compute the mean of 1000 clusters containing 20 random elements from the distribution of c. I then calculate every clusters' z-score by subtracting its mean from the mean of the random clusters' means and divide by the standard deviation in those means to get:

[1] -4.96401933 -4.05057547 -0.03142247  2.89159789  5.99730703

So even though all clusters are distinctly characterised in their internal distribution of c, the z-score for cluster 3 is basically 0, as its mean tends towards the mean of the sample distribution of c.

Typing this out, it makes sense to me to not be able to use a cluster's z-score as a predictor for its homogeneity, however, as intially stated, I would like some ideas on this.

As part of my thesis, we wanted to investigate the ability of using z-scores as indicators for the homogeneity of clusters (or speaking more generally of partitions of a vector). After some modelling I believe this is not entirely possible and I would like to get some opinions on this.

Suppose I have some 2D data with one predictor c and my cluster assignment cl, which splits c into 5 groups of exactly 20 elements. There is little variation among the elements of c in every cluster and every cluster contains a distinct set of elements of c(i.e. cluster 1 contains mostly ones, cluster 2 mostly twos, a.s.o.). I then want to compute each indivdual cluster's z-score based on c as follows:

# library(data.table)
# dat<-as.data.table(dat)
meanCl<-dat[,mean(c),by=cl][,V1,]
mean0<-sapply(1:1000,function(x){
              mean(sample(dat[,c,],20))}
sd0<-sd(mean0)
mean0<-mean(mean0)
z<-(meanCl-mean0)/sd0

... i.e. I compute the mean of c for every subgroup in cl. I go on to compute the mean of 1000 clusters containing 20 random elements from the distribution of c. I then calculate every clusters' z-score by subtracting its mean from the mean of the random clusters' means and divide by the standard deviation in those means to get:

[1] -4.96401933 -4.05057547 -0.03142247  2.89159789  5.99730703

So even though all clusters are distinctly characterised in their internal distribution of c, the z-score for cluster 3 is basically 0, as its mean tends towards the mean of the sample distribution of c.

Typing this out, it makes sense to me to not be able to use a cluster's z-score as a predictor for its uniqueness, however, as intially stated, I would like some ideas on this.

As part of my thesis, we wanted to investigate the ability of using z-scores as indicators for the homogeneity of clusters (or speaking more generally of partitions of a vector). After some modelling I believe this is not entirely possible and I would like to get some opinions on this.

Suppose I have some 2D data with one predictor c and my cluster assignment cl, which splits c into 5 groups of exactly 20 elements. There is little variation among the elements of c in every cluster and every cluster contains a distinct set of elements of c(i.e. cluster 1 contains mostly ones, cluster 2 mostly twos, a.s.o.). I then want to compute each indivdual cluster's z-score based on c as follows:

# library(data.table)
# dat<-as.data.table(dat)
meanCl<-dat[,mean(c),by=cl][,V1,]
mean0<-sapply(1:1000,function(x){
              mean(sample(dat[,c,],20))}
sd0<-sd(mean0)
mean0<-mean(mean0)
z<-(meanCl-mean0)/sd0

... i.e. I compute the mean of c for every subgroup in cl. I go on to compute the mean of 1000 clusters containing 20 random elements from the distribution of c. I then calculate every clusters' z-score by subtracting its mean from the mean of the random clusters' means and divide by the standard deviation in those means to get:

[1] -4.96401933 -4.05057547 -0.03142247  2.89159789  5.99730703

So even though all clusters are distinctly characterised in their internal distribution of c, the z-score for cluster 3 is basically 0, as its mean tends towards the mean of the sample distribution of c.

Typing this out, it makes sense to me to not be able to use a cluster's z-score as a predictor for its homogeneity, however, as intially stated, I would like some ideas on this.