Revisions to Compute BIC clustering criterion (to validate clusters after K-means)

added 2 characters in body

Source Link

edited Apr 23, 2020 at 6:58

269
3
7

To calculate the BIC for the kmeans results, I have tested the following methods:

The following formula is from: [ref2]

The r code for above formula is:

  k3 <- kmeans(mt,3)
  intra.mean <- mean(k3$within)
  k10 <- kmeans(mt,10)
  centers <- k10$centers
  BIC <- function(mt,cls,intra.mean,centers){
    x.centers <- apply(centers,2,function(y){
      as.numeric(y)[cls]
    })
    sum1 <- sum(((mt-x.centers)/intra.mean)**2)
    sum1 + NCOL(mt)*length(unique(cls))*log(NROW(mt))
  }
#

the problem is when i using the above r code, the calculated BIC was monotone increasing. what's the reason?

[ref2] Ramsey, S. A., et al. (2008). "Uncovering a macrophage transcriptional program by integrating evidence from motif scanning and expression dynamics." PLoS Comput Biol 4(3): e1000021.

2. I have used the new formula from https://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r

    BIC2 <- function(fit){
    m = ncol(fit$centers)
    n = length(fit$cluster)
    k = nrow(fit$centers)
    D = fit$tot.withinss
    return(data.frame(AIC = D + 2*m*k,
                      BIC = D + log(n)*m*k))
    }

This method given the lowest BIC value at cluster number 155.

using @ttnphns provided method, the corresponding R code as listed below. However, the problem is what the difference between Vc and V? And how to calculate the element-wise multiplication for two vectors with different length?

 BIC3 <- function(fit,mt){
 Nc <- as.matrix(as.numeric(table(fit$cluster)),nc=1)
Vc <- apply(mt,2,function(x){
    tapply(x,fit$cluster,var)
  })
 V <- matrix(rep(apply(mt,2,function(x){
 var(x)
 }),length(Nc)),byrow=TRUE,nrow=length(Nc))
 LL = -Nc * colSums( log(Vc + V)/2 ) ##how to calculate this? elementa-wise multiplication for two vectors with different length?
 BIC = -2 * rowSums(LL) + 2*K*P * log(NRoW(mt))
 return(BIC)
 }

To calculate the BIC for the kmeans results, I have tested the following methods:

The following formula is from: [ref2]

The r code for above formula is:

  k3 <- kmeans(mt,3)
  intra.mean <- mean(k3$within)
  k10 <- kmeans(mt,10)
  centers <- k10$centers
  BIC <- function(mt,cls,intra.mean,centers){
    x.centers <- apply(centers,2,function(y){
      as.numeric(y)[cls]
    })
    sum1 <- sum(((mt-x.centers)/intra.mean)**2)
    sum1 + NCOL(mt)*length(unique(cls))*log(NROW(mt))
  }
#

the problem is when i using the above r code, the calculated BIC was monotone increasing. what's the reason?

[ref2] Ramsey, S. A., et al. (2008). "Uncovering a macrophage transcriptional program by integrating evidence from motif scanning and expression dynamics." PLoS Comput Biol 4(3): e1000021.

2. I have used the new formula from https://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r

    BIC2 <- function(fit){
    m = ncol(fit$centers)
    n = length(fit$cluster)
    k = nrow(fit$centers)
    D = fit$tot.withinss
    return(data.frame(AIC = D + 2*m*k,
                      BIC = D + log(n)*m*k))
  }

This method given the lowest BIC value at cluster number 155.

using @ttnphns provided method, the corresponding R code as listed below. However, the problem is what the difference between Vc and V? And how to calculate the element-wise multiplication for two vectors with different length?

 BIC3 <- function(fit,mt){
 Nc <- as.matrix(as.numeric(table(fit$cluster)),nc=1)
Vc <- apply(mt,2,function(x){
    tapply(x,fit$cluster,var)
  })
 V <- matrix(rep(apply(mt,2,function(x){
 var(x)
 }),length(Nc)),byrow=TRUE,nrow=length(Nc))
 LL = -Nc * colSums( log(Vc + V)/2 ) ##how to calculate this? elementa-wise multiplication for two vectors with different length?
 BIC = -2 * rowSums(LL) + 2*K*P * log(NRoW(mt))
 return(BIC)
 }

To calculate the BIC for the kmeans results, I have tested the following methods:

The following formula is from: [ref2]

The r code for above formula is:

  k3 <- kmeans(mt,3)
  intra.mean <- mean(k3$within)
  k10 <- kmeans(mt,10)
  centers <- k10$centers
  BIC <- function(mt,cls,intra.mean,centers){
    x.centers <- apply(centers,2,function(y){
      as.numeric(y)[cls]
    })
    sum1 <- sum(((mt-x.centers)/intra.mean)**2)
    sum1 + NCOL(mt)*length(unique(cls))*log(NROW(mt))
  }
#

the problem is when i using the above r code, the calculated BIC was monotone increasing. what's the reason?

[ref2] Ramsey, S. A., et al. (2008). "Uncovering a macrophage transcriptional program by integrating evidence from motif scanning and expression dynamics." PLoS Comput Biol 4(3): e1000021.

2. I have used the new formula from https://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r

    BIC2 <- function(fit){
    m = ncol(fit$centers)
    n = length(fit$cluster)
    k = nrow(fit$centers)
    D = fit$tot.withinss
    return(data.frame(AIC = D + 2*m*k,
                      BIC = D + log(n)*m*k))
    }

This method given the lowest BIC value at cluster number 155.

using @ttnphns provided method, the corresponding R code as listed below. However, the problem is what the difference between Vc and V? And how to calculate the element-wise multiplication for two vectors with different length?

 BIC3 <- function(fit,mt){
 Nc <- as.matrix(as.numeric(table(fit$cluster)),nc=1)
Vc <- apply(mt,2,function(x){
    tapply(x,fit$cluster,var)
  })
 V <- matrix(rep(apply(mt,2,function(x){
 var(x)
 }),length(Nc)),byrow=TRUE,nrow=length(Nc))
 LL = -Nc * colSums( log(Vc + V)/2 ) ##how to calculate this? elementa-wise multiplication for two vectors with different length?
 BIC = -2 * rowSums(LL) + 2*K*P * log(NRoW(mt))
 return(BIC)
 }

replaced http://stackoverflow.com/ with https://stackoverflow.com/

Source Link

edited May 23, 2017 at 12:39

Community Bot

1

To calculate the BIC for the kmeans results, I have tested the following methods:

The following formula is from: [ref2]

The r code for above formula is:

  k3 <- kmeans(mt,3)
  intra.mean <- mean(k3$within)
  k10 <- kmeans(mt,10)
  centers <- k10$centers
  BIC <- function(mt,cls,intra.mean,centers){
    x.centers <- apply(centers,2,function(y){
      as.numeric(y)[cls]
    })
    sum1 <- sum(((mt-x.centers)/intra.mean)**2)
    sum1 + NCOL(mt)*length(unique(cls))*log(NROW(mt))
  }
#

the problem is when i using the above r code, the calculated BIC was monotone increasing. what's the reason?

[ref2] Ramsey, S. A., et al. (2008). "Uncovering a macrophage transcriptional program by integrating evidence from motif scanning and expression dynamics." PLoS Comput Biol 4(3): e1000021.

2. I have used the new formula from http://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r https://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r

    BIC2 <- function(fit){
    m = ncol(fit$centers)
    n = length(fit$cluster)
    k = nrow(fit$centers)
    D = fit$tot.withinss
    return(data.frame(AIC = D + 2*m*k,
                      BIC = D + log(n)*m*k))
  }

This method given the lowest BIC value at cluster number 155.

using @ttnphns provided method, the corresponding R code as listed below. However, the problem is what the difference between Vc and V? And how to calculate the element-wise multiplication for two vectors with different length?

 BIC3 <- function(fit,mt){
 Nc <- as.matrix(as.numeric(table(fit$cluster)),nc=1)
Vc <- apply(mt,2,function(x){
    tapply(x,fit$cluster,var)
  })
 V <- matrix(rep(apply(mt,2,function(x){
 var(x)
 }),length(Nc)),byrow=TRUE,nrow=length(Nc))
 LL = -Nc * colSums( log(Vc + V)/2 ) ##how to calculate this? elementa-wise multiplication for two vectors with different length?
 BIC = -2 * rowSums(LL) + 2*K*P * log(NRoW(mt))
 return(BIC)
 }

To calculate the BIC for the kmeans results, I have tested the following methods:

The following formula is from: [ref2]

The r code for above formula is:

  k3 <- kmeans(mt,3)
  intra.mean <- mean(k3$within)
  k10 <- kmeans(mt,10)
  centers <- k10$centers
  BIC <- function(mt,cls,intra.mean,centers){
    x.centers <- apply(centers,2,function(y){
      as.numeric(y)[cls]
    })
    sum1 <- sum(((mt-x.centers)/intra.mean)**2)
    sum1 + NCOL(mt)*length(unique(cls))*log(NROW(mt))
  }
#

the problem is when i using the above r code, the calculated BIC was monotone increasing. what's the reason?

[ref2] Ramsey, S. A., et al. (2008). "Uncovering a macrophage transcriptional program by integrating evidence from motif scanning and expression dynamics." PLoS Comput Biol 4(3): e1000021.

2. I have used the new formula from http://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r

    BIC2 <- function(fit){
    m = ncol(fit$centers)
    n = length(fit$cluster)
    k = nrow(fit$centers)
    D = fit$tot.withinss
    return(data.frame(AIC = D + 2*m*k,
                      BIC = D + log(n)*m*k))
  }

This method given the lowest BIC value at cluster number 155.

using @ttnphns provided method, the corresponding R code as listed below. However, the problem is what the difference between Vc and V? And how to calculate the element-wise multiplication for two vectors with different length?

 BIC3 <- function(fit,mt){
 Nc <- as.matrix(as.numeric(table(fit$cluster)),nc=1)
Vc <- apply(mt,2,function(x){
    tapply(x,fit$cluster,var)
  })
 V <- matrix(rep(apply(mt,2,function(x){
 var(x)
 }),length(Nc)),byrow=TRUE,nrow=length(Nc))
 LL = -Nc * colSums( log(Vc + V)/2 ) ##how to calculate this? elementa-wise multiplication for two vectors with different length?
 BIC = -2 * rowSums(LL) + 2*K*P * log(NRoW(mt))
 return(BIC)
 }

To calculate the BIC for the kmeans results, I have tested the following methods:

The following formula is from: [ref2]

The r code for above formula is:

  k3 <- kmeans(mt,3)
  intra.mean <- mean(k3$within)
  k10 <- kmeans(mt,10)
  centers <- k10$centers
  BIC <- function(mt,cls,intra.mean,centers){
    x.centers <- apply(centers,2,function(y){
      as.numeric(y)[cls]
    })
    sum1 <- sum(((mt-x.centers)/intra.mean)**2)
    sum1 + NCOL(mt)*length(unique(cls))*log(NROW(mt))
  }
#

the problem is when i using the above r code, the calculated BIC was monotone increasing. what's the reason?

[ref2] Ramsey, S. A., et al. (2008). "Uncovering a macrophage transcriptional program by integrating evidence from motif scanning and expression dynamics." PLoS Comput Biol 4(3): e1000021.

2. I have used the new formula from https://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r

    BIC2 <- function(fit){
    m = ncol(fit$centers)
    n = length(fit$cluster)
    k = nrow(fit$centers)
    D = fit$tot.withinss
    return(data.frame(AIC = D + 2*m*k,
                      BIC = D + log(n)*m*k))
  }

This method given the lowest BIC value at cluster number 155.

using @ttnphns provided method, the corresponding R code as listed below. However, the problem is what the difference between Vc and V? And how to calculate the element-wise multiplication for two vectors with different length?

 BIC3 <- function(fit,mt){
 Nc <- as.matrix(as.numeric(table(fit$cluster)),nc=1)
Vc <- apply(mt,2,function(x){
    tapply(x,fit$cluster,var)
  })
 V <- matrix(rep(apply(mt,2,function(x){
 var(x)
 }),length(Nc)),byrow=TRUE,nrow=length(Nc))
 LL = -Nc * colSums( log(Vc + V)/2 ) ##how to calculate this? elementa-wise multiplication for two vectors with different length?
 BIC = -2 * rowSums(LL) + 2*K*P * log(NRoW(mt))
 return(BIC)
 }

deleted 8 characters in body

Source Link

edited Nov 24, 2015 at 23:03

pengchy

269
3
7

To calculate the BIC for the kmeans results, I have tested the following methods:

The following formula is from: [ref2]

The r code for above formula is:

  k3 <- kmeans(mt,3)
  intra.mean <- mean(k3$within)
  k10 <- kmeans(mt,10)
  centers <- k10$centers
  BIC <- function(mt,cls,intra.mean,centers){
    x.centers <- apply(centers,2,function(y){
      as.numeric(y)[cls]
    })
    sum1 <- sum(((mt-x.centers)/intra.mean)**2)
    sum1 + NCOL(mt)*length(unique(cls))*log(NROW(mt))
  }
#

the problem is when i using the above r code, the calculated BIC was monotone increasing. what's the reason?

[ref2] Ramsey, S. A., et al. (2008). "Uncovering a macrophage transcriptional program by integrating evidence from motif scanning and expression dynamics." PLoS Comput Biol 4(3): e1000021.

2. I have used the new formula from http://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r

    BIC2 <- function(fit){
    m = ncol(fit$centers)
    n = length(fit$cluster)
    k = nrow(fit$centers)
    D = fit$tot.withinss
    return(data.frame(AIC = D + 2*m*k,
                      BIC = D + log(n)*m*k))
  }

This method given the lowest BIC value at cluster number 155.

using @ttnphns provided method, the corresponding R code as listed below. However, the problem is what the difference between Vc and V? And how to calculate the element-wise multiplication for two vectors with different length?

 BIC3 <- function(fit,mt){
 Nc <- as.matrix(as.numeric(table(fit$cluster)),nc=1)
    Vc <- apply(mt,2,function(x){
    tapply(x,fit$$cluster)),nc=1)
Vc <- apply(mt,2,function(x){
    tapply(x,fit$cluster,var)
  })
 V <- matrix(rep(apply(mt,2,function(x){
 var(x)
 }),length(Nc)),byrow=TRUE,nrow=length(Nc))
 LL = -Nc * colSums( log(Vc + V)/2 ) ##how to calculate this? elementa-wise multiplication for two vectors with different length?
 BIC = -2 * rowSums(LL) + 2*K*P * log(NRoW(mt))
 return(BIC)
 }

To calculate the BIC for the kmeans results, I have tested the following methods:

The following formula is from: [ref2]

The r code for above formula is:

  k3 <- kmeans(mt,3)
  intra.mean <- mean(k3$within)
  k10 <- kmeans(mt,10)
  centers <- k10$centers
  BIC <- function(mt,cls,intra.mean,centers){
    x.centers <- apply(centers,2,function(y){
      as.numeric(y)[cls]
    })
    sum1 <- sum(((mt-x.centers)/intra.mean)**2)
    sum1 + NCOL(mt)*length(unique(cls))*log(NROW(mt))
  }
#

the problem is when i using the above r code, the calculated BIC was monotone increasing. what's the reason?

[ref2] Ramsey, S. A., et al. (2008). "Uncovering a macrophage transcriptional program by integrating evidence from motif scanning and expression dynamics." PLoS Comput Biol 4(3): e1000021.

2. I have used the new formula from http://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r

    BIC2 <- function(fit){
    m = ncol(fit$centers)
    n = length(fit$cluster)
    k = nrow(fit$centers)
    D = fit$tot.withinss
    return(data.frame(AIC = D + 2*m*k,
                      BIC = D + log(n)*m*k))
  }

This method given the lowest BIC value at cluster number 155.

using @ttnphns provided method, the corresponding R code as listed below. However, the problem is what the difference between Vc and V? And how to calculate the element-wise multiplication for two vectors with different length?

 BIC3 <- function(fit,mt){
 Nc <- as.matrix(as.numeric(table(fit$cluster)),nc=1)
    Vc <- apply(mt,2,function(x){
    tapply(x,fit$cluster,var)
  })
 V <- matrix(rep(apply(mt,2,function(x){
 var(x)
 }),length(Nc)),byrow=TRUE,nrow=length(Nc))
 LL = -Nc * colSums( log(Vc + V)/2 ) ##how to calculate this? elementa-wise multiplication for two vectors with different length?
 BIC = -2 * rowSums(LL) + 2*K*P * log(NRoW(mt))
 return(BIC)
 }

To calculate the BIC for the kmeans results, I have tested the following methods:

The following formula is from: [ref2]

The r code for above formula is:

  k3 <- kmeans(mt,3)
  intra.mean <- mean(k3$within)
  k10 <- kmeans(mt,10)
  centers <- k10$centers
  BIC <- function(mt,cls,intra.mean,centers){
    x.centers <- apply(centers,2,function(y){
      as.numeric(y)[cls]
    })
    sum1 <- sum(((mt-x.centers)/intra.mean)**2)
    sum1 + NCOL(mt)*length(unique(cls))*log(NROW(mt))
  }
#

the problem is when i using the above r code, the calculated BIC was monotone increasing. what's the reason?

[ref2] Ramsey, S. A., et al. (2008). "Uncovering a macrophage transcriptional program by integrating evidence from motif scanning and expression dynamics." PLoS Comput Biol 4(3): e1000021.

2. I have used the new formula from http://stackoverflow.com/questions/15839774/how-to-calculate-bic-for-k-means-clustering-in-r

    BIC2 <- function(fit){
    m = ncol(fit$centers)
    n = length(fit$cluster)
    k = nrow(fit$centers)
    D = fit$tot.withinss
    return(data.frame(AIC = D + 2*m*k,
                      BIC = D + log(n)*m*k))
  }

This method given the lowest BIC value at cluster number 155.

using @ttnphns provided method, the corresponding R code as listed below. However, the problem is what the difference between Vc and V? And how to calculate the element-wise multiplication for two vectors with different length?

 BIC3 <- function(fit,mt){
 Nc <- as.matrix(as.numeric(table(fit$cluster)),nc=1)
Vc <- apply(mt,2,function(x){
    tapply(x,fit$cluster,var)
  })
 V <- matrix(rep(apply(mt,2,function(x){
 var(x)
 }),length(Nc)),byrow=TRUE,nrow=length(Nc))
 LL = -Nc * colSums( log(Vc + V)/2 ) ##how to calculate this? elementa-wise multiplication for two vectors with different length?
 BIC = -2 * rowSums(LL) + 2*K*P * log(NRoW(mt))
 return(BIC)
 }

added 14 characters in body

Source Link

edited Nov 24, 2015 at 8:33

pengchy

269
3
7

Loading

added 768 characters in body

Source Link

edited Nov 24, 2015 at 4:17

pengchy

269
3
7

Loading

added 768 characters in body

Source Link

edited Nov 24, 2015 at 4:11

pengchy

269
3
7

Loading

added 768 characters in body

Source Link

edited Nov 24, 2015 at 4:05

pengchy

269
3
7

Loading

added 531 characters in body

Source Link

edited Nov 24, 2015 at 3:30

pengchy

269
3
7

Loading

added 200 characters in body

Source Link

edited Nov 24, 2015 at 2:54

pengchy

269
3
7

Loading

added 428 characters in body

Source Link

edited Nov 24, 2015 at 2:28

pengchy

269
3
7

Loading

Source Link

created Nov 23, 2015 at 8:28

pengchy

269
3
7

Loading

Stack Exchange Network

Return to Answer