1
$\begingroup$

How can I compute a distance matrix and cluster analysis on data with grouped measurements?

For example, here's a table of 10 samples, A to J. Each sample has nine groups of measurements (group_2, group_3 ... group_10). The grouped measurements are not replicates, they are of closely related variables. In these data, for a given sample, the values within a group of variables sum to 1. So the values are a distribution within each group of variables.

I want to know which samples cluster together based on these measurements. I do not want the measurements to be treated independently in the analysis, like hclust(dist(...) does.

Looking around for ideas, I saw here that one option might be to 'use Mahalanobis distance between the centroids, with within-group covariance as the variance matrix'. But I'm not sure how to implement that in R.

Here's a look at some example data, the measurements are grouped by the column names, so that 'group_2' labels the 1st group of measurements, etc. (there is no order in the measurement groups):

# A tibble: 10 × 67
   sample group_2_11 group_2_12 group_2_13 group_2_14 group_3_9 group_3_10 group_3_11 group_3_12
    <chr>      <dbl>      <dbl>      <dbl>      <dbl>     <dbl>      <dbl>      <dbl>      <dbl>
1       A         NA         NA         NA         NA        NA    1.00000         NA         NA
2       B         NA         NA         NA         NA        NA    0.07143    0.17857    0.35714
3       C         NA         NA         NA         NA        NA         NA         NA         NA
4       D         NA         NA         NA         NA        NA    0.09091    0.27273    0.31818
5       E         NA         NA         NA         NA   0.07143    0.67857    0.25000         NA
6       F    0.06250    0.50000     0.2500     0.1875        NA         NA         NA         NA
7       G    0.28571    0.53968     0.1746         NA   0.03571    0.28571         NA    0.14286
8       H         NA         NA         NA         NA        NA    0.07317    0.21951    0.39024
9       I         NA         NA         NA         NA        NA    0.07143    0.17857    0.46429
10      J         NA         NA         NA         NA        NA    0.04167    0.28125    0.41667

And here is the output from dput for easy reuse:

structure(list(sample = c("A", "B", "C", "D", "E", "F", "G", 
"H", "I", "J"), group_2_11 = c(NA, NA, NA, NA, NA, 0.0625, 0.28571, 
NA, NA, NA), group_2_12 = c(NA, NA, NA, NA, NA, 0.5, 0.53968, 
NA, NA, NA), group_2_13 = c(NA, NA, NA, NA, NA, 0.25, 0.1746, 
NA, NA, NA), group_2_14 = c(NA, NA, NA, NA, NA, 0.1875, NA, NA, 
NA, NA), group_3_9 = c(NA, NA, NA, NA, 0.07143, NA, 0.03571, 
NA, NA, NA), group_3_10 = c(1, 0.07143, NA, 0.09091, 0.67857, 
NA, 0.28571, 0.07317, 0.07143, 0.04167), group_3_11 = c(NA, 0.17857, 
NA, 0.27273, 0.25, NA, NA, 0.21951, 0.17857, 0.28125), group_3_12 = c(NA, 
0.35714, NA, 0.31818, NA, NA, 0.14286, 0.39024, 0.46429, 0.41667
), group_3_13 = c(NA, 0.25, NA, 0.22727, NA, NA, 0.33929, 0.14634, 
0.21429, 0.21875), group_3_14 = c(NA, 0.14286, NA, NA, NA, NA, 
0.19643, 0.12195, 0.07143, 0.04167), group_3_15 = c(NA, NA, NA, 
0.04545, NA, NA, NA, 0.04878, NA, NA), group_3_16 = c(NA, NA, 
NA, 0.04545, NA, NA, NA, NA, NA, NA), group_4_13 = c(NA, NA, 
NA, NA, NA, 0.075, NA, NA, NA, NA), group_4_14 = c(NA, 0.25, 
0.55172, NA, 0.34483, 0.3, NA, 0.36842, NA, NA), group_4_15 = c(NA, 
0.65, 0.37931, NA, 0.58621, 0.575, NA, 0.63158, NA, NA), group_4_16 = c(NA, 
0.1, 0.06897, NA, 0.06897, 0.05, NA, NA, NA, NA), group_5_13 = c(NA, 
NA, 0.11111, NA, 0.04348, NA, NA, NA, NA, NA), group_5_14 = c(NA, 
NA, 0.69444, NA, NA, 0.25, NA, 0.24545, 0.04082, NA), group_5_15 = c(NA, 
NA, 0.19444, NA, 0.21739, 0.54167, NA, 0.47273, 0.36735, 0.47826
), group_5_16 = c(NA, NA, NA, NA, 0.45652, 0.20833, NA, 0.21818, 
0.32653, 0.30435), group_5_17 = c(NA, NA, NA, NA, 0.23913, NA, 
NA, 0.06364, 0.26531, 0.21739), group_5_18 = c(NA, NA, NA, NA, 
0.04348, NA, NA, NA, NA, NA), group_6_11 = c(NA, 0.04545, NA, 
NA, NA, NA, NA, NA, NA, 0.03333), group_6_12 = c(NA, 0.04545, 
NA, NA, NA, NA, NA, NA, NA, 0.15556), group_6_13 = c(NA, 0.63636, 
0.14286, NA, NA, NA, NA, 0.10714, NA, 0.15556), group_6_14 = c(NA, 
0.27273, 0.85714, NA, NA, NA, NA, 0.78571, NA, 0.53333), group_6_15 = c(NA, 
NA, NA, NA, NA, NA, NA, 0.10714, NA, 0.1), group_6_16 = c(NA, 
NA, NA, NA, NA, NA, NA, NA, NA, 0.02222), group_7_9 = c(NA, 0.16667, 
NA, NA, NA, NA, 0.21622, NA, NA, NA), group_7_10 = c(NA, 0.83333, 
NA, NA, NA, NA, 0.78378, NA, NA, NA), group_7_16 = c(NA_real_, 
NA_real_, NA_real_, NA_real_, NA_real_, NA_real_, NA_real_, NA_real_, 
NA_real_, NA_real_), group_7_17 = c(NA_real_, NA_real_, NA_real_, 
NA_real_, NA_real_, NA_real_, NA_real_, NA_real_, NA_real_, NA_real_
), group_7_19 = c(NA_real_, NA_real_, NA_real_, NA_real_, NA_real_, 
NA_real_, NA_real_, NA_real_, NA_real_, NA_real_), group_8_10 = c(0.32836, 
0.1028, 0.14151, 0.19048, 0.17143, 0.09804, 0.03704, 0.11321, 
0.08696, NA), group_8_11 = c(0.52239, 0.4486, 0.51887, 0.80952, 
0.45714, 0.35294, 0.41975, 0.50943, 0.91304, NA), group_8_12 = c(0.14925, 
0.33645, 0.23585, NA, 0.28571, 0.33333, 0.39506, 0.28302, NA, 
NA), group_8_13 = c(NA, 0.08411, 0.06604, NA, 0.08571, 0.15686, 
0.12346, 0.09434, NA, NA), group_8_14 = c(NA, 0.02804, 0.03774, 
NA, NA, 0.05882, 0.02469, NA, NA, NA), group_9_22 = c(NA, NA, 
NA, NA, NA, NA, 0.05882, NA, NA, NA), group_9_23 = c(NA, NA, 
NA, NA, NA, NA, NA, NA, 0.09524, NA), group_9_26 = c(NA, NA, 
NA, NA, NA, NA, 0.08824, 0.05263, NA, NA), group_9_27 = c(NA, 
NA, NA, NA, NA, NA, 0.11765, NA, NA, NA), group_9_28 = c(NA, 
NA, NA, NA, NA, NA, 0.08824, 0.10526, 0.09524, NA), group_9_29 = c(NA, 
NA, NA, NA, NA, NA, 0.14706, 0.15789, 0.19048, NA), group_9_30 = c(NA, 
NA, NA, NA, NA, NA, 0.20588, 0.15789, 0.28571, NA), group_9_31 = c(NA, 
NA, NA, NA, NA, NA, 0.20588, 0.21053, 0.2381, NA), group_9_32 = c(NA, 
NA, NA, NA, NA, NA, 0.08824, 0.07895, 0.09524, NA), group_9_33 = c(NA, 
NA, NA, NA, NA, NA, NA, 0.18421, NA, NA), group_9_36 = c(NA, 
NA, NA, NA, NA, NA, NA, 0.05263, NA, NA), group_10_9 = c(NA, 
NA, NA, NA, NA, NA, NA, 0.01064, NA, NA), group_10_10 = c(NA, 
NA, NA, NA, NA, 0.02299, NA, NA, NA, NA), group_10_11 = c(NA, 
NA, NA, NA, NA, NA, NA, 0.01418, NA, NA), group_10_12 = c(NA, 
NA, NA, 0.04706, 0.03608, NA, 0.06015, 0.01773, NA, NA), group_10_13 = c(NA, 
NA, NA, 0.03529, 0.05155, 0.06897, 0.08271, 0.02128, NA, NA), 
    group_10_14 = c(NA, NA, NA, 0.10588, 0.11856, 0.06897, 0.15789, 
    0.07447, NA, NA), group_10_15 = c(NA, NA, NA, 0.02353, 0.14948, 
    0.12644, 0.12782, 0.11702, NA, NA), group_10_16 = c(NA, 0.07143, 
    NA, 0.10588, 0.09794, 0.03448, 0.03759, 0.07801, NA, NA), 
    group_10_17 = c(NA, 0.09524, NA, 0.04706, 0.05155, 0.05747, 
    0.02256, 0.07092, NA, NA), group_10_18 = c(NA, 0.16667, NA, 
    0.04706, 0.06701, 0.09195, 0.05263, 0.06738, NA, NA), group_10_19 = c(NA, 
    0.11905, NA, 0.08235, 0.10825, 0.11494, 0.11278, 0.0922, 
    NA, NA), group_10_20 = c(NA, 0.2381, NA, 0.16471, 0.12887, 
    0.17241, 0.11278, 0.15248, NA, NA), group_10_21 = c(NA, 0.14286, 
    NA, 0.07059, 0.08247, 0.12644, 0.09774, 0.07447, NA, NA), 
    group_10_22 = c(NA, 0.04762, NA, 0.11765, 0.07732, 0.03448, 
    0.08271, 0.09929, NA, NA), group_10_23 = c(NA, 0.07143, NA, 
    0.07059, 0.02062, 0.02299, NA, 0.04965, NA, NA), group_10_24 = c(NA, 
    0.04762, NA, 0.04706, NA, 0.02299, 0.03008, 0.0461, NA, NA
    ), group_10_25 = c(NA, NA, NA, 0.03529, 0.01031, 0.03448, 
    0.02256, 0.01418, NA, NA)), .Names = c("sample", "group_2_11", 
"group_2_12", "group_2_13", "group_2_14", "group_3_9", "group_3_10", 
"group_3_11", "group_3_12", "group_3_13", "group_3_14", "group_3_15", 
"group_3_16", "group_4_13", "group_4_14", "group_4_15", "group_4_16", 
"group_5_13", "group_5_14", "group_5_15", "group_5_16", "group_5_17", 
"group_5_18", "group_6_11", "group_6_12", "group_6_13", "group_6_14", 
"group_6_15", "group_6_16", "group_7_9", "group_7_10", "group_7_16", 
"group_7_17", "group_7_19", "group_8_10", "group_8_11", "group_8_12", 
"group_8_13", "group_8_14", "group_9_22", "group_9_23", "group_9_26", 
"group_9_27", "group_9_28", "group_9_29", "group_9_30", "group_9_31", 
"group_9_32", "group_9_33", "group_9_36", "group_10_9", "group_10_10", 
"group_10_11", "group_10_12", "group_10_13", "group_10_14", "group_10_15", 
"group_10_16", "group_10_17", "group_10_18", "group_10_19", "group_10_20", 
"group_10_21", "group_10_22", "group_10_23", "group_10_24", "group_10_25"
), row.names = c(NA, -10L), class = c("tbl_df", "tbl", "data.frame"
))

How can I compute the distances between each sample with respect to the grouped measurements?

Do I need to summarise each group of variables into a single variable somehow (take the mean?), or is there a method to incorporate the groups of variables into the distance measurements?

$\endgroup$
  • $\begingroup$ First, let me notice that you haven't propose any specific wish or idea about why and how you want to "group" the variables and why seeing them as just one group isn't to your liking. So far, your only intuitive and broad. Second, mind missings in your data - so much NAs is a great obstacle for any cluster analysis. $\endgroup$ – ttnphns Oct 23 '16 at 8:34
  • $\begingroup$ That's right, I'm curious to see if there are any general solutions to this type of problem. Yes, the NAs are not ideal, but R has functions that can perform clustering on data with NAs, so that's a minor issue here. $\endgroup$ – Ben Oct 23 '16 at 8:48
  • $\begingroup$ if there are any general solutions, R has functions that can perform clustering on data with NAs I'm sorry, Ben, to me these phrases are conspicuous that you don't have any specific plan of your analysis. The "problem" itself hasn't been described as a problem. Why chunking variables in subsets must be done? You say the values are a distribution within each group of variables So what? How is this a difficulty to your clustering plan? $\endgroup$ – ttnphns Oct 23 '16 at 9:01
  • $\begingroup$ No need to apologize if you don't understand the question, thanks for taking a look. $\endgroup$ – Ben Oct 23 '16 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.