6
$\begingroup$

I have a dataset with 15 variables. Some variables are numeric, continuous. Other variables are boolean, dichotomous (true/false). There's also one variable categorical, nominal.

str(df) 'data.frame': 30 obs. of 15 variables:
    nom : Factor w/ 3 levels "a","b","c": 1 1 1 1 1 1 1 1 1 1 ... 
    X1  : logi  FALSE TRUE FALSE TRUE TRUE FALSE ...
    X3  : logi  TRUE TRUE TRUE TRUE FALSE FALSE ... 
    X3  : logi  TRUE FALSE FALSE FALSE TRUE FALSE ... 
    X4  : logi  FALSE TRUE FALSE TRUE FALSE FALSE ... 
    X5  : logi  TRUE FALSE FALSE FALSE FALSE TRUE ... 
    X1.1: num  1.026 -0.285 -1.221 0.181 -0.139 ... 
    X2.1: num  -0.045 -0.785 -1.668 -0.38 0.919 ... 
    X3.1: num  1.13 -1.46 0.74 1.91 -1.44 ... 
    X4.1: num  0.298 0.637 -0.484 0.517 0.369 ... 
    X5.1: num  1.997 0.601 -1.251 -0.611 -1.185 ... 
    X6  : num  0.0597 -0.7046 -0.7172 0.8847 -1.0156 ... 
    X7  : num  -0.0886 1.0808 0.6308 -0.1136 -1.5329 ...
    X8  : num  0.134 0.221 1.641 -0.219 0.168 ... 
    X9  : num  0.704 -0.106 -1.259 1.684 0.911 ..
    X10 : android android OS windows7 windows8...
    [...]

I would like to cluster the variables (not data cases) x1, x2, ..., x9 (probably omitting the nominal X10) into clusters or subsets of correlated variables, for example (x1,x2,x6),(x3,x5), ...

As the variable have mixed types, it is impossible to use cor(), I think. It is also impossible to use Gower similarity coefficient, because it is a similarity between data cases.

Can you help me to find an idea to process this, please? I would prefer a solution in R.

Improve this question
$\endgroup$
3
  • $\begingroup$ Bio, please inspect the edit done to your question. Does it correctly describe what you intended? $\endgroup$
    – ttnphns
    Jun 20, 2016 at 16:33
  • $\begingroup$ I've taken liberty to remove r tag because binary data tag I added seems to me more important here. $\endgroup$
    – ttnphns
    Jun 20, 2016 at 16:41
  • $\begingroup$ Why do you want to use cor in the first place then?!? $\endgroup$
    – Has QUIT--Anony-Mousse
    Jun 21, 2016 at 18:26

5 Answers 5

Reset to default
2
$\begingroup$

Traditional FA and cluster algorithms were designed for use with continuous (i.e., gaussian) variables. Mixtures of continuous and qualitative variables invariably give erroneous results. In particular and in my experience, the categorical information will dominate the solution.

A better approach would be to employ a variant of finite mixture models which are often intended for use with mixtures of continuous and categorical information. Latent class mixture models (which are FMMs) have a huge literature built up around them. Much of that literature is focused in the field of marketing science where these methods see wide use for, e.g., consumer segmentation...but that's not the only field where they are used.

The software I know and recommend for latent class modeling is neither free nor R-based but, in terms of proprietary software, it's not that expensive. It's called Latent Gold, is sold by Statistical Innovations and costs about $1,000 for a perpetual license. If your project has a budget, it could easily be expensed. LG offers a wide suite of tools including FA for mixtures, clustering of mixtures, longitudinal markov chain-based clustering, and more.

Otherwise, the only R-based freeware I know about (polCA, https://www.jstatsoft.org/article/view/v042i10) is intended for use with multi-way contingency tables. I'm not aware that this tool can accept anything other than categorical information. There may be others. If you poke around, maybe you can find some alternatives.

Improve this answer
$\endgroup$
5
  • 3
    $\begingroup$ I would partly agree, but... Mixtures of continuous and qualitative variables invariably give erroneous results That's a strong message. It probably should be proven some way. categorical information will dominate Often yes, but you might weight variables/categories, although there is difficulty to define "right" weights. $\endgroup$
    – ttnphns
    Jun 20, 2016 at 16:20
  • $\begingroup$ @ttnphns "Proof"? I can't remember reading anything in the literature that directly addresses this issue of mixtures of variables with traditional, OLS-based FA and cluster solutions. I can speak to considerable experience in trying to get such approaches to work before I became aware of FMMs -- the results were invariably on the side of undesireable and worse. Let no one gainsay that experience. $\endgroup$
    – Mike Hunter
    Jun 20, 2016 at 16:33
  • $\begingroup$ Using latent class modeling is an option (like factor, only assumining categorical latent trait). Still, one should not - to my taste - pass a latent variable model for clustering model, in the narrow sense of "clustering": as collecting similar entities. This isn't a critique for your answer, - just a remark for a reader. $\endgroup$
    – ttnphns
    Jun 20, 2016 at 17:02
  • $\begingroup$ @ttnphns These models don't generate inherently categorical traits as the LC approach is more like a grade of membership model, producing probabilistic estimates of assignment to a cluster. Discretized final solutions are typically decided on based on minimizing a metric such as AIC or BIC. $\endgroup$
    – Mike Hunter
    Jun 20, 2016 at 17:09
  • $\begingroup$ (That is so, probability; but probability implies a [latent] event, i.e. category) $\endgroup$
    – ttnphns
    Jun 20, 2016 at 17:23
1
$\begingroup$

Clusters of correlations are best investigated by factor analysis. There is a number of different implementations of factor analysis in R and I would recommend the package 'psych' on CRAN as a starting point:

http://www.personality-project.org/r/psych/

http://www.personality-project.org/r/#factoranal

You can trick cor() into accepting logicals, because every logical is either 0 or 1 in R:

> TRUE*7
[1] 7
> FALSE*7
[1] 0

You just need to change the type using as.numeric() as in 

a <- c(TRUE, TRUE, FALSE, FALSE, FALSE)
as.numeric(a)

Hope that helps!

Improve this answer
$\endgroup$
2
  • $\begingroup$ @geekoverdose : thank you for you reply, but I have another variable which contains discrete variables (X10) $\endgroup$
    – Bio
    Jun 20, 2016 at 15:01
  • $\begingroup$ @Bio you mean because of the one-hot encoding? This works fine with discrete variables too. $\endgroup$
    – geekoverdose
    Jun 20, 2016 at 15:41
1
$\begingroup$

So, you have a mixture of categorical boolean and numeric continuous variables. You want to cluster the variables (not data cases) based on their similarity.

A correlation coefficient could be assumed the similarity measure. We could, for example, compute Pearson $r$. Given that boolean true/false is convertible into 1/0 binary values, $r$ is computable. $r \text {(numeric,numeric)}$ is classic $r$; $r \text {(binary,binary)}$ is point-point $r$ or Phi coefficient; $r \text {(numeric,binary)}$ is point-biserial $r$. All these are hypostasized Pearsonian correlation.

You may go straightforward and do the analysis (cluster) based on those three kinds of correlation values collected in one matrix. You may do it if you see the boolean/binary data as profoundly dichotomous, where no underlying continuous variable is conceivable in the background.

But then some critic might take a stance to say that there is no theoretical (philosophical) way at all to compare a similarity between categorical features with a similarity between scale features. That view would suggest you then to dichotomize your continuous variables - some way, and forget that they were scale before. So all the data are binary and you are fine.

Whereas if you choose to accept the idea of underlying continuous variable then using the aforesaid initial correlation matrix directly in the analysis stambles against another snag. The problem is that - due to the fact that a manifest binary variable (i.e. dichotomized underlying one) is only 2-valued but a continuous manifest variable is many-valued - the magnitudes of the three coefficients is risky to compare directly. See, for example 2nd paragraph here. In short, coefficients including binary variable are heightened sensible to the cut point taken at the hypothetical dichotomization of its underlying precursor variable. One way out would be to try to "restore" (infer) correlation values which "existed" before dichotomizations. That means computation of tetrachoric correlations in place of point-point $r$s and biserial correlations in place of point-biserial $r$s. If needed, the whole matrix might be then "smoothed" towards positive-definitness.

Another approach (not unquestionable, as any is) might be to rescale correlations in their empirically accessible range in the given data. This trick is, so to speak, atheoretical, it may or may not imply the existence of underlying continuous variable for dichotomous ones. The idea is simply to take away the effect of any skew of variables' marginal distributions on the coefficients. $r_{rescaled}=r/r_{max}$; for example if the observed $r$ is $.4$ and the maximal possible value for these two variables is $.95$ (which you get after sorting their data both ascendingly) then the rescaled value is $.42$. The whole matrix might call be "smoothed" into p.s.d. in the end.

An approach alternative to the previous one (taking away the marginal effect) might be to compute nonparametric correlations instead of $r$ - such as rank-based Spearman rho or Kendall tau. It is also an option. And from this point we begin to sight, logically, having done a circle, the further option of dichotomizing the scale variables (instead of ranking them) - from what we started the discussion.

After you compute correlations (or you would like other similarity measures?) you will have to decide on the clustering method - for example one of hierarchical methods. But here starts another story. You might also want to use Factor analysis in place of Cluster analysis: although factor analysis is not clustering but rather latent variable technique, it gives "clusters", in some sense.

Improve this answer
$\endgroup$
1
1
$\begingroup$

You could one-hot encode your binary features and normalize your data to enable correlation computation:

library(caret)
df <- data.frame(scale(data.frame(predict(dummyVars(~., df), df))))
library(corrplot)
corrplot(cor(df))

Based on this you could apply any clustering approach (example with K-Means, but also look into the details of factor analysis as suggested by @Bernhard):

km <- kmeans(x = t(df), centers = 3, iter.max = 1000)
print(km)
print(km$cluster)
print(km$centers)
Improve this answer
$\endgroup$
5
  • $\begingroup$ one-hot encode You mean convert into numrical, binary codes 1 vs 0? $\endgroup$
    – ttnphns
    Jun 20, 2016 at 16:43
  • $\begingroup$ @ttnphns correct, a transformation into binary dummy variables. $\endgroup$
    – geekoverdose
    Jun 20, 2016 at 17:03
  • $\begingroup$ (1) Why do we have to normalize explicitly to enable correlation computation? (2) It is well-known (and pointed by many on this site) that K-means is not recommended, even not valid, for categorical or mixed data. Besides, K-means needs data and not proximity matrix, to work with; moreover, what might be a sense to use K-means to cluster variables (not data points)? $\endgroup$
    – ttnphns
    Jun 21, 2016 at 9:17
  • $\begingroup$ @ttnphns A valid point. 1) As you pointed out, mixing continuous and categorial variables is not optimal - scaling relaxes this problem to some extent (dividing by $\frac{1}{n}$ for $n$ levels etc. would be better, but still imperfect) 2) I transposed the data to cluster by variables over samples instead of vice versa - each clustering algorithm could be applied then. This just gives clusters of variables (not correlation), hence is somewhat orthogonal to using cor(). This answer was meant to provide a simple approach - and now, with answers like yours, more serves as basis of discussion ;) $\endgroup$
    – geekoverdose
    Jun 21, 2016 at 9:43
  • $\begingroup$ to provide a simple approach I'm uneasy to say that, to my mind, the answer in its brief form as currently and the issues wouldn't be very convincing for the OP. Sorry. $\endgroup$
    – ttnphns
    Jun 21, 2016 at 9:58
1
$\begingroup$

Because you have mostly either continuous variables or binary variables, the suggestion made by @geekoverdose is certainly an option. The main issue that arises when taking this approach is dealing with nominal variables with more than two categories (or binary variables with rare classes). In this case, 1-1 matches are important and 0-0 matches probably aren't. In other words, your variable is asymmetric binary (see here for a nice explanation).

Just using Euclidean distance with k-means will ignore this. On the other hand, using your suggestion of Gower similarity will not. This is because nominal variables are handled via the dice coefficient, which essentially just one-hot encodes the data and ignores 0-0 when computing the similarity. This is easily done using the daisy function in the cluster package, just be sure to have each variable set as the correct type in the data frame.

To cluster this distance matrix, you then just need to choose an algorithm that can handle a custom distance matrix. K-medoids is one, and it has an implementation in R using the pam function.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.