I have a bunch of non-normal and normal data that I would like to cluster on. The normal data are things like height and weight. In total, it is 201 observations and 44 variables. The non-normal data is the frequency of an event divided by a players minutes times a scalar factor. So for example,

$$PointsperX = Points/Minutes * X$$

I have performed both a square root and log transformation, and the data is slightly more normal, but still very far off ($p-value$ increases by $0.0001$ to $p=0.0001$ for most variables).

Anyway, I proceeded with clustering my data, first I scaled it to have mean 0 and sd of 1, then used NbClust andBIC from the mclust package. I get 2 and 1 clusters respectively. However, I have a suspicion that NbClust only shows 2 because that is the minimum number. The number of clusters makes sense because the data is not normal so it won't cluster very well.

How should I handle the issue of non-normal data? Should I reduce the number of variables I look at? My idea revolves around using where players scored there points or performed blocks, hence I would prefer not to do this method unless others thought it would help clustering from past experience.

Many people have assumed data to be normal in past research when it is not, but that is a really poor thing to do in my mind. I have not tried fuzzy clustering yet, but I have a feeling it would result in simlar results as above.

  • $\begingroup$ The fact that decimal data (0 < x < 1) becomes negative in log space isn't a reason in and of itself to avoid a log transform. I don't understand your statement that this transformation "does not make sense" because of that. $\endgroup$ Oct 23 '18 at 19:54
  • $\begingroup$ From what I have been told, Log transformations of data remove the units of data. I know having negatives isn't a reason not to do it, but it did not make the data normal (made it a little bit more normal, p value increased by 0.0001 for some variables). I see what you are saying and I guess transforming data in cluster models is acceptable because we simply want to know what the clusters are, then can find medians/means and build profiles of those clusters with original data. $\endgroup$ Oct 23 '18 at 21:06
  • 1
    $\begingroup$ I'm not aware of any clustering method that assumes the data are normally distributed. In fact, if the data do have cluster structure, this implies a multi-modal (and therefore non-normal) distribution. Some methods (e.g. Gaussian mixture models) do assume that data within a cluster are normally distributed. But, even in this case, the overall distribution (including all data from all clusters) will typically be far from normal. $\endgroup$
    – user20160
    Oct 23 '18 at 21:34
  • $\begingroup$ The data within a cluster are normally dsitrbuted? Can you elaborate more on that? $\endgroup$ Oct 24 '18 at 7:51

DBSCAN is a cool clustering algorithm that doesn't make assumptions about how data are distributed. See http://scikit-learn.org/stable/modules/clustering.html#dbscan. Even though that description is from a Python library, there is an R package for DBSCAN, as well, called dbscan


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.