6
$\begingroup$

I'm trying to gain a better understanding of kmeans clustering and am still unclear about colinearity and scaling of data. To explore colinearity, I made a plot of all five variables that I am considering shown in the figure below, along with a correlation calculation. colinearity

I started off with a larger number of parameters, and excluded any that had a correlation higher than 0.6 (an assumption I made). The five I choose to include are shown in this diagram.

Then, I scaled the date using the R function scale(x) before applying the kmeans() function. However, I'm not sure whether center = TRUE and scale = TRUE should also be included as I don't understand the differences that these arguments make. (The scale() description is given as scale(x, center = TRUE, scale = TRUE)).

Is the process that I describe an appropriate way of identifying clusters?

Improve this question
$\endgroup$
7
  • $\begingroup$ I realize that these are two slightly different questions, but I want them to be together to give some background to what I am using the results of the scale() function. $\endgroup$
    – celenius
    Mar 24, 2011 at 14:52
  • 1
    $\begingroup$ @celenius center=true just means remove the mean, and scale=TRUE stands for divide by SD; in other words, with both options active, you're getting standardized variables (with mean 0, unit variance, and values expressed in SD units). $\endgroup$
    – chl
    Mar 24, 2011 at 14:58
  • $\begingroup$ @chl Ah! Thank you. That is a much clearer explanation to me than the help file. Does it make a difference how a variable is standardized for clustering with kmeans though? I would assume not, but don't know for sure. $\endgroup$
    – celenius
    Mar 24, 2011 at 15:01
  • 3
    $\begingroup$ @celenius I can suggest this article, Standardizing Variables in K-means Clustering (Steinley, 2004) to get an idea of the effects of different kind of transformations. The utility of scaling depends on the data you have, but usually one reason to use standardized (scaled) variables is to avoid obtaining clusters that are dominated by variables having the largest amount of variation. $\endgroup$
    – chl
    Mar 24, 2011 at 15:18
  • 1
    $\begingroup$ @celenius It shouldn't introduce bias. Take, for instance, the extreme of perfect collinearity in 2D: all observations are of the form $(a+t x, b+t y)$ for constants $a,b,x,y$. K-means uses Euclidean distances and the distance between two such points given by $t$ and $s$ is just $|s-t|$: it's really a 1D problem and there's no special weighting given to any of the data. $\endgroup$
    – whuber ♦
    Mar 24, 2011 at 19:31

1 Answer 1

Reset to default
3
$\begingroup$

As CHL has already explained the use of center and scale to obtain standardized variables, I'll address collinearity:

There is good reason to reduce collinear variables when clustering.

Curse of Dimensionality

The more dimensions you use, the more likely you are to fall victim of Bellman's 'curse of dimensionality'. In brief, the greater the number of dimensions, the greater the total volume, and the greater the sparsity of your data within it. (See the link for more detail.)

Dimension Reduction --- manually by inspecting of pairwise collinearity...

You mention that you have already reduced variables from some larger number down to 5 using pairwise collinearity measures.

While this will work, it is quite tedious, since in general you will have $n\choose 2$ number of pairs to check. (So for example with 10 variables, you would have ${10 \choose 2} = 45$ different pairs to examine -- a few too many to do manually in my opinion!

Dimension Reduction --- automatically using Principal Components Analysis (PCA)...

One way to handle this automatically is to use the PCA (principle components analysis) algorithm. The concept is more or less what you're doing manually -- ranking the variables by how much unique information each variable is contributing.

So you provide PCA your $n$-variable dataset as input, and PCA will rank order your variables according to the greatest variance each explains in the data -- essentially picking out the non-collinear variables.

Depending on whether you want 2-D or 3-D clusters, you would use the top 2 or 3 variables from PCA.

Principal Components in R

The PCA algorithm is available (built-in) from R.

Actually there are several functions in R that do principal components.

I've had success with prcomp().

Standard Reference available free online

One of the best references available is the classic:

Elements of Statistical Learning, by Trevor Hastie, Robert Tibshirani, and Jermoe Friedman

The authors have graciously made the entire book available (for free) as a PDF download from their Stanford website.

There are excellent chapters on Clustering, Principal Comoonents, and a great section on the curse of dimensionality.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.