I have a sample of about 300 subjects who have been measured at 3 different times (morning, afternoon, evening). The variable of interest can be assumed to be approximately normal. It appears that most subjects have an increase between between morning and afternoon, followed by a decrease from afternoon to evening. Some however show an opposite pattern (devrease->increase), while yet others remain approximately the same.

What I am interested in is clustering, or classifying the subjects according to their trajectories. After a bit of googling, I have discovered GMM. I don't really understand what goes on behind the scenes, but it looks like the classification is done based on straight line fits when we only have 3 points. This seems highly inappropriate to me since straight line does not capture the increase followed by a decrease type of behaviour. Another thing is, people seem to be using Mplus package, which I am not familiar with and would rather avoid buying and learning (I am very comfortable with R and Matlab).

  • 1
    $\begingroup$ Perhaps you might want to specify better what is the exact problem. Your question asks for best approaches; both clustering and a mixture of gaussians are a good approach. I don't understand the part about a straight line fit, but it seems you are simply looking for a tutorial on Mixtures of Gaussians. $\endgroup$ – means-to-meaning Nov 7 '13 at 22:36
  • $\begingroup$ Classifying indicates supervised learning (group membership is known), so this is a purely a clustering problem. A GMM can be thought of as a mix of a clustering method and a latent growth curve model. One thing to keep in mind is that a GMM does not need to be linear. I'd be wary of specifying a quadratic trend with three time points though, because it's the minimum number of points required. For software, you can use GMMs using the hlme function in the lcmm package in R. Another possible option would be to use something like k-means clustering. $\endgroup$ – dmartin Nov 7 '13 at 22:47
  • $\begingroup$ @means-to-meaning Well, I have no problem fitting the mixture of Gaussians to my 3-dimenstional variable, but that will ignore the fact that there is inherent order in my measurements, i.e. that they are trajectories. I don't really understand GMM very well, but it appears that to cluster based on quadratic curves, not straight line trajectories, I would need a minimum of 4 measurements. $\endgroup$ – Brian Nov 7 '13 at 22:47
  • 1
    $\begingroup$ Brian, please spell out "GMM" acronym in your question. $\endgroup$ – ttnphns Nov 8 '13 at 8:51
  • $\begingroup$ There are 9 combinations of direction of shifts between three values, of which you mention 3. Do you want to deal with all 9 possibilities, or only the three you mention? $\endgroup$ – Glen_b Jan 12 '14 at 21:09

In order to use a mixture of Gaussians for your problem, you have to assume that your three measurements are multivariate normal. In that setting, you have many measurements of a mixture of 3 dimensional normals, generated by k different underlying densities.

You could start by setting k=3, one for "/ shape", one for "/\ shape", and one for "-- shape" you mentioned having observed. This will allow you to model the underlying mixture distribution, estimate the covariance of each cluster and of course, classify all measurements. Before comparing them, I would also think about subtracting the mean (and potentially also scaling), since you probably don't care about the general level of an individual, but about the change.

Please provide some details where you found the information about the "quadratic curves" you are mentioning, or why you think the Gaussian mixture would classify your measurements using a linear boundary, if you wish to follow up on that. Based on my understanding, the results are labeled based on the likelihood of being generated by a particular density (one of the k you started with). I am not aware of any condition that would restrict the boundary to be a polynomial in the general approach, or even suggest it could be so for that matter.

EDIT (in response to your comment):

The order in your measurements will be captured, since you will encode the morning,afternoon, evening measurements for each individual as a 3dimensional, multivariate normal, with unknown mean and unknown covariance. Imagine there are 3 such Gaussians - one for each characteristic shape of the measurements.

A gaussian mixture model is called a "mixture" model since it presumes that the measurements come from a probability, which is some weighted combination of the three underlying Gaussians. Each measurement can have a different weighting.

When you fit the model, you need to infer the parameters of all three Gaussians and simultaneously, infer the responsibility (weighting) of each Gaussian for having generated a particular measurement.

If you find this concept too hard to understand, or if my explanation is completely incomprehensible I would suggest you try a much simpler and much more straightforward approach: k-means.

You simply take the $3xN$ table of data, stick it in a k-means function and set the number of clusters to be 3. I have written a snippet in R to illustrate:

First I generate some data along the lines of your description.


# Generate training data
Sigma <- diag(3)
mu1 <- c(3,0,3)
mu2 <- c(0,0,0)
mu3 <- c(0,3,0)    

group1 <- mvrnorm(n = 5, mu1, Sigma)
group2 <- mvrnorm(n = 5, mu2, Sigma)
group3 <- mvrnorm(n = 5, mu3, Sigma)    

# Generate test data
new_measurements <- rbind(mvrnorm(n = 2, mu1, Sigma), mvrnorm(n = 2, mu2, Sigma),mvrnorm(n = 2, mu3, Sigma))    

Here is what it looks like: (I labeled the measurements according to the Gaussian which generated them purely for visual convenience. In your case, your data is not labeled.)

# Plot training data and cluster

Simulated measurements

Now we can cluster the data using k-means, by setting the number of cluster to 3, since that is your intuition. (We could also try to investigate what is the right number of clusters using the data.)

# Cluster
data <- rbind(group1,group2,group3)
fit <- kmeans(scale(data), 3)
# get cluster means 
cluster_means <- aggregate(data,by=list(fit$cluster),FUN=mean)    

And we get the mean values for each cluster, which we can then use to make predictions for new measurements.

# Predict cluster
predicted_clusters <- cl_predict(fit,scale(new_measurements))

We get the means of the clusters,

        [,1]       [,2]       [,3]
1 -0.8618375 -0.6464979 -0.5279966
2 -0.3605748  0.9793615 -0.5128992
3  1.1221597 -0.6580355  1.0378763

as well as the predicted cluster for the our new measurements.

Class ids:
[1] 3 3 1 1 2 2

As you can see, k-means correctly predicted to which cluster they belong to, though the k-means estimations of the means are relatively far off. The third cluster corresponds to our first simulated Gaussian with means (3,0,3).

Lastly, we can plot the new data and the estimated means.

legend("topright",legend=c("New data","k-means Cluster Means"),lty=1,col=c(1,3,4))

New data and k-means Cluster means

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.