Looking for a hierarchical-clustering method for multiple data types

I would like to find a hierarchical-clustering method useful to assign a group membership into k groups for all individuals in my dataset. I have considered several classic ordination methods, PCA, NMDS, "mclust", etc., but three of my variables are categorical (see data description below). Further, I was wondering if it is preferable to a method that reports a posterior probability of group membership for each individual? I am using R.

Data description: I have sampled almost 2000 individual birds (single species representing two subspecies or phenotypes) across Sweden. All individuals are adult males. Although this is one species, in middle of Sweden there is a (migratory) divide where the southern individuals presumably migrate to West Africa and north of the divide they presumably migrate to East Africa. There is a zone of overlap approximately 300 km wide at the migratory divide.

Variables:

• Wing (mm) - continuous
• Tail (mm) - continuous
• Tarsus (mm) - continuous
• Mass (g) - continuous
• Colour (9 levels) - categorical
• Stable carbon-isotopes (parts per mil) - continuous
• Stable nitrogen-istopes (parts per mil) - continuous
• SNP WW1 (0, 1, 2) - molecular marker, 0 and 2 are fixed and 1 is heterozygote
• SNP WW2 (0, 1, 2) - molecular marker, 0 and 2 are fixed and 1 is heterozygote

Description of the colour variable: (brightest yellow) S+, S, S-, M+, M (medium), M-, N+, N, N- (dullest yellow-grey)

• Can you say what methods you've already looked into, what seems to work and what difficulties you're facing? A quick search yields some references that should be relevant, e.g, gatsby.ucl.ac.uk/~heller/bhcnew.pdf and dx.doi.org/10.1016/S0304-3800(02)00256-9 – Jack Tanner Feb 2 '12 at 21:26
• I feel I am being led down the rabbit hole! I am a statistically-minded ecologist who has read a few papers suggesting it might be possible to assign a probability to group assignment using such methods. Sadly, your first reference is too challenging for me at this point. I have been through the literature looking for a method I could implement in R looking at discriminant function analysis, non-metric multidimensional scaling and other ordination techniques. But since I do not want to want to make an a priori group decision to the model I turned to Baysian methods naively. – Keith Larson Feb 2 '12 at 21:41
• It sounds like you're looking not necessarily for Bayesian methods, but generally for methods that can do so-called "soft clustering", a.k.a. "fuzzy clustering". Maybe have a look at the mclust package in R. – Jack Tanner Feb 3 '12 at 4:30
• Yes, I should have mentioned I looked at 'mclust' in R as well. In the help file it explicitly states that categorical data is not allowed. Is it reasonable to recode our colour variable to numeric as it represents a continuum from yellow to grey (albeit subjective)? Further, how would I treat my two SNP's (0 and 2 are fixed and 1 is heterozygous)? – Keith Larson Feb 3 '12 at 7:43
• @Keith I described some possible ways to perform ordination with mixed data in this response, as well as some clustering procedure that might be used afterwards here. Does that help? – chl Feb 3 '12 at 11:58

Have a look at OPTICS. It will find hierarchical clusters. You don't need to specify the number of clusters you need (which doesn't make that much sense with hierarchical clusters, actually!). And you can customize the distance function to suite your needs, because obviously euclidean distance is not really sensible here, as a delta of 1 mm and a delta of 1 g is not the same. So you'd first define an appropriate distance function, then run OPTICS with it to obtain clusters.
When OPTICS doesn't find any clusters, that can also indiciate that the data set just doesn't cluster with these parameters (distance, minPts). The results of other algorithms such as k-means can be quite misleading, as they will always force the data set to cluster, and may easily return a random partitioning.