I have a linguistic dataset consisting of 389 sentences and my aim is to classify these sentences into meaningful groups. If the size of the dataset would be smaller – say, only twenty or thirty sentences – I could just read each sentence, compare it to other sentences and conclude that: - sentences 1, 3, 8, 14 and 20 resemble each other and can be treated as a distinct group - sentences 4 and 5 resemble each other and can be treated as a distinct group - etc..

However, as the dataset is larger, analyzing each sentence manually would be laborious and, I supppose, methodologically unsound. Luckily, I have some categorical variables that describe each sentence – for instance, the semantic class of the main verb of the sentence, the presence / absence of certain thematically important words, the dependency roles of these words etc. So, I have been experimenting with these variables and creating different groupings of the data. Here’s a made-up example of what I’ve done:

Step 1. I categorize my data by variable A, getting seven different groups with group 1 containing, say, 20% of the cases, group 2 18%, group 3 14,5% and so on.

Step 2. I further categorize the groups from step 1. Group 1 could be divided into subgroups by variable B, which would give me roughly three subgroups (let’s call them s1, s2 and s3). One of these – say, s1, might be further split in to subsubgroups by variable C. s2 and s3 are so small that there is no point in creating any more subgroups from them.

Step 3. Group 2 from step 1 seems to split nicely if I use variable C right away. This produces three subgroups, with which I'm already happy.

This hypothetical process is illustrated in the following graph:

a manually drawn tree illustrating a possible grouping

Now, my problem is, that although this kind of approach in the end gives me distinct groups that I am quite happy with, the methodology involved seems unexplainable / ad hoc. I would like to know

a) Is there some "real" statistical method that could be used for trying out different kind of groupings? E.g. a way to find out by which variable I should split the data first in order to minimize the number of further subdivisions

b) If I just look at the data and manually, try out different ways of grouping it and end up with a particular grouping, would that be a serious method that could actually be used? Is there a name for such an approach – something like “manual cluster analysis”?

I have a feeling that some form of cluster analysis might be appropriate, but so far I haven’t got any sensible results. It feels like the data is not clean enough for that to work – the process, as if it were, cries out for human intervention at some stage. I also tried out conditional inference trees by just arbitrarily selecting one variable as the response variable, but that also seemed to be somewhat dubious.

  • $\begingroup$ It's a bit dated now (2009) but David Blei's lectures on latent dirichlet allocation topic models may be informative ... videolectures.net/mlss09uk_blei_tm It would help potential CV respondents if you were to describe a bit about what methods and/or approaches wrt NLP topic models you are aware of since the literature in this area is vast. $\endgroup$ – Mike Hunter Sep 25 '17 at 12:02
  • $\begingroup$ @DJohnson Well, frankly, I must admit that topic modeling is actually a completely new term for me. Seems like it might be useful, have to look into that. I suspect that the fact that I don't have an especially large dataset (389 cases) might be a problem, though $\endgroup$ – jharme Sep 25 '17 at 15:35

You should consider constraint clustering.

You would "eyeball" subsets, and input which sentences are similar, and which are not. These manual decisions are yourconstraints.

The constraint clustering will then attempt to find a clustering that best matches this constraints (I.e. where pairs that you considered similar are in the same cluster, and pairs you found dissimilar are separate).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.