I have a linguistic dataset consisting of 389 sentences and my aim is to classify these sentences into meaningful groups. If the size of the dataset would be smaller – say, only twenty or thirty sentences – I could just read each sentence, compare it to other sentences and conclude that: - sentences 1, 3, 8, 14 and 20 resemble each other and can be treated as a distinct group - sentences 4 and 5 resemble each other and can be treated as a distinct group - etc..
However, as the dataset is larger, analyzing each sentence manually would be laborious and, I supppose, methodologically unsound. Luckily, I have some categorical variables that describe each sentence – for instance, the semantic class of the main verb of the sentence, the presence / absence of certain thematically important words, the dependency roles of these words etc. So, I have been experimenting with these variables and creating different groupings of the data. Here’s a made-up example of what I’ve done:
Step 1. I categorize my data by variable A, getting seven different groups with group 1 containing, say, 20% of the cases, group 2 18%, group 3 14,5% and so on.
Step 2. I further categorize the groups from step 1. Group 1 could be divided into subgroups by variable B, which would give me roughly three subgroups (let’s call them s1, s2 and s3). One of these – say, s1, might be further split in to subsubgroups by variable C. s2 and s3 are so small that there is no point in creating any more subgroups from them.
Step 3. Group 2 from step 1 seems to split nicely if I use variable C right away. This produces three subgroups, with which I'm already happy.
This hypothetical process is illustrated in the following graph:
Now, my problem is, that although this kind of approach in the end gives me distinct groups that I am quite happy with, the methodology involved seems unexplainable / ad hoc. I would like to know
a) Is there some "real" statistical method that could be used for trying out different kind of groupings? E.g. a way to find out by which variable I should split the data first in order to minimize the number of further subdivisions
b) If I just look at the data and manually, try out different ways of grouping it and end up with a particular grouping, would that be a serious method that could actually be used? Is there a name for such an approach – something like “manual cluster analysis”?
I have a feeling that some form of cluster analysis might be appropriate, but so far I haven’t got any sensible results. It feels like the data is not clean enough for that to work – the process, as if it were, cries out for human intervention at some stage. I also tried out conditional inference trees by just arbitrarily selecting one variable as the response variable, but that also seemed to be somewhat dubious.