I'm working on a text classification project. We have around 300k documents (small, 1~2 phrases) and we don't know the set of labels or how many labels there are. The recommended approach to me was the following:
- Use clustering techniques to find an appropriate number of clusters. Hopefully some of these clusters will represent classes.
- Check semi-manually if the clusters seem to make sense
- Send them clusters to the domain experts. They will check if the clusters are "ok". Ok means some clusters contain one class and some contain a few different classes. If not ok, rethink the clustering and go back to 1.
- Ask experts to label some observations from the clusters.
- Create a classification model based on the labeled data.
The idea is that the clusters will make it easier for the experts to identify labels and finally label the data.
I can't stop thinking that this approach seems flawed, but I am not sure. I'm using k-means with tf-idf because it is easy to interpret the clusters and improve preprocessing. Also, the text is very specific so I don't think I would benefit a lot from pretrained embeddings.
The resulting clusters for now are:
- Good clusters. Some documents have a very similar vocabulary and we identify them.
- Ok clusters. Documents share vocabulary, but contain 2~4 classes
- Bad clusters. Documents share vocabulary, but the shared vocabulary is irrelevant
- "Unclustered" clusters. Documents that do not fit in any other cluster. (centroid is around 0 for all tfidf components)
Most of the phrases are good or ok, and we have some bad and some big "unclustered" clusters. If we decided to label say 50 observations of each cluster we would certainly be biasing the training set since the clusters have different sizes. Using that approach, the proportions of number of observations between the classes would be different from reality.
Would sampling observations for each cluster with sample sizes proportional to the cluster sizes would be a good approach? Are there any other problems with this approach?