I am trying to design a modelling framework which can be completely automated. The approach involves automatic creation of features, multi-class target labels and a supervised learning model $M$. In the end, $M$ should predict class-membership probabilities for $N$ fixed classes. This is a hard requirement on the output.
It is a bit tricky to automatically generate target labels however because the distribution across classess is not homogenous. Some classes receive much fewer labels than others and some labels don't get produced at all.
I also do not want to go the clustering route because profiling the clusters to relate them to any one given class label cannot be automated. So options such as Latent-class cluster analysis is not a good approach for me.
My question is how can I either modify the heuristic that generates the labels or modify $M$ or modify the output of $M$ post-modelling (maybe some form of calibration) to ensure model learns well across all classes. Currently if a given label $L$ simply gets missed, then then the model output is simply $P(X|L) = 0$.