I'm working on a difficult classification problem where there are a very large number of known classes (~50k). I have only about 20k labelled points, but these only represent <1% of the possible classes since they're Zipfian distributed. The vast majority of the classes have never been observed, but I still want to have the ability to predict them in the future.
My idea is to create a generalized model where I've invented several useful ways of measuring the similarities between a record and each of the 50k potential classes. Instead of learning a ton of weights ($\propto |classes| * |vocabulary|$) that are class specific, I can learn only a dozen by creating a linear combination of the vectors (where vector length = |classes|), applying softmax at the end, and minimizing cross entropy.
I'm unable to find any resources about regression over vector functions. Why does it seem so rare for regression to be performed over vector functions? Is there a technical reason one should be wary about using vector features other than it's usually the case that someone wants to predict classes independently (k-1 logistic regressions)? It seems like this would be common for the case I described where labelled data is limited but there are a large number of potential classes and one has reason to believe that the features are equally important across classes and there's no interaction between classes.