Consider a typical machine learning problem where you try to do object classification from a high-dimensional set of features. Suppose we know that the features are actually a collection of distinct "descriptors" of comparable (high) dimensionality; each of which was selected because (hopefully) it captures different facets of the available information.

Support we have prior knowledge of what these descriptors are. Under what circumstances is it better to concatenate these descriptors into a single feature vector (i.e. ignore the aforementioned prior knowledge) and train a single classifier, compared to training separate classifiers for each of these descriptors and trying to combine the classifier outputs?

My ad-hoc thinking here;

Favoring the second approach:

  • Training separate classifiers over subsets of the features ameliorates the "curse of dimensionality" somewhat, especially if there's not a lot of training data.

  • We can use our "meta-knowledge" of these individual descriptors, in particular, which classification methods are known to work well with each of them; for example, if descriptor A is known to typically give good results with k-nearest neighbor classification whereas B is typically used with random forests, then we can train a "favorable" classifier for each descriptor and (somehow) combine the output of these classifiers.

On the other hand:

  • It is hard to tell how the correlation between the descriptors will influence a single (good) classifier, the disadvantage of throwing away this information might harm the classification more than helping it.

  • We don't have to do the additional step of having to combine the result of multiple classifiers (as opposed to the multiple classifier situation).

So what are the considerations to make here? Are there any "rules-of-thumb"? References would be appreciated too.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.