12
$\begingroup$

So, I've got a matrix of about 60 x 1000. I'm looking at it as 60 objects with 1000 features; the 60 objects are grouped into 3 classes (a,b,c). 20 objects in each class, and we know the true classification. I'd like to do supervised learning on this set of 60 training examples, and I'm interested both in classifier accuracy (and related metrics) as well as feature selection on the 1000 features.

First, how's my nomenclature?

Now the real question:

I could throw random forests on it as stated, or any number of other classifiers. But there's a subtlety - I really only care about differentiating class c from classes a and b. I could pool classes a and b, but is there a good way to use the a priori knowledge that all non-c objects likely form two distinct clusters? I'd prefer to use random forests or a variant thereof, since it's been shown to be effective on data similar to mine. But I could be convinced to try some other approaches.

Improve this question
$\endgroup$
3
  • $\begingroup$ I don't see any problems with your nomenclature. Is this 60 objects all you have? Then, in order to compute a classification accuracy, you need to split your dataset into train, test (and also validation) sets. There are various ways to do this but $k$-fold cross validation is the most common, I think. $\endgroup$
    – emrea
    Aug 16, 2012 at 21:49
  • 1
    $\begingroup$ Yes, just these 60. But I think for random forests, each decision tree is created with a subset of the samples, so you can get a estimated generalization error by applying each of the existing 60 samples to only those trees within the forest that didn't see that sample during construction. (stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#ooberr) So maybe typical cross validation isn't needed here. $\endgroup$
    – user116293
    Aug 17, 2012 at 6:12
  • $\begingroup$ you should be very worried about overfitting here. You have 1000 variables with 60 objects, I would feel a lot better if it was the other qay around and I would still say, you should worry about overfitting. Is there a logical or semantic way to redue the number of variables before doing the analysis? $\endgroup$
    – jank
    Oct 16, 2014 at 13:13

1 Answer 1

Reset to default
8
$\begingroup$

is there a good way to use the a priori knowledge that all non-c objects likely form two distinct clusters

If you are using a tree based method I don't think it matters as these classifiers partition the feature space then look at the proportion of samples in each class. So all that matters is the relative occurrence of class c in each terminal node.

If however you were using something like a mixture of normals, LDA, etc then combining two clusters would be a bad idea (assuming classes a and b form unique clusters). Here you need to preserve the class structure to accurately describe the feature space that maps to a,b and c. These models assume the features for each class have a different Normal distribution. If you combine a and b you will force a single Normal distribution to be fit to a mixture.

In summary for trees it shouldn't matter much if you:

I. Create three classifiers (1. a vs b, 2. a vs c and 3. b vs c) then predict with a voting based method.

II. Merge classes a and b to form a two-class problem.

III. Predict all three classes then map the prediction to a two class value (e.g. f(c) = c, f(a) = not c, f(b) = not c).

However if you use a method that is fitting a distribution to each class then avoid II. and test which of I. or III. works better for your problem

Improve this answer
$\endgroup$
3
  • $\begingroup$ III sounds good - though I think if the classifier says a sample is .33 a, .33 b, and .34 c, I should probably sum the probabilities for a and b and thus choose 'not c'. $\endgroup$
    – user116293
    Aug 17, 2012 at 5:48
  • 1
    $\begingroup$ For (I), what's a good procedure for a split vote (1:a, 2:c, 3:b), or is that likely too rare to really matter? $\endgroup$
    – user116293
    Aug 17, 2012 at 5:56
  • $\begingroup$ For III. what you suggest is correct. For I. on 3 class data I don't think there is any measure to split votes (1 for each) since the transitive property would have to be violated. However for 4+ classes you can conceivably have ties at the top, in which case you could use a numeric quantity rather than win/loss; i.e. sum the weights take the max weight class. $\endgroup$
    – muratoa
    Aug 17, 2012 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.