2
$\begingroup$

I am currently trying to perform classification on a set of data.

I have 19 category_1 observations and 15 category_2 observations. What is the best way to sample from this dataset to train and test my classifier, or are there simply not enough observations?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ How many dimensions do you have? What classification algorithm do you intend to use? $\endgroup$ – Marc Claesen Jan 10 '14 at 16:27
  • $\begingroup$ Logistic regression, probably around 3 or 4 dimensions $\endgroup$ – rwolst Jan 12 '14 at 16:33
4
$\begingroup$

The sample size of your test data can be estimated using probability inequalities. You can calculate the required sample number to get a desired accuracy with high probability.

But it is not easy to estimate the training size since it depends on both your model and target complexity. You can try leave-one-out cross validation strategy to get a learning curve in order to choose such sample number that best balance the bias and variance issue.

If your feature is high dimensional, this paper may provide an estimation of how large a sample size is needed with low sample size.

Improve this answer
$\endgroup$
1
  • $\begingroup$ However, with that small sample sizes even if a decent classifier can be derived, it cannot be shown that the classifier works well. You may want to have a look on our paper discussing this problem: Beleites, C. et al.: Sample size planning for classification models., Anal Chim Acta, 760, 25-33 (2013). DOI: 10.1016/j.aca.2012.11.007](dx.doi.org/10.1016/j.aca.2012.11.007). Briefly, the problem is that with a total of 34 cases you won't even be able to get a useful estimate of the learning curve because of the small test sample size ($\leq 34$). $\endgroup$ – cbeleites unhappy with SX Jan 10 '14 at 19:32
0
$\begingroup$

The answer to this question depends on whether your categories overlap one another and how much.

If your categories are completely separated, then I'd say you have enough data. Just use 70% for training and 30% for testing - perhaps in a bootstrap.

However, in most cases that I've seen in life sciences the categories usually overlap considerably, and 19+15 observations is just not enough to train a good classifier.

Improve this answer
$\endgroup$
0
$\begingroup$

This definitely depends on too many factors to give a good answer. The data's specificities and the classifying power of the feature can make a lot of difference. Usually such a low number of samples (34) would be too low for any classifying method, which will likely overfit however hard you try to use any variance reduction techniques (or just have terrible predictive performance).

But if your features classify your data exceptionally well, you might train a decent classifier notwithsanding.

Imagine the following situation : you are trying to classify whether a person is male or female. One of your features is "has male sexual organs". Then it really won't matter how many samples you have, your classifier will always be correct (this is somewhat equivalent to Michal's explanation, as in this case the classes do not overlap at all in the space of features, and there is no variance within a class). Of course this is a gross exaggeration but you get the idea.

Bottom line - 1) you can't tell if a classifier will work without knowing the data's specificities, and 2) you won't know until you try.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.