3
$\begingroup$

I have 20 observations labeled for each class. The number of classes is 5, so I have a total of 100 observations. I want to classify one class vs. other classes (one vs. all).

For this, I first labeled '1' for one of five classes (= 20 observations), and '2' for the other classes (= 80 observations). After that, I spiltted the whole data into training data set and test data set for each class ('1' or '2'). Since I used 10-fold cross-validation method, 20 observations for class '1' were separated into 18 and 2 for training data set and test data set, respectively (62 training data and 18 test data for class '2'). I trained a LDA classifier for '1' or '2' labeled observations using training data set, and estimated classification accuracy using test data set.

My question is that isn't there any bias problem when using different number of observations to train a classifier. In my case, the difference is quite big (20 vs 80). I am worrying about this issue. If so, please tell me a proper solution (e.g., a specific classifier) and reference paper to overcome this problem.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

It is answered by Bayesian statistics. The training sample signal to background ratio 20/80 is your prior probability. The classifier is your likelihood and doesn't depend on the signal to backround ratio. This is easily understandable: the predicted numbers for the signal samples are the same if you calculate them for the whole prediction sample (with background) or only for the signal subsample. So the a posteriory probability (the predicted signal probability) does only depend on the prior ratio.

In your example, the ratio is the same with or without X-validation, so you are safe.

In addition it would be possible to correct the posterior probability even if you trained with a different ratio using Bayes theorem.

EDIT: Be carefull with the interpretation of the results of the classification. Sensitivity and specificity (otherwise often called purity and efficiency, or type I type II errors) heavily depend on the prior distribution. The classification purity efficiency in your training mixture of signal and backround is meaningless. It only makes sense to compare the classification power with the mixture in your testsample (e.g. compare two classificators with the same testsample)! Imagine your testsample wouldn't contain a single signal sample but of course your training sample did. What would be the Sensitivity and specificity of your classifier.

In addition, the purity efficiency is furthermore a function of the cut value where you classify a sample as signal or background. Depending on the ratio in your testsample you can just choose a different cut-value and get back to the Sensitivity and specificity in your training. Have you tried different values?

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your kind comment. So, you mean it is ok if I would set the prior probability to 20/80, right? $\endgroup$
    – user27795
    Commented Jul 9, 2013 at 12:29
  • $\begingroup$ What do you mean by 'set'? Your prior probability IS 20/80 in both cases, with and without x-validation. But you are right, if you would artificially change the ratio (e.g. arbitrary signal and background samples) the best choice would be the same ratio as in the prediction sample. Otherwise you have to correct in order to get a proper probability interpretation for your posterior. But also note: If you are not interested in a proper probability interpretation, it all just doesn't matter! $\endgroup$
    – SebastianNeubauer
    Commented Jul 9, 2013 at 12:42
  • $\begingroup$ The 'set' means that I use 20/80 the prior probability to train a classifier. Before posting my question, I used the default prior probability (50% vs 50%) to train a classifier in Matlab. $\endgroup$
    – user27795
    Commented Jul 9, 2013 at 13:36
  • $\begingroup$ Anyway, I trained a classifier using 20/80 prior probability, and classified testing data. However, sensitivity was very low while specificity is much high. When I used the default prior probability (50% vs 50%) regardless of the number of samples for each class, sensitivity and specificity were quite resonable considering classification accuracy (= (sensitivity + specificity)/2). If I use 50/50 prior probability to train a classifier, is it wrong in this case? Could you please give me some advice about that? $\endgroup$
    – user27795
    Commented Jul 9, 2013 at 13:37
  • $\begingroup$ I updated my answer $\endgroup$
    – SebastianNeubauer
    Commented Jul 9, 2013 at 14:17
0
$\begingroup$

I had the same problematic with LDA: discrimination worked a lot better when the number of observations in each class were matching.

I ended up artificially growing the numbers by adding more observations to the classes through boosting. There are several ways to do that, the easiest (and probably worst) being duplicating existing observations, a better one would create random observations according to the covariance matrix.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.