3
$\begingroup$

There is a marked dataset of $n=2879$ objects on 7 classes. Each object (an object is a text of 1-3 sentences) was marked up by different users. Class 1 includes $n_1=790$ objects, Class 3 has the fewest objects: $n_3=192$. On can see the dataset is not balanced because the ratio $n_1/n_3 = 790/192 =4.11$ is large.

An expert analysed the dataset and concluded: objects were erroneously assigned to other classes during marking. The expert's decision based on human experience only: he read texts (remain the object for classification is a small text) and made decision: right or not right. In particular, there is a hypothesis that objects from class 3 were labeled with errors more often than others.

We applied various classifiers (SVM, kNN, logistic regression) on the 1) original dataset and 2) balanced sample and obtained an accuracy within only 78-82%. Firstly, we divided the dataset on the train sample and test one in proportion 80/20. In experiments, we also tried to use balanced sample by taking 192 objects from all classes randomly.

Question. Let's assume that the classifiers are implemented correctly. Can it be argued that a low percentage of classification confirms the presence of errors in the marking?

Is it possible to statistically prove that class 3 really contains the most errors?

$\endgroup$
3
  • 2
    $\begingroup$ Which analysis showed that objects were erroneously labeled ? what are the "original and balanced samples" ? $\endgroup$
    – J. Delaney
    Commented Jun 2, 2022 at 10:51
  • $\begingroup$ @J.Delaney, I have added details $\endgroup$
    – Nick
    Commented Jun 2, 2022 at 15:18
  • 1
    $\begingroup$ You still didn't explain what is the "original" sample. Is that a different sample? How was it labeled? Can you assume it has the same class distribution as the sample you are looking at? $\endgroup$
    – J. Delaney
    Commented Jun 2, 2022 at 15:52

1 Answer 1

2
$\begingroup$

"Can it be argued that a low percentage of classification confirms the presence of errors in the marking?"

No. If the data from each class overlap in the attribute space to a high degree, then the accuracy will be low even if the data are labelled correctly. Basically it may be an indication that none of the attributes provides sufficient information to discriminate between the classes.

I would recommend trying classifiers that accommodate "label noise" (also investigate "learning from an unreliable teacher"). If that provides a substantial improvement in performance, that would suggest the problem may well be incorrectly labelled training data. If not, it would likely mean that the distribution of patterns belonging to each class have a substantial overlap, and you need to find some more discriminative attributes/features.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.