0
$\begingroup$

I am classifying Portuguese tweets in to three classes, news, noise and relevant. I have used the weka gui to identify a classifying pipeline that gave good results. STWV -> Attribute Selection -> SMOTE -> Random Forest

I have programmed this as follows:

public class TrainBuildClassifier {

public static void main(String[] args) throws Exception {

    DataSource dataSource = new DataSource("data/tweets.arff");
    Instances instances = dataSource.getDataSet();
    instances.setClass(instances.attribute("class"));

    StringToWordVector filter = new StringToWordVector();
    filter.setWordsToKeep(100000000);

    NGramTokenizer tokenizer = new NGramTokenizer();
    tokenizer.setNGramMinSize(1);
    tokenizer.setNGramMaxSize(4);
    filter.setTokenizer(tokenizer);

    SMOTE smote = new SMOTE();
    smote.setPercentage(100);

    AttributeSelection as = new AttributeSelection();
    Ranker search = new Ranker();
    InfoGainAttributeEval evaluator = new InfoGainAttributeEval();
    search.setNumToSelect(-1);
    search.setThreshold(0);
    as.setEvaluator(evaluator);
    as.setSearch(search);

    System.out.println("Applying Filters");
    MultiFilter multi = new MultiFilter();
    multi.setFilters(new Filter[] {filter, as, smote});

    System.out.println("Creating Classifiers");
    FilteredClassifier fc = new FilteredClassifier();
    RandomForest rf = new RandomForest();
    rf.setNumTrees(10);
//  NaiveBayesMultinomial nb = new NaiveBayesMultinomial();
    fc.setClassifier(rf);
    fc.setFilter(multi);


   // Train and build the classifier
    System.out.println("Building Classifier.....");
    fc.buildClassifier(instances);

    ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream("data/rf.model"));
    out.writeObject(fc);
    out.flush();
    out.close();
    }
}

The issue I am having is that the classifier always returns the same class. Even with a cross validation accuracy of approx 80%.

Here is the arff header:

@RELATION tweets

@ATTRIBUTE tweet string
@ATTRIBUTE class {news,noise,relevant}


@DATA
'retweet ir zika pego bom prevenir :joy: :joy: url',noise

And the tweets are passed to the classifier by the following method:

public Category classify(String document) throws Exception {
    String[] features = normalize(document);
    Instances instance = featuresToInstance(features);
    Filter filter = fc.getFilter();
    Filter.useFilter(instance, filter);
    return Category.fromPrediction((int)fc.classifyInstance(instance.instance(0)));
}

If I change the classifier to be multinomial bayes everything seems to work okay. However Random Forest should deliver much better accuracy.

Am I doing something obviously wrong?

$\endgroup$
2
$\begingroup$

Things started to go wrong when you chose to do classification instead of probability estimation. Details are at http://www.fharrell.com/2017/01/classification-vs-prediction.html

$\endgroup$
  • 1
    $\begingroup$ Thanks for the response. Unfortunately I am required to use a random forest classifier for this work. Is this something you have come across before? My cross validation accuracy is 82%. Yet the classifier always chooses News as the class. $\endgroup$ – calmcalmuncle Feb 3 '17 at 13:44
  • 1
    $\begingroup$ Why "required"? Does the requestor have an interest in science? $\endgroup$ – Frank Harrell Feb 3 '17 at 13:46
  • $\begingroup$ Its was requested as a specification point $\endgroup$ – calmcalmuncle Feb 3 '17 at 13:52
  • $\begingroup$ Please state the specification. $\endgroup$ – Frank Harrell Feb 3 '17 at 16:38
  • $\begingroup$ Independently of whether you apply this very narrow definition of the word classification, or the more widely used definition which does also include algorithms that compute probability estimates and then classify based on those, there are some things you can try here. It matters more what you do than how you decide to call it. $\endgroup$ – David Ernst Feb 6 '17 at 13:57
1
$\begingroup$

Whenever you have high classification accuracy in spite of ignoring some classes, it means that you have class imbalance. Accuracy can be high while ignoring a minority class. Class imbalance is only a problem if you also have cost imbalance (ignoring very rare but very severe cases is unacceptable but ignoring very rare and yet benign cases is the rational thing to do).

Random forests are a relatively complex model. They are only justified if they empirically outperform simpler models like the logistic regression. Perform a baseline comparison and renegotiate the requirements based on that evidence if necessary.

I could help you better if you told me the reasoning behind each of the steps in your pipeline. Nevertheless, the following should apply: Be sure to use an adequate performance metric, not accuracy if you have imbalance. This metric could be macro averaged F measure which gives equal weight to each class or could be based on the different misclassification costs if they are known to you.

Random forest can output probability estimates (based on the number of trees and the labels they give for a record and the confidence with which each tree gives it's label). You can use those probabilities to adjust the classification cutoff so that it conforms with your misclassification costs. This is easier to do in binary classification, so maybe you can think about first filtering the useless records out and then finding the most interesting records among the news records.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.