2
$\begingroup$

We are trying to generate a model to summarize Persian news. About 14000 news were summarized with help of humans(supervised) and then we extracted all sentences (about 180000) and labeled them (true if were selected in summarization, false if not). We have also calculated 9 features for these sentences (all features are between 0-1). Finally we used MaxEnt classifier (logistic regression) for binary classification on our data set.

And here is the result :
enter image description here

I really don't know low recall and high precision is normal for our classifier or something is wrong in our work? Can anyone come up with an explanation?

$\endgroup$
5
  • $\begingroup$ What data do you use to train your classifier? Are you testing it on a separate dataset? $\endgroup$
    – Diego
    Apr 4, 2016 at 19:30
  • $\begingroup$ no the same dataset.. $\endgroup$
    – Oli
    Apr 4, 2016 at 19:41
  • $\begingroup$ Then your results, the model itself, is probably not good at all. Try to split the dataset in two and use one part as train and another as test. Then let us know what the precision is. $\endgroup$
    – Diego
    Apr 4, 2016 at 19:47
  • $\begingroup$ actually we are using k-fold validation with k=10 $\endgroup$
    – Oli
    Apr 4, 2016 at 19:50
  • $\begingroup$ Hi @mpiktas , I am working on a smiliar task and I was wondering as you mentioned, you balanced your classes. Do you mean that you balanced your training classes or your test classes or both? $\endgroup$
    – noor h
    Sep 5, 2019 at 8:22

1 Answer 1

2
$\begingroup$

Users often prefer higher precision to higher recall. For example, you (and Google) really want the first 15 or 30 results for a search engine query to be accurate (=high precision), but neither of you are particularly concerned if you miss one or two of the millions on pages on the web (=low recall). Obviously, this depends a bit on the application, but I would imagine summarization is similar.

The bad news is that rapid miner has apparently computed precision and recall for each class separately, and it looks like your recall is quite bad (~13%) on the in-summary class, which is presumably more important than the not-in-summary sentences.

I suspect that this is because you have fairly unbalanced classes--it looks like there are ~3x more not-in-summary examples in your data set. You could potentially tweak the logistic regression's decision threshold (probably 0.5 by default) to give extra weight to the positive examples.

$\endgroup$
3
  • $\begingroup$ two question:1-is there any good operator in rapidminer to balance our data ? or any good algorithm? 2- there is only C parameter in rapidminer logistic regression operator the default value is 1 how can i change threshold? threshold = C ? $\endgroup$
    – Oli
    Apr 4, 2016 at 19:14
  • $\begingroup$ I've never actually used rapid miner, but this may do the trick: docs.rapidminer.com/studio/operators/modeling/predictive/… There should be some way to change the logistic regression's threshold internally too.... $\endgroup$ Apr 4, 2016 at 19:25
  • $\begingroup$ your answer helped me a lot. thanks!! after balancing my dataset recall increased. please see my new question stats.stackexchange.com/questions/205684/… thanksss $\endgroup$
    – Oli
    Apr 5, 2016 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.