My data basically consists of two sets of features:

  • $F_{nl}$: Non-linear selfmade features like a few scorings and counts (10)
  • $F_{l}$: Linear features generated by TfIdf-vectorization of the text (30000)

The classification task is binary. I'm currently using the standard accuracy metric as performance measure, even tho i should slightly modify it in the future to penalize false negatives (since they are more expensive than false positives for the task).

I have trained two different models with each combination of the features {$F_l$}, {$F_{nl}$} and {$F_l, F_{nl}$} which yield the following results:

  • RandomForest trained on $F_{l}$

    • Specificity: 0.46
    • Sensitivity:0.91
    • Miss-Rate: 0.09
    • Fall-Out: 0.54
    • Accuracy: 0.73
    • AUC-ROC: 0.80
  • RandomForest trained on $F_{nl}$

    • Specificity: 0.76
    • Sensitivity:0.82
    • Miss-Rate: 0.18
    • Fall-Out: 0.24
    • Accuracy: 0.80
    • AUC-ROC: 0.88
  • RandomForest trained on $F_{l} + F_{nl}$

    • Specificity: 0.78
    • Sensitivity:0.84
    • Miss-Rate: 0.16
    • Fall-Out: 0.22
    • Accuracy: 0.81
    • AUC-ROC: 0.89
  • Pegasos-SVM trained on $F_{l}$

    • Specificity: 0.72
    • Sensitivity:0.83
    • Miss-Rate: 0.17
    • Fall-Out: 0.22
    • Accuracy: 0.79
    • AUC-ROC: 0.85
  • Pegasos-SVM trained on $F_{nl}$

    • Specificity: 0.33
    • Sensitivity:0.69
    • Miss-Rate: 0.31
    • Fall-Out: 0.67
    • Accuracy: 0.55
    • AUC-ROC: 0.52
  • Pegasos-SVM trained on $F_{l} + F_{nl}$

    • Specificity: 0.33
    • Sensitivity:0.69
    • Miss-Rate: 0.31
    • Fall-Out: 0.67
    • Accuracy: 0.55
    • AUC-ROC: 0.52

For the Pegasos-SVM I have used the framework sofia-ml from google. The RandomForest is from sklearn.

As you can see, the RandomForest classifier performs well on the nonlinear features and gains little improvement when the linear features are added as well. It completely fails when only trained on the linear features. Where as the Pegasos-SVM classifier performs good on the linear features and very bad on the nonlinear features or the combination of both (which is not surprising since its meant for linear-separable classification tasks).

So the actual question is: Is it possible - or what is the best-practice - to combine the nonlinear model of the RandomForest (trained on $F_{nl}$) with the linear model of Pegasos-SVM (trained on $F_l$)?


Without knowing exactly what you did the following will be part speculation but:

  1. With a nonlinear kernel SVM generates its own nonlinear features internally. This may be telling you the features it inferred are better than the handcoded ones - $F_{nl}$. However SVM can overfit, and by augmenting the data with redundancy, i.e. attaching $F_{nl}$, you are inviting it to do so.
  2. With the last two paras, you should check what your code is doing. Getting the exact same numbers is a yellow card for me, especially when the last entry is a somewhat unexpected.
  3. Random Forest doesn't usually overfit. So you typically won't do any worse by adding sensible additional features.
  • $\begingroup$ 1. For the Pegasos-SVM I have used the framework sofia-ml from google. It's using the linear kernel internally. I'm not sure what you mean with 'by augmenting the data with redundancy' since the $F_{nl}$ features are mostly independent from $F_l$. 2. That looks odd for sure. Even tho they are not 100% the same since I've just rounded them on the last two decimals. Please tell me which information I have left out so i can attach them. Thank you for your help! $\endgroup$ – Tak3r07 Sep 27 '16 at 9:17
  • $\begingroup$ Presumably $F_{nl} = G(F_l)$? If so there is redundancy. $\endgroup$ – conjectures Sep 27 '16 at 11:01
  • $\begingroup$ I'm honest I can't realy remember what $G(F_l)$ meant, but if you mean that $F_{nl}$ is just $F_l$ put into a higher dimension like building the polynomials $(1, x_0, x_1, x_0^2, x_1^2, x_0x_1)$, thats not the case. My input data are articles with different properties, its author and the domain plus the content as text. These properties are used for $F_{nl}$. The text ist tfidf-encoded and builds the $F_l$ feature set. $\endgroup$ – Tak3r07 Sep 27 '16 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.