Interpretation of "one" feature change in a supervised classifier

Question

i'm making experiments using app. 5000 labeled dataset.i'm trying different supervised ML algorithm to evaluate the results.The vector size is 13 with the labels (totally 12 features+1 label) and i have 15 vector of labeled "flower" class. experiments consist of all data set using 10k cross validation. All features are continuous.

1 experiments using the "pure" features of all dataset.
2 experiments using only "one" feature (out of 12) change of the flower class.

i applied naive bayes, C4.5 but all results of 1 and 2 is same, however logistic regression gives different results and lasted longer.

1- To your best experiences, what causes the difference between naive bayes, c.45 and logistic regression, how should i evaluate the results to make the audiences satisfied?

2- if performance is an important metric, and classifier is used for IDS systems, which ML algorithms do you offer?

Edit: More explanation to make the question clear:

we have 8 different class labels. flower + other 7 labels. In experiment 2 we change the only one attribute of the flower class out of 12 attributes (15 flower labeled class stays same but only one attribute is changed.). all other dataset stays same. so we make experiments using logistic regression, naive bayes and c4.5 seperately with two different dataset. (1- with 5000 dataset, 2- other dataset has difference of change in one attribute of flower class, all other classes stays same).

comparision: we have results of situation number 1 and 2, in C4.5 and naive bayes, nothing changes, FP and FN. but logistic regression gives interesting results.

12 0.4 0.4 0.5 2.333 434 12.2 10 2 10 12 12 flower 
........................................... flower
...........................................
........................................... flower  (total 15 flower class.)

// one feature change 2nd feature.

12 0.8 0.4 0.5 2.333 434 12.2 10 2 10 12 12 flower 
........................................... flower
...........................................
........................................... flower  (total 15 flower class,2nd feature all changed.)

For example can i make i comment like that: because C4.5 uses the maximum number of class in leaf nodes, the change of one feature in flower class will not affect/change the leaf node classes However, logistic regression uses ... so we observe this kind of differences.???

Answer 1

Ok, as far as I see your central question can be summarized in the following way:

I transform one feature F in my dataset, but only for those rows where the label is "flower", the rest of the rows remain unchanged. I evaluate NB,C4.5 and Logistic Regression before and after the transformation. The results obtained via NB and C4.5 did not change, but the result via LogisticRegression. Why ?

In general, the transformation of values of one feature (and additionally, restricted only to a subset of the data), may cause nearly anything. Imagine for example that one replaces the old value by a constant or values mainly occurring in another class. In this cases the predictive power of the feature may decrease rapidly. On the other hand, if the values do not overlap with the values of F of the other classes before AND after the transformation, the predictive power is not afflicted at all.

So in general it is hard to explain your results without knowing the whole dataset + algorithm parameterization/implementation. I wonder on the other hand whether the results differ significantly, given that the flower class only occurs 15 times in 5000. Did you analyse the confusion matrix ?

However, I can provide some hints:

• NB in my experience is rather robust. So it may that F has less impact because other features have more predictive power and have already sealed the deal ;). You can e.g. calculate InformationGainRatio to estimate the predictive power before and after the transformation.
• C4.5: Do the trees differ ? If F is not selected as split node or removed during pruning, the results won't be affected at all. If F has been selected it would be interesting to see whether the new split-point can be inverse transformed to get the old splitpoint. Maybe the overlapping of classes at this point (i.e. given the restriction of the dataspace when F is selected as split-node) has not changed.
• Logistic Regression: My knowledge is not thaaat solid here, in the implementations I used the coefficients are evaluated via an evolutionary approach. Maybe the results differ because it ran into a local minima ? In this case on can rerun the learner multiple times (before and after the transformation) to see if/how the results change.

---

Regarding the second question: Since the two questions differ extremely, I recommend to open a new question for IDS. Meanwhile you may some of these links interesting.