3
$\begingroup$

i'm making experiments using app. 5000 labeled dataset.i'm trying different supervised ML algorithm to evaluate the results.The vector size is 13 with the labels (totally 12 features+1 label) and i have 15 vector of labeled "flower" class. experiments consist of all data set using 10k cross validation. All features are continuous.

1 experiments using the "pure" features of all dataset.
2 experiments using only "one" feature (out of 12) change of the flower class.

i applied naive bayes, C4.5 but all results of 1 and 2 is same, however logistic regression gives different results and lasted longer.

1- To your best experiences, what causes the difference between naive bayes, c.45 and logistic regression, how should i evaluate the results to make the audiences satisfied?

2- if performance is an important metric, and classifier is used for IDS systems, which ML algorithms do you offer?

Edit: More explanation to make the question clear:

we have 8 different class labels. flower + other 7 labels. In experiment 2 we change the only one attribute of the flower class out of 12 attributes (15 flower labeled class stays same but only one attribute is changed.). all other dataset stays same. so we make experiments using logistic regression, naive bayes and c4.5 seperately with two different dataset. (1- with 5000 dataset, 2- other dataset has difference of change in one attribute of flower class, all other classes stays same).

comparision: we have results of situation number 1 and 2, in C4.5 and naive bayes, nothing changes, FP and FN. but logistic regression gives interesting results.

12 0.4 0.4 0.5 2.333 434 12.2 10 2 10 12 12 flower 
........................................... flower
...........................................
........................................... flower  (total 15 flower class.)

// one feature change 2nd feature.

12 0.8 0.4 0.5 2.333 434 12.2 10 2 10 12 12 flower 
........................................... flower
...........................................
........................................... flower  (total 15 flower class,2nd feature all changed.)

For example can i make i comment like that: because C4.5 uses the maximum number of class in leaf nodes, the change of one feature in flower class will not affect/change the leaf node classes However, logistic regression uses ... so we observe this kind of differences.???

$\endgroup$
4
  • 1
    $\begingroup$ I have some questions (just to be sure). 1. You have 5000 records, 13 features (12 predictors + 1 response) and the response variable is binary with values "flower" (15 times) and "not flower"(5000-15 times) => is this correct ?. 2. I do not understand experiments number 2. Can you try to explain it again ? 3. Do you mean that 1 and 2 did not differ for C4.5 and NB respectively or do you mean that the results of NB and C4.5 did not differ for 1 and 2 respectively ? $\endgroup$
    – steffen
    Mar 29, 2011 at 6:29
  • $\begingroup$ @steffen, i edited thanks for the great questions. $\endgroup$
    – berkay
    Mar 29, 2011 at 17:29
  • $\begingroup$ Sorry, I am not able to understand this part: In experiment 2 we change the only one attribute of the flower class out of 12 attributes (15 flower labeled class stays same but only one attribute is changed.). What is the "attribute of flower class" ? Maybe you can provide some example data (i.e. showing one or two exemplary rows before and after the change). What is "15 flower labeled class" ? 15 rows with label = flower ? Maybe can one of those who have upvoted this question explains what it means. $\endgroup$
    – steffen
    Mar 31, 2011 at 6:36
  • $\begingroup$ @steffen, we have a class called flower and it has 15 row vectors consisting 12 features, we change only one of the features out of 12 in flower class and again make experiments to evaluate the results of 1 and 2. $\endgroup$
    – berkay
    Mar 31, 2011 at 17:13

1 Answer 1

2
+25
$\begingroup$

Ok, as far as I see your central question can be summarized in the following way:

I transform one feature F in my dataset, but only for those rows where the label is "flower", the rest of the rows remain unchanged. I evaluate NB,C4.5 and Logistic Regression before and after the transformation. The results obtained via NB and C4.5 did not change, but the result via LogisticRegression. Why ?

In general, the transformation of values of one feature (and additionally, restricted only to a subset of the data), may cause nearly anything. Imagine for example that one replaces the old value by a constant or values mainly occurring in another class. In this cases the predictive power of the feature may decrease rapidly. On the other hand, if the values do not overlap with the values of F of the other classes before AND after the transformation, the predictive power is not afflicted at all.

So in general it is hard to explain your results without knowing the whole dataset + algorithm parameterization/implementation. I wonder on the other hand whether the results differ significantly, given that the flower class only occurs 15 times in 5000. Did you analyse the confusion matrix ?

However, I can provide some hints:

  • NB in my experience is rather robust. So it may that F has less impact because other features have more predictive power and have already sealed the deal ;). You can e.g. calculate InformationGainRatio to estimate the predictive power before and after the transformation.
  • C4.5: Do the trees differ ? If F is not selected as split node or removed during pruning, the results won't be affected at all. If F has been selected it would be interesting to see whether the new split-point can be inverse transformed to get the old splitpoint. Maybe the overlapping of classes at this point (i.e. given the restriction of the dataspace when F is selected as split-node) has not changed.
  • Logistic Regression: My knowledge is not thaaat solid here, in the implementations I used the coefficients are evaluated via an evolutionary approach. Maybe the results differ because it ran into a local minima ? In this case on can rerun the learner multiple times (before and after the transformation) to see if/how the results change.

Regarding the second question: Since the two questions differ extremely, I recommend to open a new question for IDS. Meanwhile you may some of these links interesting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.