Feature Selection using (low) MCC

Question

I have approximately 1200 input parameters that I am trying to whittle down with the following rough process:

1) Fit rbf SVM with n = 1200 parameters and calculate Matthews Correlation Coefficient(cross validation with 10 partitions)

2) Fit linear SVM, find the lowest weight (absolute value) parameter

3) Fit rbf SVM with n-1 parameters, compare the average MCC and delete the parameter IF my MCC is the same or better

Unfortunately, I didn't have high predictive power in the first place. With all 1200 parameters, the MCC of the rbf SVM was around 0.018. And deleting just one parameter significantly lowered the MCC, sometimes to 0.0.

I'd like some help interpreting these two things:

1) Why should there be a significant difference in predictive power when I use 1200 parameters versus 1200-1 parameters?

2) What does an MCC of around 0.018 mean? Is it all noise? So there is no true predictive power?

And as a last question: Is comparing the MCC not the best approach? Perhaps I can use ROC, but I don't know why one would be preferred over the other. What else would you suggest and why? Any other comments about my procedure would be appreciated!

Answer 1

Based on my own experience, albeit with a smaller number of inputs than you are using, here are some comments & suggestions that I hope might be helpful to you:

A) Matthews correlation coefficient is generally a good, unbiased metric. I had a very similar problem with zero values of MCC sometimes occurring incorrectly. If you are using python, as i am, then your problem may be the same as mine was, as follows. In calculating MCC, the denominator term involves taking the square root of (TP+FP)(TP+FN)(TN+FP)*(TN+FN), all parts of which are >= 0. In the code, there should be a "> 0" test before evaluating the square root. With TP, FP, TN, FN all being integers, the true value for the denominator of MCC may be a very large number and, although python itself can handle integers of any size, numpy cannot and gives overflow errors in int32. As a result, the denominator calculation for MCC is sometimes an overflow and an apparent negative result, which will then give a zero value for MCC. There are several different solutions to this but effectively they all correspond to forcing TP, FP, TN, FN to "float" in the MCC calculation. For me, this worked fine and removed the erroneous "zero MCC" problem.

B) Using ROC is fine and, from the ROC chart of TPR vs FPR, the difference TPR-FPR is in fact equal to Informedness, a measure of the distance-from-random, which you are obviously seeking to maximize. If you display unit-slope iso-quality lines on your ROC chart, you may find this to be a useful visual display tool to help you.

C) The other good unbiased metric to complement Informedness is Markedness, which is in fact an unbiased version of Precision and is defined as Markednes = TP/(TP+FP) - FN/(TN+FN).

D) The MCC, which you are already using, is just the geometric mean of Informedness & Markedness, so basically you can either use MCC as a single metric, or both Informedness & Markedness together. Whichever you choose, these 3 metrics are generally better than other alternatives such as F1 score because they give a measure of the quality of your ML itself, without the problem of results being biased by changes in input data.

E) Although using MCC as a single metric may be convenient, you can probably gain more insight into what is happening if you look at your ROC chart and values of Informedness & Markedness.

Best wishes.

Answer 2

Just as a record for future readers. ROCR package in R makes calculating MCCs much easier. In the beginning I was manually calculating MCC for each threshold alone and I could not have a good grip on defining the plot points. MCC calculation through ROCR is straightforward. All you have to do is read in your data. Then measure the predictions,0 and 1 can be used to label the predictions as T and F. Then measure the performance before proceeding to plot the performance estimation.

So in ROCR you only have to call two lines to do all of that.

target_cm_table_pred<-prediction(data_item_col, T_or_F_col)
target_cm_table_perf<-performance(target_cm_table_pred,"mat")