cross_val_score Scikit Learn not giving expected result

Question

I am trying to perform K Fold cross validation in scikit learn but I'm having a hard time understanding the results returned from it. My objective is to maximize the recall as I am using this to test a fraud detection system with a highly unbalanced dataset (approximately 98% to 2%, being the 2% the fraudulent percentage).

So my question is: how are X and y related? I know X is supposed to contain a vector {n_samples, n_features} where the samples are my data and the features are the variable(s) to predict. In my case I only have one variable so only ine feature: is it fradulent or not? The y is supposed to contain the target, i.e., the values to predict, in my case 0 or 1.

However I get strange results. For example, using this code:

X = np.array([[1], [1], [1], [1], [1], [1], [1], [1], [0], [0], [0], [0], [0], [0], [0], [0]])
y = np.array([1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0    
classifier = LogisticRegression()
scores = cross_val_score(classifier, X, y, scoring='recall', cv=10, n_jobs=1)

I was hoping to have a mean score of 0.75 since from 8 positives i only got 6 of them correctly predicted. However I got a recall average of 0.60

In another example, using this data:

X = np.array([[1], [1], [1], [1], [1], [1], [1], [1], [0], [0], [0], [0], [0], [0], [0], [0]])
y = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

I get a recall average of 0 when I expected an average of 0.50 since only 4 of the 8 positive results were correctly predicted.

What am I doing wrong?

Answer 1

cross_val_score is a helper function that plugs your X and Y inputs into an estimator (that you specify), trains the model, and looks at the results. I suspect that what's happening here is that cross_val_score isn't calculating results directly on the X and Y that you've provided, but rather it's training the logistic regression and comparing the predicted results to your Y instead. Since your X only has a single feature, the resulting models probably aren't going to be very effective.

To get the results you're expecting given those inputs, you'll want to use something like the classification_matrix function, which simply performs precision/recall calculations on whatever data you provide rather than training a model first. Here's what that looks like:

import numpy as np
from sklearn.metrics import classification_report

#Example 1
print ('EXAMPLE 1 RESULTS:')
X1 = np.array([[1], [1], [1], [1], [1], [1], [1], [1], [0], [0], [0], [0], [0], [0], [0], [0]])
#The line below was cut off in your example, but I think this is what it was supposed to be
y1 = np.array([1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
print(classification_report(X1, y1))

print("")

#Example 2
print ('EXAMPLE 2 RESULTS:')
X2 = np.array([[1], [1], [1], [1], [1], [1], [1], [1], [0], [0], [0], [0], [0], [0], [0], [0]])
y2 = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
print(classification_report(X2, y2))

This gives you the recall results you'd expect:

EXAMPLE 1 RESULTS:
             precision    recall  f1-score   support

          0       0.80      1.00      0.89         8
          1       1.00      0.75      0.86         8

avg / total       0.90      0.88      0.87        16


EXAMPLE 2 RESULTS:
             precision    recall  f1-score   support

          0       0.67      1.00      0.80         8
          1       1.00      0.50      0.67         8

avg / total       0.83      0.75      0.73        16