I am trying to perform K Fold cross validation in scikit learn but I'm having a hard time understanding the results returned from it. My objective is to maximize the recall as I am using this to test a fraud detection system with a highly unbalanced dataset (approximately 98% to 2%, being the 2% the fraudulent percentage).
So my question is: how are X and y related? I know X is supposed to contain a vector {n_samples, n_features} where the samples are my data and the features are the variable(s) to predict. In my case I only have one variable so only ine feature: is it fradulent or not? The y is supposed to contain the target, i.e., the values to predict, in my case 0 or 1.
However I get strange results. For example, using this code:
X = np.array([[1], [1], [1], [1], [1], [1], [1], [1], [0], [0], [0], [0], [0], [0], [0], [0]])
y = np.array([1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0
classifier = LogisticRegression()
scores = cross_val_score(classifier, X, y, scoring='recall', cv=10, n_jobs=1)
I was hoping to have a mean score of 0.75 since from 8 positives i only got 6 of them correctly predicted. However I got a recall average of 0.60
In another example, using this data:
X = np.array([[1], [1], [1], [1], [1], [1], [1], [1], [0], [0], [0], [0], [0], [0], [0], [0]])
y = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
I get a recall average of 0 when I expected an average of 0.50 since only 4 of the 8 positive results were correctly predicted.
What am I doing wrong??
