true negative is 0% whereas true positive is 100% correctly classified I used Naive Bayes from Spark's MlLib to train a model and test it on the data (in the form of an RDD). The results were confusing.
the data and results are as follows:
The problem is a binary classification one.
The outcome should be either a label with '0' or '1'.
total number of labels with '0' in the testing dataset - 11774
total number of labels with '1' in the testing dataset  - 246
Code for reference:
from pyspark.mllib.classification import LogisticRegressionWithLBFGS, LogisticRegressionModel
from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.util import MLUtils
from pyspark.mllib.evaluation import MulticlassMetrics

def parsePoint(line):
    values = [float(x) for x in line]
    return LabeledPoint(values[-1], values[0:-1])

data = myRDD.map(parsePoint)

# Split data aproximately into training (60%) and test (40%)
training, test = data.randomSplit([0.6, 0.4], seed=0)

# Train a naive Bayes model.
model = LogisticRegressionWithLBFGS.train(training, 1.0)

#labelsAndPreds = test.map(lambda p: (p.label, model.predict(p.features)))
predictionAndLabels = test.map(lambda lp: (float(model.predict(lp.features)), lp.label))
accuracy =1.0 * predictionAndLabels.filter(lambda (v, p): v == p).count() / test.count()
accuracy

after applying the model and obtaining the predictions :
True Positives - 11774
False Positives - 0
False Negatives - 246
True Negatives - 0
All my '0' labels are correctly classified and 
whereas all the '1' labels are incorrectly classified! 
Now, this is a part of my project and I'm not sure if the results are fine to be submitted.
The code I wrote using Spark's Python API does this: it gets the data from a file and builds the RDD. I just fed this RDD into the Spark MlLib's Naive Bayes documentation provided on the website and the result is as above.
Can someone please tell me if this result is normal?
 A: This result is indeed normal and this is a classic problem in machine learning. It is called unbalanced problem. Those are problems with the number of samples from one class significantly higher than the number of sample from the other class.
Thus what is happening is that the optimization simply assign all to 1's to 0's. At high level it is simple to understand. When your model tries to assign some 1's in the training phase, it immediately incur an error on some of the 0's. This error is higher than assigning all samples to 0's. Now if you compute it, the accuracy is indeed really high almost 100% since you have a strongly unbalanced problem.
There are 2 main solutions to this problems that I know of:


*

*Training multiple models - each of those will have only a sample of the 0's and all the 1's and then average. The number of 0's and 1's samples should be about the same.

*Using weight. Weight each samples with the inverse of the number of time that class appear (1/97000 for 0, and 1/2300 for 1's)
