0
$\begingroup$

I am predicting a binary variable called "response". Using CART, neural network, ,logistic regression, or any other binary classification algorithm, how can I penalize/greatly reduce the prediction of 1s in my machine learning model? No matter what method I use, my learning model on any random training set always predicts over 99% of the time 1 while there is only 59% of 1s in my data.

$\endgroup$
  • 1
    $\begingroup$ Most ML models do not predict zeros and ones, they predict probabilities. How are you transforming these probabilities into class membership indicators? $\endgroup$ – Matthew Drury Aug 4 '17 at 14:39
  • $\begingroup$ form your description, it seems you are not using the model correctly. adding an explicable example will help others to answer the question. $\endgroup$ – hxd1011 Aug 4 '17 at 14:40
0
$\begingroup$

It is possible to generate data sets where the result that you describe is the best possible answer.

Here is an (artificial) example. Generate two types of data points. 1% of your data will be type 1 and 99% of your data will be type 2. Type 1 and type 2 points are easily distinguishable using standard classifiers. Make a class variable that assigns class = 0 to all points of type 1 and 40/99 of all points of type 2 at random. The remaining 59/99 of the points of type 2 will be assigned class = 1. Because the points of type 2 are identically distributed, no classifier can do better than guessing the majority class (class = 1) for type 2 points. This will produce the result that pretty much any classifier will separate out the type 1 points (getting 1% correct for class = 0) but guess all other points are class = 1 producing the result that you are finding - 99 are classified as class = 1 even though only 59% are actually class = 1.

Of course, this example is quite contrived and not typical of naturally occurring data sets. So what can you do about this? You should start the way that you should start with any such problem - by getting to know your data. Part of why my contrived example works is because most of the class 0 and class 1 points have identical distributions. Is that really true with your data? You might start with feature selection. Presumably, there is some difference between the class=0 and class=1 points in your data and that difference is reflected in at least some of the attributes that describe your points. For every variable, you can check the mean and standard deviation for the two classes. If the two classes differ on some of the attributes, maybe you should build your classifiers using only those attributes. Of course, mean and standard deviation are fairly coarse descriptions of the distributions. You could try looking at density plots by class to find attributes that distinguish class=0 and class=1. Assuming that there is some difference between your class=0 and class=1, it is your job to find what it is that distinguishes them. Study the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.