I am working on a current project where I need to classify and detect data that doesn't follow the standards and parameters required. I have past data which is the data that I will train my model with. I know what data is wrong and why, so I have labeled is as isWrong and assigned it an either 0 or 1. I know this is a supervised machine learning project, more specifically, a binary classification problem.

I decided to first work with dummy data before going into the real data to correctly understand the functionality of these algorithms. I have the following data set:

Name    Age    Office#   SSN   isWrong

Name: String

Age: int

Office: int

SSN: int

isWrong: int

A record is wrong if the person has an Age over 60 years old or under 18 years old. A record is also wrong if the Office# is negative or greater than 2. Either one of those condition makes the record to be labeled as wrong.

I've tried both Logistic Regression and Naïve Bayes. I trained my model with this data set and then decided to test it with another data set that follows the same schema (without the isWrong column) and same pattern. However, when I print the prediction column, I only get at most three out of ten bad data. In other words, I don't get the prediction I wanted because it failed to classify some wrong data as wrong, meaning that my model gave a label of 0 to an obviously wrong data.

Why is this? Is there something I need to specify before going into my training and testing part?

For training I've done something as simple as this:

LogisticRegression lr = new LogisticRegression().setLabelCol("isWrong");

EDIT: Here is a sample of the dummy, example data (top 25)

Name    Office  Age    SSN     isAnomaly
  F1        1   32  42376794    0
  F2        2   61  72353599    1
  F3        1   35  49001025    0
  F4        0   25  25613860    0
  F5        1   36  85720367    0
  F6        2   52  85725161    0
  F7        2   48  78016430    0
  F8        2   53  89762357    0
  F9        2   51  84866475    0
 F10        0   24  24062706    0
 F11        4   34  46233821    1
 F12        0   21  17686431    0
 F13        2   51  85547952    0
 F14        0   27  32159403    0
 F15        1   37  52685792    0
 F16        1   35  47836229    0
 F17        1   67  52770341    1
 F18        0   21  17104127    0
 F19        1   41  63042591    0
 F20        1   36  50374422    0
 F21        0   25  26303816    0
 F22        0   22  19645563    0
 F23        2   16  69963996    1
 F24        0   62  19884506    1
 F25        0   26  28498270    0
  • 1
    $\begingroup$ Are the actual criteria for a wrong record that simple? Because it's not clear why you need to build a classifier for this. $\endgroup$ – Dan Hicks Jul 21 '17 at 19:46
  • $\begingroup$ No, not at all. As I mentioned I'm doing this with dummy data so I can correctly understand classification algorithms and know what to do when I get the more complex-real data. I realize this is a simple example, but I thought the simpler the easier it would be to apply the algorithm. What are the specifications I need for building a classifier? $\endgroup$ – Guillermo Herrera Jul 21 '17 at 20:12
  • $\begingroup$ In that case, it would be good to provide your simulated dataset / reproducible code to generate your simulated dataset. $\endgroup$ – Dan Hicks Jul 21 '17 at 21:00
  • $\begingroup$ Gotcha. I will put it on the question. Thanks for the feedback! $\endgroup$ – Guillermo Herrera Jul 21 '17 at 21:36
  • $\begingroup$ EDIT: Dataset added $\endgroup$ – Guillermo Herrera Jul 21 '17 at 21:45

For (standard) logistic regression, there is an assumption being made that the true decision boundary is linear. This isn't the case with your data, because the chance of being "wrong" does not strictly increase with Age or with Office# since you have upper and lower limits.

Using logistic regression after adding squared terms should improve your classifier (approximate a square decision boundary with an ellipse), though a better model would be something like nearest neighbors or decision trees/random forest; the latter would also quickly rule out the other predictors making a difference. (Incidentally, the predictors are not independent for naive Bayes, since SSN should be highly correlated with Age).

  • $\begingroup$ Thanks! A Random forest worked fine and I'm starting to read more about them. I'm going to use this approach when working with my real data. $\endgroup$ – Guillermo Herrera Jul 21 '17 at 22:34
  • $\begingroup$ Hi Kevin. Going back to the same question. So far the program works in detecting values in a column within a specific range, like being lower or higher than something. However, is it possible a GBT or Random Forest can detect differences between columns? So let's say that if the difference between two numerical columns is greater than a certain threshold, that record is considered bad. Can these algorithms be able to detect that? $\endgroup$ – Guillermo Herrera Jul 26 '17 at 16:29
  • $\begingroup$ Hey Guillermo; in short, the answer is yes. One way to think of it is that these algorithms are generalizations of decision trees. $\endgroup$ – Kevin Jul 26 '17 at 17:08
  • $\begingroup$ Okay that's great, but how can I achieve that? I'm using a GBT library already. Is there something I need to update or modify? Or should I treat the two numerical columns as one? $\endgroup$ – Guillermo Herrera Jul 26 '17 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.