Here on the following picture, we can identify two clouds. We can identify a lot of points that are out of the two clouds and can disrupt a bit the the training and test accuracy (%). Is there an unsupervised machine learning model or other techniques we can use to get rid of those points?

I thought to try to train a neural network to maximize inter class distances, but minimize intra class distances, but it is a bit unclear.

enter image description here

To explain the image here, I have over 76 features I pass to a LSTM model. The graph axis you see are the three first principal components (i.e. related to PCA). The red dots are -1 labels and blue dots are the 1 labels.

  • $\begingroup$ Why would you get rid of those points? So that the model performance looks better than it actually is? $\endgroup$ – Aksakal Jul 16 '18 at 18:03
  • $\begingroup$ Yes, this is the main purpose. I have a LSTM model that process the labels, and I am pretty sure the performance will be inscreased if we remove the disrupting points. $\endgroup$ – Jeremie Jul 16 '18 at 18:05
  • $\begingroup$ Ok, I can propose you a simple but inefficient algorithm: remove points from set one by one checking if the performance improved after removal. $\endgroup$ – Aksakal Jul 16 '18 at 18:08
  • $\begingroup$ Ok, figure the graph you see here is a single day. Image I have to remove the disrupted points for ten years of data. It should be exhausting very quickly. $\endgroup$ – Jeremie Jul 16 '18 at 18:10
  • 3
    $\begingroup$ @Jeremie You're missing the subtext of Aksakal's messages. If those are real datapoints, you want to find a methodology that respects them, instead of using a methodology that does not, and pretending they do not exist. Dropping data to increase model performance is not a good idea. If you want to drop the data, you will need to support that decision in another way. $\endgroup$ – Matthew Drury Jul 16 '18 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.