I apologize for the lengthy question, but this problem has been troubling me for quite some time now and I can't seem to find an answer to another question which directly applies to my situation.

I have been tasked with creating a machine learning classifier with binary output. I will be building the model using Python. I have around 100 unique features before performing any feature engineering, many of which are text features. The data is not a time series. I have decided to use One Hot Encoding to represent these features as the cardinality is relatively small for most of them (< 500) and there is no inherent ordering.

The problem:

  • The training set consists of 5 million+ unlabelled samples, with numeric, binary and text data.
  • The task is to train a classifier to recognize the correct representation of these data points based off the training data, however there are some incorrect/noisy samples in the training set that will either need to be removed or accounted for in some other way.
  • Values can be expected to fall within a certain range based on what is learned from the training data.
  • Feature A has an extremely high cardinality (>50,000), but most of the other features rely on the value of this feature. (e.g. if feature A is X, feature B must be Y and feature C must be Z, again based on training data). One Hot Encoding may not work here as this feature will have to have some importance assigned to it to align with the other features correctly.
  • There are a finite number of valid inputs for feature A, if an invalid value is input, I would like the model to fall back on other features to make its classification.
  • Self consistencies are a major factor here, e.g. for if feature B takes on a value of X feature V cannot have a certain value of Y (these will be one hot encoded, as previously stated.)

The question:

I have had extreme difficulty correctly framing this problem. Up to this point I have been viewing it as an anomaly detection problem as I am looking for data points which differ from their expected values. However, I am unsure what algorithms suit this particular problem. I had been exploring the possibility of using Isolation forests and have been looking into autoencoder networks, however I am not experienced in this area. Any help at all is greatly appreciated.

(This is my first post, feel free to edit any mistakes and if more information is required I will gladly oblige.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.