Context: I'm developing a system that analyzes clinical data to filter out implausible data that might be typos.
What I did so far:
To quantify the plausibility, my attempt so far was to normalize the data and then compute a plausibility value for point p based on its distance to known data points in set D (= the training set):
With that quantification, I can then select a threshold that separates the plausible data from the implausible data. I'm using python/numpy.
- This algorithm cannot detect independent dimensions. Ideally, I could put anything I know about the record into the algorithm and let it find out by itself that dimension X does not influence the plausibility of the record.
- The algorithm doesn't really work for discrete values like booleans or select inputs. They could be mapped on continuous values, but it is counter-intuitive that Select 1 is closer to Select 2 than to Select 3.
What sort of algorithms should I look into for this task? There seems to be a ton of options including nearest neighbour based, clustering based and statistical approaches. Also, I have trouble finding papers that deal with anomaly detection of this complexity.
Any advice is highly appreciated.
Suppose the data consisted of the Height of a Person, Weight of a Person and Timestamp - so it's 3D-Data. Weight and Height are correlated, but the timestamp is completely independent. If I just consider the euclidean distances, I would have to choose a small threshold to fit most of my cross validation data. Ideally, the algorithm would just ignore the timestamp dimension, because it is irrelevant to determine whether a record is plausible, because the timestamp does not correlate with the other dimensions in any way. Any timestamp is plausible.
On the other hand, one could make up examples where the timestamp does matter. For example it could be that value Y for feature X is plausible when measured before a certain date, but not after a certain date.