0
$\begingroup$

I have the following issue. I want to perform clustering based anomaly detection on high dimensional timeseries data. To be able to do so I'm performing PCA before clustering, and before that I'm performing standardization of my parameters.

Here is the problem - a number of parameters has a lot of spikes, but these are not outliers, so I cannot remove them just like that. As this is a history of a long (more than a month of recording) time serie it contains a lot more of "flat" data than the spikes. Hence, the spikes act as outliers as there are not that many of them. This causes normalization to mess-up things a bit, as some of the parameters have spikes while others do not. This means that standardization basically "fails" due to the spikes in certain parameters, causing my standardized parameters to have different scale again.

As the normalization is not good, neither is the PCA and clustering of course suffers from parameters on different scale so clusters are hidden due to parameters on higher scale which act as "boosted".

How do I overcome this problem? What is the procedure to make them on more or less same scale when some have these spikes during time?

I should add that at the end I'm trying to find anomalies by clustering data, hopping for either small clusters or "points" which are far from cluster representatives.

$\endgroup$
  • $\begingroup$ Can you filter out the spikes with some sort of lowpass filter, and use the filtered version to compute mean & standard deviation that you need for normalization? $\endgroup$ – amoeba Feb 9 '17 at 9:22
  • $\begingroup$ Well, the thing is, as I mentioned, I probably should not loose them as they represent work of some machine which works periodically for short moments, so if I do the filtering I will be left with not much information (I'll basically get a flat line). This results in being basically the same as completely removing those parameters. I do lack domain knowledge, but I think I can't do that... $\endgroup$ – Marko Feb 9 '17 at 9:31
  • $\begingroup$ No, I meant that you should apply standardization to the original data with spikes, you just use the filtering to get the parameters for standardization. $\endgroup$ – amoeba Feb 9 '17 at 9:32
  • $\begingroup$ Hmm... So you're saying to find the mean and deviation on filtered data and use them to find z-scores on non-filtered data? Very interesting idea. And that shouldn't ruin the reset of procedure (firstly PCA, which needs data to be at least centered)? $\endgroup$ – Marko Feb 9 '17 at 9:35
  • $\begingroup$ Yes, that's what I mean. There is nothing sacred about z-scoring. It's just one possible transformation. Sometimes it makes sense, sometimes not, but there is nothing to prevent you from using any other transformation if it happens to make more sense for you particular data. $\endgroup$ – amoeba Feb 9 '17 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.