0
$\begingroup$

I have read some papers on clustering application to detect outliers in the data set. In many places the euclidean distance between any 2 data points is calculated using minimum , maximum , mean and standard deviation of the data unit. Let's say the data set consists of multiple samples each of 1 hour duration. Now 1 hour of data is summarized to minimum , maximum , mean and standard deviation and euclidean distance is calculated on these parameters.
My doubt is what is the significance and rationale behind selecting the above 4 descriptive statistical parameters for subsequent use in clustering application.

$\endgroup$
  • 1
    $\begingroup$ Could you add a reference? This practice is non standard, though to be clear any distance metric can be used $\endgroup$ – Harsh Feb 20 '16 at 18:41
  • $\begingroup$ I have never heard of using min, max, mean & SD to calculate Euclidean distance, & I cannot see how it is possible. The Euclidean distance between 2 points is the square root of the sum of the squared distances on each dimension. Can you say more about your situation & your data? $\endgroup$ – gung - Reinstate Monica Feb 20 '16 at 19:19
  • $\begingroup$ @Harsh and @gung It was used like d(X,Y) = sqrt( (avgx - avgy )^2 + (stdx - stdy )^2 + (maxx - maxy )^2 + (minx - miny )^2 $\endgroup$ – Soumajit Feb 21 '16 at 2:54
  • $\begingroup$ @Harsh [ arc.aiaa.org/doi/book/10.2514/MSPOPS12 ] Here you can find the paper "New Telemetry Monitoring Paradigm with Novelty Detection" $\endgroup$ – Soumajit Feb 21 '16 at 3:03
1
$\begingroup$

Sure. You can also add other moments such as kurtosis and skew. In essence, this is the approach to clustering time series suggested by Rob Hyndman in this paper, Dimension Reduction for Clustering Time Series Using Global Characteristics where he develops the rationale.

http://www.robjhyndman.com/papers/wang2.pdf

$\endgroup$
  • $\begingroup$ @Johnson Thanks for the reference. But the rationale for selecting "min,max,mean and std deviation" is not mentioned in the paper. $\endgroup$ – Soumajit Feb 21 '16 at 3:06
  • $\begingroup$ "Moments" refer to key statistics of any distribution, even if the metrics you're focused on aren't explicitly mentioned. $\endgroup$ – Mike Hunter Feb 21 '16 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.