I have a particular segment of temporal data for 3 days. It looks like this:

Day_1_Hour_14 = [d1x1,d1x2,...,d1xn] and [d1y1,d1y2,...,d1yn]
Day_2_Hour_14 = [d2x1,d2x2,...,d2xn] and [d2y1,d2y2,...,d2yn]
Day_3_Hour_14 = [d3x1,d3x2,...,d3xn] and [d3y1,d3y2,...,d3yn]

When I use these 3 days 1 hour each worth of data on an unsupervised algorithm like DBSCAN, is it better to give the x_axis as an increasing order of index with their respective y_value?


x_axis_total = [1,2,3,4.,...,n] from d1x1 to d3xn
y_axis_total = [d1y1,d1y2,,.d1yn,d2y1,d2y2,...,d3y1,d3y2,..,d3yn]

And then I pass x_axis_total and y_axis_total to DBSCAN

I am quite confused about how to pass the data to the algorithm. Or is it better to include the actual timestamp of each observation as the x_value? When I use the timestamp the data points are aligned in a straight line from 1 - n in 4 rows in the graph and DBSCAN does not work good when applied like this.

Any suggestions will be appreciated

If I make the x_axis in an increasing order I get this result [Day 1 - 3 of Hour 14]: enter image description here

If I use the timestamp as x_axis I get this result [Day 1 - 3 of Hour 14]:

enter image description here

My aim is to detect Outliers in my data and the data is vehicular traffic data - It is contextual. That is the reason I am selecting 4 similar days of the same hour for this example.

My main question is am I losing any information if I do it the first way I mentioned? That is, the x_axis in increasing order from 1-n because using timestamp gives me bizzart results.

Is there a better way to do it?

  • $\begingroup$ Do you wish to detect outlier datapoints or outlier time-series? It seems to me that you are right now clustering the datapoints and not the time-series. $\endgroup$ – Nikolas Rieble Mar 7 '18 at 16:26
  • $\begingroup$ By the way: You would always want to standardize all features before passing them to DBSCAN. Otherwise the feature with the highest variance dominates the clustering result. $\endgroup$ – Nikolas Rieble Mar 7 '18 at 16:36
  • $\begingroup$ @NikolasRieble - I am looking to cluster the data points. I currently have only two features. The x_index and the speed which is the y_axis. Does this seem correct? $\endgroup$ – RPT Mar 7 '18 at 16:40
  • $\begingroup$ Alright, this clears up a lot. You want to detect irregular behavior within a time-series. This is not equal to time-series clustering. I would recommend you to use a sliding window approach instead of considering only a single datapoint. Thereby you can consider the previous points as well and detect things such as a sudden increase of speed. I would neglect the time stamp for all further processing since you are not so much interested in when a behaviour occurs rather than if a deviant behaviour occured. $\endgroup$ – Nikolas Rieble Mar 7 '18 at 19:14
  • $\begingroup$ Thanks for the reply, Nikolas. Could you explain briefly how I could use a sliding window approach here? Or refer me to some useful paper? Thanks a lot. Also I could easily get back the time at which the deviant behaviour has occurred, once clustered and anomalies have been detected $\endgroup$ – RPT Mar 7 '18 at 19:24

Your question would benefit from more explanation regarding the data as well as the target. I am not certain whether I fully understand your question, but i will summarize my thoughts to your problem.

It seems you wish to work on the task of time-series clustering.

1) Using DBSCAN:

  • You did not specify the dissimilarity measure that you are using. The quality of the clustering results strongly depends on the measure you choose to compare the time-series. A standard measure to use would be Euclidean Distance, yet the are quite a few reasons why not to use Euclidean Distance on time-series best explained here by Eamon Keogh (This is a link to one of his tutorials on time-series analysis).
  • Although DBSCAN is quiet a wonderful algorithm, it is highly sensitive to its parameters. Therefore I would suggest to firstly use simpler algorithms. You can find alternatives here.

2) Time-series clustering: You can cluster time-series either directly on the time-series data using dissimilarity measures such as Dynamic Time Warping (DTW) or you can transform your time-series into a feature space (such as mean, max, min, kurtosis, skewness per dimension) and use Euclidean Distance in the feature space. The choice of relevant/appropriate features depends on the nature of the data - which you did not specify.

If you wish to get more specific answers, please plot some time-series, explain what sort of clusters you expect and describe the origin of your data.

  • $\begingroup$ Thanks for your reply. I have edited my question. Regarding the DBSCAN - I am aware of the parameters selection process. I am mainly confused on how to pass the data to the algorithm. $\endgroup$ – RPT Mar 7 '18 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.