I want to cluster a set of 512 time series. The time series have sampling intervals of 1 day over a time period of 5 years. Thus, each time series consists of about 1800 samples. However, many of the samples are zero. So, my question is: Is it ok to aggregate the samples to larger sampling intervals, e.g. days or even years, despite the information loss that comes with aggregation? Isn't aggregation some form of data tampering? If not, how do I chose the appropriate temporal unit for aggregation?

  • I don't think there can be an "OK" here without knowing much more about what the data are like, what interests you and what criteria you want to drive the clustering. Assuming that 0 is a valid value, and not a code for "missing" or "no data" then the zeros are part of the data too. – Nick Cox Aug 28 '13 at 12:28
  • It is crime data. 0 is totally a valid value, indicating no crime has been recorded at this time stamp. Thus, you think aggregating is bad thing here? I tend to agree, however, without aggregation i have to deal with about 1800 dimensions. – Julian Aug 28 '13 at 12:45
  • My goal is to find clusters of time series with similar criminal trends. The problem is, with only 512 time series, but 1800 dimensions it is hard to find reasonable clusterings since the time series are very dissimilar. Aggregatings the samples in order to reduce the dimensions might lead to more reasonable clusterings. – Julian Aug 28 '13 at 12:53
up vote 2 down vote accepted

Aggregation like many other techniques has both statistical and non-statistical aspects. Looking at it statistically, what's evident is the loss of information entailed. (Here I don't try to provide a rigorous or formal definition of information, but it could be done.) It's elementary but fundamental that aggregation is typically irreversible: you can go from daily totals to weekly totals, but the reverse is usually impossible. (There are exceptions: if the weekly total is 0, then unless negative values are possible, each daily total must be 0 too.)

The non-statistical aspect should also be evident: you (should) aggregate over details you don't care about or don't want to try to interpret. Thus in the crime examples, zeros for some days may seem unsurprising or uninteresting details, but even aggregating up to weekly totals discards all possibilities of investigating time of week as a source of variation (e.g. whether crimes are more common or less common at certain times of the week). So, you have to choose depending on priorities and possibilities. Aggregation is mostly a matter of what's interesting and practical; there's no statistical bird to sit on your shoulder to tell you right or wrong.

Clustering of time series is fraught with difficulties. Most clustering methods, like most multivariate methods, pay no attention to any sequence or series information in the data. I don't have specialist criminological knowledge to know what to advise, but I would not try clustering the time series themselves. I would try reducing each time series to a few descriptors, starting with some measure of level (mean etc.) and following with some overall measure of trend. It would seem quite likely to me that the data might benefit from transformation.

Is it ok to aggregate the samples to larger sampling intervals, e.g. days or even years, despite the information loss that comes with aggregation?

Isn't aggregation some form of data tampering? If not, how do I chose the appropriate temporal unit for aggregation?

I am not a criminologist but i am a researcher of crime analysis. It really depends on the method you are using to cluster. There are methods that can dimensionally reduce a time series with little loss of data at all such as PAA. I am conducting a similar data mining exercise and have found the need to aggregate daily crime into Monthly crime. I am also using a bioinformatic technique to cluster the series.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.