I have a question about cluster analysis. There are 3000 companies, which have to be clustered according to their power usage over 5 years. Each company has values for every hour during 5 years. I would like to find out if some companies have the same pattern in usage power over the time period. The results should be used for daily prediction of power usage. If you have some ideas how to cluster time series in SPSS, please share with me.
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ I'd suggest you check out the related links on the right hand side of the page. There are quite a few questions of very similar nature, see Is it possible to do time-series clustering based on curve shape? and Modelling longitudinal data where the effect of time varies in functional form between individuals for just two examples. $\endgroup$– Andy WCommented Aug 11, 2013 at 12:35
-
$\begingroup$ Proc Similarity in SAS can cluster time series. $\endgroup$– forecasterCommented Aug 14, 2013 at 1:55
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
A) Spend a lot of time on preprocessing the data. Preprocessing is 90% of your job.
B) Choose an appropriate similarity measure for the time series. For example, threshold crossing distance may be a good choice here. You probably won't desire dynamic time warping distance, unless you have different time zones. Threshold crossing may be more appropriate to detect temporal patterns, while not paying attention the the actual magnitude (which will likely be very different from company to company).
C) Cluster the resulting dissimlarity matrix using methods such as hierarchical clustering or DBSCAN that can work with arbitrary distance functions.
answered Aug 10, 2013 at 18:07
-
$\begingroup$ can you explain why dynamic time warping distance is no a good option for time series clustering? $\endgroup$ Commented Nov 25, 2018 at 6:22
-
$\begingroup$ That was not a general statement. Whether it is good or not depends on whether you want to allow time warping, or not. $\endgroup$ Commented Nov 25, 2018 at 10:25
$\begingroup$
$\endgroup$
1
You might want to look at Forecasting hourly time series with daily, weekly & annual periodicity for a discussion of hourly data involving daily data and holidays/regressors. You have 5 years of data while the other discussion involved 883 daily values. What I would suggest is that you could build an hourly forecast incorporating regressors such as day-of-the-week; week-of-the-year and holidays using daily totals as an additional predictor. In this way you would have 24 model for each of the 3,000 companies. Now what you want to do is by hour, estimate the 3,000 models using a common ARIMAX structure accounting for the pattern of response around each of the regressors, the day-of-the-week, changes in the day-of-the-week parameters and weekly indicators while isolating outliers. Then you could estimate the parameters globally using all 3000 companies. Perform a Chow Test http://en.wikipedia.org/wiki/Chow_test for constancy of parameters and upon rejection cluster the companies into homogenous groups . I have referred to this as single dimension cluster analysis. Since SPSS has very limited capabilities in time series you might want to look elsehere for software.
-
1$\begingroup$ "vanilla" seems an odd word to use about R; not clear on the translation in more usual R terminology. Any distinction between base R and extra contributed packages from CRAN really doesn't bite experienced users, or even novice ones, as they are equally free and equally accessible. My impression is that someone who had access to SPSS can say fairly easily that something isn't possible in SPSS at present without programming; to say the same about R requires familiarity with all the time series packages. $\endgroup$– Nick CoxCommented Aug 10, 2013 at 19:07