7
$\begingroup$

I have a longitudinal (panel) dataset for investment growth for 120 countries covering the time from 1960-2008. Essentially it's viewed as 120 time series.

What I am interested in is to group countries based on their shape of their growth curves over time. Thus whether they share similar Shape of their curves are the only criteria I need for grouping those countries.

I have tried KmL package (K-means for Longitudinal Data), but it seems that (please correct me if I am wrong) this methodology produces the result that group countries exhibiting similar (investment growth) mean value (or magnitude), not exactly according to the similar shape. For example, KmL tends to group countries with high investment growth, median average investment growth, low investment growth, etc. The countries within those groups may have very different shape of curves over time.

What I am looking for is regardless of the absolute value of investment growth. As long as the two countries exhibit similar pattern of their growth over time curve, they should be grouped together in one group.

Could anyone tell me a way to implement this clustering? I have noticed from previous posts that cointegration test may work. Any suggestions will be greatly appreciated!

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Calculate some characteristics of shape for each time series and then cluster on them. One simple solution would be to do principal component analysis and do clustering on loadings on first few principal components. $\endgroup$
    – mpiktas
    Commented Sep 25, 2012 at 6:07

6 Answers 6

Reset to default
3
$\begingroup$

If you z-standardize each of your series, $(X_i-\bar{X})/\sigma$, that is, unify level of the series firstly and swing of the series secondly, then the only difference that remains is the difference in shape. Compute euclidean distances (or similar measure) between 120 series and perform hierarchical clustering. You might also want (maybe) to do mild smoothig of the curves prior all.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Do the estimates, $\overline{X}$ and $\hat \sigma$ account for the temporal dependence or are they the naive sample mean and standard deviation? $\endgroup$
    – Macro
    Commented Jun 26, 2012 at 0:52
  • $\begingroup$ @ttnphns, what would you suggest for smoothing? I have a pretty erratic series (daily download counts of documents), which I feel might benefit from some smoothing. $\endgroup$
    – pocketfullofcheese
    Commented May 16, 2014 at 2:53
2
$\begingroup$

All of the recommendations so far rely on the standard moment-based approaches to time series analysis and all are a type of HAC model. The question, though, specifically queried the patterning or shape in the data. Andreas Brandmaier at the Max Planck Institute has developed an non-moment-based, information and complexity theoretic pattern analysis time series model that he calls permutation distribution analysis. He's written an R module to test the similarities in shape. PDC has a long history in biostatistics as an approach to two group similarities. Brandmaier's dissertation was on PDC and structural equation modeling trees.

pdc: An R Package for Complexity-Based Clustering of Time Series, J Stat Software, Andreas Brandmaier

Permutation Distribution Clustering and Structural Equation Model Trees, Brandmaier dissertation PDF

In addition, there is Eamon Keogh's machine learning, iSax method for this.

http://www.cs.ucr.edu/~eamonn/

Improve this answer
$\endgroup$
1
$\begingroup$

You should first differentiate your time series, i.e. consider $X_t = S_t - S_{t-1}$. Then, a correlation-based clustering will do it. You should use Spearman correlation rather than the Pearson one, as it is more generic and robust to strong variations. If you suspect that the strength of variations matters somehow, you could use a correlation+distribution clustering, i.e. each time series can be viewed as a random variable, and the values of its variations $X_t$ are realizations. Sklar's theorem asserts that the whole information, assuming i.i.d. sampling of the $X_t$, can be captured by a "pure correlation" and distribution separately, cf. this paper which illustrates the approach by clustering financial time series, and eventually this link for a snippet of Python code.

Improve this answer
$\endgroup$
0
$\begingroup$

Alternatively (tentative suggestion!!!): could you not create a new variable, delta.growth, which is growth at t=i - growth at t=i-1, for each time point t=0 ... t=n. I am not sure exactly how this would differ to Z-score assessment. Would be interesting to find out!

You could also use joint trajectory interpretation to model both absolute growth and delta.growth. This should give weighting to both shape and value, although I am naive to this approach.

Improve this answer
$\endgroup$
0
$\begingroup$

There is the dynamic time warping R package dtw which allows you to compare shapes of curves and goes beyond one to one matching.

There's also the dtwclust R package:

Time series clustering along with optimized techniques related to the Dynamic Time Warping distance and its corresponding lower bounds.

Improve this answer
$\endgroup$
0
$\begingroup$

I might be late for this question but there is now an R package called KmlShape (produced by the same author that made the Kml package), that clusters trajectories based on their shape. It uses the K-means algorithm with the Fréchet distance.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.