I have a binary dependent variable that I am interested in predicting with historical data. I have 19.000 observations from 1900 and onwards, and I would like to study the relationship between year/decade and the DV. Normally, in my field, what people do is that they code for the decade (1900, 1910, 1920, etc.) without taking into consideration variation within the decades. Also, coding by decade seems a bit arbitrary to me, since it is just a construction of time which has no validity in reality.

Taking that into account, I would like to find another way of grouping historical data than "decade". More specifically, I would like to identify meaningful clusters where the data points group dependending on the value of the DV. In that way, I would theoretically be able to find groups like "1900-1933; 1934-1978; 1979-2019", etc. These different clusters would then constitute a new predictor that I could enter into a GLM as a categorical factor. So the assigned group for each observations needs to be saved as a new variable in the frame.

Another option would of course be to introduce "year" as a numerical factor. However, I am interested in exploring the above explained option.

Does anyone know what type of cluster analysis would be required to be conducted?

Thank you in advance. I am using R for this.



The are methods to detect abrupt change in mean. You could use these to chunk your data.

Basseville, Michèle, and Igor V. Nikiforov. Detection of abrupt changes: theory and application. Vol. 104. Englewood Cliffs: Prentice Hall, 1993.

  • 1
    $\begingroup$ I think you would like to add this as a comment. $\endgroup$ – usεr11852 says Reinstate Monic Mar 2 '19 at 13:40
  • $\begingroup$ The recommendation to use the usual and we'll known abrupt-change-in-mean methods is more than just a comment. $\endgroup$ – Has QUIT--Anony-Mousse Mar 2 '19 at 23:04
  • $\begingroup$ "meaningful" in a sense that the groupings are based on actual observations rather than a priori groupings, like "decade". The different chunks would (I'm guessing) be clustered according to degree of homogeneity. Would it make sense to measure mean when the DV is categorical? $\endgroup$ – mmarttin Mar 3 '19 at 14:05
  • $\begingroup$ Well, a standard quantile approach would also be data based... Usually, I would consider a decade a "meaningful" grouping (as it has a defined meaning), whereas a data-based grouping could be quite arbitrary (1923-1942). $\endgroup$ – Has QUIT--Anony-Mousse Mar 3 '19 at 16:49
  • $\begingroup$ Thank you for your comment. Sure, I understand your point. However, what I mean is that "decade" doesn't really mean anything in relation to your data - you could for example have significant heterogeneity within a decade, and then that grouping no longer makes sense - right? I would like some way of finding a data-based grouping rather than a priori. $\endgroup$ – mmarttin Mar 4 '19 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.