I have a question about calculating correlations with aggregated data and I would be really happy if someone could answer my question.

I have a population (not a sample) composed by more than 100.000 tickets of a customer care service. These tickets contain several information, however I focus on:

1) the keywords added by the user when submitting the ticket (a ticket can be tagged with not more than 10 keywords)

2) the time needed to close the ticket

I'm trying to check whether there exists a correlation between the number of keywords used and the time needed to close the tickets. Thus, I've grouped the tickets according to the keywords used, I've calculated the median of each group (since it is more robust to extreme values) and finally I've calculate the correlation (Spearman) between the number of keywords and these median values, finding a high positive rho.

Since I know that this result cannot be used to infer something about the single tickets (http://en.wikipedia.org/wiki/Ecological_fallacy), I would like to know if what I did make sense and how I could interpret this correlation.

  • 2
    $\begingroup$ You might enjoy this seminal paper on aggregation bias: Gehlke, C. E. and Biehl, K. (1934). Certain Effects of Grouping Upon the Size of the Correlation Coefficient in Census Tract Material. Journal of the American Statistical Association, 29(185):169–170.. It's aging quite well. $\endgroup$ – Alexis Jan 14 '15 at 18:17
  • 2
    $\begingroup$ If you are really sure that your data are the entire population (so, in particular, you have no intention of drawing any conclusions about present or future customer care), then you are not in the business of drawing inferences, but only of describing the data. It sounds like you have a found an interesting description that is meaningful to you, so there's nothing problematic about it at all. Why anyone would engage in such a purely historical exercise is a mystery, though. Perhaps you would like to reconsider whether the data should be treated only as a population? $\endgroup$ – whuber Jan 14 '15 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.