Mining survey data to discover high and low performing cohorts

Question

I'm analyzing datasets of users that range in size from 5,000 users up beyond 150,000 users, and each of these users have at least 32 data-points (sometimes more, sometimes a few less) with (about) 5 possible responses for each point. Each of these users have a single value that evaluates how they contribute as a high or low performing user.

I'd like to programmatically identify cohorts, defined by some combination of data-points and responses, which are in the 90th and 10th percentiles by performance and sample size. All the methods I've explored end up taking way too much time (3+ hours for a small set of ~7,000 users) to complete at a depth of 3 dimensions (and throwing out all 1 and 2 dimensional cohort definitions), which will not scale.

Are there any algorithms or statistical/mathematical principals I can look at/read about and/or experiment with to try to identify these cohorts more efficiently and faster?

The methods I've attempted thus far have all involve iterating over each user's responses and building a list of the 3-dimensional cohort definitions that the user could fit in, then incrementing counts for each of the cohorts the user fits within which corresponds with how well the user performs. I've tried different methods to prune down the dimensions that yield less reliable results (as defined by a low incidence rate/number of users in the cohort). I've tried different optimization strategies to optimize the amount of loops I have to execute over each user, with little performance increase.

I can't help but think there's some statistical mathematics rule or theory that will help analyze this data set more efficiently.

Any information, advice, or suggestions are very much welcomed and appreciated. Thanks!

Edit 7/20/13

To answer some questions about objectives and the data and format:

The objective here is to identify the most interesting clusters of people (say the 90% percentile of clusters, ranked by NPS score) without really knowing where to start. For any given data set it could be any cluster.

The data is stored as a map of QuestionID => ResponseID entries for each user. There are some caveats to that, sometimes the ResponseID value is list of responses that the user has selected for that question (in the case of multi-response questions), and sometimes the ResponseID value is another map of SubQuestionID => ResponseID values for that question. The data structure doesn't next any further than that, though. The data is serialized into a JSON object and written to a file, and I'm loading the data out of that file.

I've also flattened the data structure down to a 1 dimensional map of QuestionID[.SubQuestionID] => ResponseIDs, where ResponseIDs are represented as a map of ResponseID => True (in an effort to optimize the speed in which I could check the users' response).

I put an example record at the bottom of this post to help visualize/communicate the data format.

To address some of the questions regarding what the data tells us, it's (just) survey response data. As you'll see from the data structure below, there are specific responses to each question. The data we collect from users is mostly the same across all data-sets, but there are filters that are executed to only include users we know we're looking for, based on some other metrics that aren't included in the data file. This is done to limit the working set to something more manageable than the larger set (which is orders of magnitude larger).

To address the specific question about the NPS value, yes we have the 0-10 values stored, but in the NPS algorithm/methodology the actual values don't matter and are mostly a psychological representation of the responses for the survey taker, so we just map the response values to the group they fall in to.

All of the responses are stored categorically (if I'm understanding the definition of a categorical response correctly), though some of the questions' responses have a progressive order to the responses. Eg, one of the questions is regarding the highest education a user has completed, and the responses are something like "Some high school", "High school", "Some College", "Associates Degree", etc, and so on. Even age is put into a bucket so we're not dealing with 50 different responses for a single question (age).

Peter Ellis asked if it "would be good enough to have a predicted performance for each individual given their values on the 32 categories? You could easily enough identify the risk (positive and negative) levels of each of the variables."

I think that question is leading into the direction of probabilistically determining which clusters are going to perform the best and only using computation cycles to calculate the performance of those clusters. I think the answer here is yes. If there was a reasonable method of calculating, or knowing, what the performance gain/reduction (risk, I guess?) implications were of some cluster definition, without having to test and calculate all users, that would be a method that we could put to use.

As stated above, the objective is to find top/bottom performing clusters. We don't have to match all of the top/bottom performers 100%, but if we were reasonably certain (like 80%+ certain, or something) that the clusters we uncovered were in the top 90% or bottom 10%, that would be good enough.

I hope that helps clear things up. I am not sure where to go from here, so I'm hoping y'all can at least show me the right direction that I need to go in, or some possible routes to get me to where I want to be.

Thanks Again!

---

The data. This is for a single user record, all the other user records look largely similar. If a larger set would be significantly more beneficial, I can pull some together. I've been focusing on the user-data solely because it's a known set; I think that if I used the definition of the questions/responses to iterate through my users, I'd waste much more CPU cycles generating/evaluating cluster definitions which my data-set doesn't have any representation of.

The un-flattened version:

{
   "NPS":"promoter",
   "11279":[13204,13205,13206,13207,13209,13210],
   "2":12,
   "3":17,
   "4":19,
   "23":148,
   "11303":{
      "1":13418,
      "2":13419,
      "3":13419,
      "4":13424,
      "5":13424,
      "6":13424,
      "7":13418,
      "8":13424,
      "9":13418,
      "10":13424,
      "11":13424,
      "12":13420,
      "21":13419
   },
   "11305":{
      "19":13435,
      "20":13434
   },
   "11304":{
      "13":13425,
      "14":13427,
      "15":13426,
      "16":13426,
      "17":13429,
      "18":13433,
      "22":13433,
      "23":13433,
      "24":13433
   },
   "11306":[13448],
   "11307":[13450,13453],
   "11309":13473,
   "11308":[13459,13460,13461,13466,13469],
   "11999":[17031],
   "12111":18235,
   "age":"55-64",
   "gender":"f"
}

and, the flattened version:

{
   "2":{"12":true},
   "3":{"17":true},
   "4":{"19":true},
   "23":{"148":true},
   "11279":{"13204":true,"13205":true,"13206":true,"13207":true,"13209":true,"13210":true},
   "11303.10":{"13424":true},
   "11303.1":{"13418":true},
   "11303.11":{"13424":true},
   "11303.12":{"13420":true},
   "11303.2":{"13419":true},
   "11303.21":{"13419":true},
   "11303.3":{"13419":true},
   "11303.4":{"13424":true},
   "11303.5":{"13424":true},
   "11303.6":{"13424":true},
   "11303.7":{"13418":true},
   "11303.8":{"13424":true},
   "11303.9":{"13418":true},
   "11304.13":{"13425":true},
   "11304.14":{"13427":true},
   "11304.15":{"13426":true},
   "11304.16":{"13426":true},
   "11304.17":{"13429":true},
   "11304.18":{"13433":true},
   "11304.22":{"13433":true},
   "11304.23":{"13433":true},
   "11304.24":{"13433":true},
   "11305.19":{"13435":true},
   "11305.20":{"13434":true},
   "age":{"55-64":true},
   "gender":{"f":true},
   "11306":{"13448":true},
   "11307":{"13450":true,"13453":true},
   "11308":{"13459":true,"13460":true,"13461":true,"13466":true,"13469":true},
   "11309":{"13473":true},
   "11999":{"17031":true},
   "12111":{"18235":true}
}

Answer 1

OK, time to try to answer this as the comments section is getting a bit out of control and not really helping us clarify things.

First, I would suggest you put aside all questions of finding cohorts and clusters, and all the machine learning terminology, and buy yourself a text book on basic statistics and surveys and their analysis.

Second you should go back to your actual business problem. I simply don't believe that it is to find clusters of respondents defined by a combination of six variables out of 32. Your real business problem is probably one of:

• "given the various characteristics of a new individual against up to 32 variables, what will be their attitude towards each of my products?" This makes it an individual prediction problem; OR
• "which demographics should I market each of my products to?" This makes it a problem of identifying the most important factors in explaining the NPS score.

In either of those cases, the basic toolkit is going to be a statistical model with attitude to each product as the response variable, and the up to 32 other variables as explanatory variables.

As you have the underlying data that has been summarised in the NPS score, you should use that as your response variable - it has much more information in it that the simple NPS score, and in this situation you want to use all the information you can get.

Unfortunately the strategy for determining the best model is difficult and fraught with conceptual pitfalls for the inexperienced, and the type of model you need is not taught in a basic stats course. I would recommend Harrell's Regression Modeling Strategies as a guide, but you will need to get some more basic stats before you are ready for this. Frankly, you are probably best off getting a statistician to do the work for you.

On the positive side, the numerical and computational problems are all solved and the problems you started with - length of time to the computations etc - will all vanish.

If you really want to go the self learning route, then look for articles on ordinal regression. The free open source stats environment R has an implementation.