Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have a variable Net Promoter Score (NPS) which is an item on a survey, the answer format is ordinal, 0 to 10.
Then, there are sub-questions on the same survey, some are ordinal, some are categorical.
I would like to determine which sub-question or which group of sub-questions are the best predictors of NPS. The goal is to be able to prioritize which area(s) to focus on in order to improve NPS over time.
What would be some appropriate methods to identify and verify such a relationship?
I love your question, and I'm going to answer it a little more broadly then you asked. There are four main approaches that I would use to identify opportunities to improve Net Promoter Score, in order of analytical sophistication and the maturity of the project you're using NPS to evaluate. (For those not familiar, Net Promoter is a common satisfaction score methodology used by many companies. Unless you tend to avoid surveys, you have probably answered the "net promoter" question many times.)
Segmentation: A business can usually divide its customers into a small number of groups along a couple of key dimensions. For example, between "new" and "returning" customers, or "enterprise" and "small business". These will often closely align with the distinctions made in pricing and in the businesses financial reporting, and, happily, they often have dramatically different NP scores. The differences can sometimes surprise. The great thing about this approach is that it requires very little communications overhead. Most people listening to your data will already be familiar with the segments you describe and can envision the aspects of the product that could drive some groups to be more or less satisfied than others. It also makes it relatively easy to organize followup work and report on subsequent improvements.
Qualitative evaluation of open-ends: A common component of Net Promoter implementations is a followup question along the lines of "why did you answer the way that you did". Often, for a project new to Net Promoter, simply reading these responses and associating them with their ordinal responses will have people slapping their foreheads. "I thought our customers liked that feature!" "People love the rubberized grips - let's put them on everything!" This approach often peters out fairly soon, as a business identifies all the surface-level opportunities and makes strategic decisions to resolve or not resolve them.
Quantitative evaluation of open-ends: Following up on the qualitative evaluation, quantitative evaluation can yield some additional surprises. This approach would generally be to label each open-ended response with one or more categories, which can be done by hand at small scales, or using a variety of software at larger scales. This can reveal additional insights along the lines of "I knew that was an issue... but I didn't realize it affected 30% of our customers!" or "Most people who complain about xyz also complain about abc - maybe there is a common root cause"
Experimentation: It's often difficult to perform rigorous controlled-variable random-assignment experiments with Net Promoter. Because Net Promoter is intended to be a very holistic measure of the relationship between a company and its customers, companies are usually limited in their ability to experiment with the entirety of the relationship, due to the expense and risk of maintaining what might amount to two separate businesses at the same time. That said, other less rigorous forms of experiments are possible.
Prediction: I know that I didn't address your real question about using other questions in surveys to "predict" NPS, and that's intentional. The approach that you describe has certain limitations and although I'm planning to try it again myself in the near future, I've had limited success with it in the past.
There are two main predictor approaches I've seen: correlation, and logit regression. Correlation looks at the strength of the relationship between two variables (do they always go up and down together? opposite? no relation?) The idea would be to find the variables most strongly correlated with NPS. Logit regression is a technique that takes all the variables you give it and uses them to try to predict the probability of a particular outcome. For example, you can try to predict the probability of each of your three NPS outcomes (Promoter, Neutral, Detractor). From this you can build a model that shows how much variation in different variables can affect NPS up or down, taking the NPS outcome with the highest predicted probability as the definitive outcome. I have also heard of people using Structural Equation Modelling with NPS, although I know little of this approach.
Some limitations of the predictor approach
The NPS works well as a performance monitoring or controlling tool. But what you want to find out is something else: You want to identify the true drivers of satisfaction.
The NPS is an aggregate score. If you want to say something about drivers of satisfaction, than you have to develop a model which explains the 10-point-scale "ultimate question".
However, rating scales are contaminated with scale usage heterogeneity (see paper). If you run a regression model, you will find that those sub-questions are highly correlated, causing high VIFs and maybe no significant betas, depending on how many sub-questions you have. You might also be overconfident in the significance of sub-questions, depending on the exact model set-up.
Consider this simple R script:
nobs=1000
p=12
means = rnorm(nobs)
vars = 1/rgamma(nobs,3,3)
data=matrix(,nobs,p)
for(i in 1:nobs){
data[i,]=rnorm(p,means[i],sqrt(vars[i]))
}
cor(data)
Despite the fact that there is no relationship between the 12 variables, the correlations are far from being 0.
If you further reduce the scale level (Promotor yes/no), you throw away information. However, scale-adjusting models are very challenging, especially when you have ordinal and categorical variables, and if you look for a ready-made solution.
You could treat bottom, middle and top box as categorical outcomes and use a Bayesian Network to model the relationships. BNs treat everything categorical, thus providing a nice general purpose tool.
Have you considered a conjoint experiment? Depending on the service or product this is a nice alternative.
Some good answers already here. Here is a much simpler one, which is really an addition to the good answers you have received, not a substitute. Ordered logistic regression is the appropriate tool for modelling an ordinal response variable. There is an implementation - polr() - in Benables & Ripley's library(MASS) in R.
Then you are back in the standard world of model selection, on which there are many good questions and good answers on Cross-Validated.
