# How much sub-questions impact one ordinal question in a survey?

I have a variable Net Promoter Score (NPS) which is an item on a survey, the answer format is ordinal, 0 to 10.

Then, there are sub-questions on the same survey, some are ordinal, some are categorical.

I would like to determine which sub-question or which group of sub-questions are the best predictors of NPS. The goal is to be able to prioritize which area(s) to focus on in order to improve NPS over time.

What would be some appropriate methods to identify and verify such a relationship?

I love your question, and I'm going to answer it a little more broadly then you asked. There are four main approaches that I would use to identify opportunities to improve Net Promoter Score, in order of analytical sophistication and the maturity of the project you're using NPS to evaluate. (For those not familiar, Net Promoter is a common satisfaction score methodology used by many companies. Unless you tend to avoid surveys, you have probably answered the "net promoter" question many times.)

1. Segmentation
2. Qualitative evaluation of open-ended responses
3. Quantitative evaluation of open-ended responses
4. Experimentation

Segmentation: A business can usually divide its customers into a small number of groups along a couple of key dimensions. For example, between "new" and "returning" customers, or "enterprise" and "small business". These will often closely align with the distinctions made in pricing and in the businesses financial reporting, and, happily, they often have dramatically different NP scores. The differences can sometimes surprise. The great thing about this approach is that it requires very little communications overhead. Most people listening to your data will already be familiar with the segments you describe and can envision the aspects of the product that could drive some groups to be more or less satisfied than others. It also makes it relatively easy to organize followup work and report on subsequent improvements.

Qualitative evaluation of open-ends: A common component of Net Promoter implementations is a followup question along the lines of "why did you answer the way that you did". Often, for a project new to Net Promoter, simply reading these responses and associating them with their ordinal responses will have people slapping their foreheads. "I thought our customers liked that feature!" "People love the rubberized grips - let's put them on everything!" This approach often peters out fairly soon, as a business identifies all the surface-level opportunities and makes strategic decisions to resolve or not resolve them.

Quantitative evaluation of open-ends: Following up on the qualitative evaluation, quantitative evaluation can yield some additional surprises. This approach would generally be to label each open-ended response with one or more categories, which can be done by hand at small scales, or using a variety of software at larger scales. This can reveal additional insights along the lines of "I knew that was an issue... but I didn't realize it affected 30% of our customers!" or "Most people who complain about xyz also complain about abc - maybe there is a common root cause"

Experimentation: It's often difficult to perform rigorous controlled-variable random-assignment experiments with Net Promoter. Because Net Promoter is intended to be a very holistic measure of the relationship between a company and its customers, companies are usually limited in their ability to experiment with the entirety of the relationship, due to the expense and risk of maintaining what might amount to two separate businesses at the same time. That said, other less rigorous forms of experiments are possible.

• Every major decision the business makes can be evaluated for its potential to impact NPS, and then the resulting changes can be subsequently measured. Because it fails to isolate a single variable at a time, care must be taken to build realistic predictions and also to be very cautious about results that were not predicted in advance.
• Sometimes we are lucky enough that natural experimental groups emerge that allow us to experiment with the past. A national retailer may roll out a new store format only on the west coast due to the logistical constraints of construction, or a content company may need to censor some types of searches in countries where the local regulations are different. It's not random assignment, but sometimes it's close.

Prediction: I know that I didn't address your real question about using other questions in surveys to "predict" NPS, and that's intentional. The approach that you describe has certain limitations and although I'm planning to try it again myself in the near future, I've had limited success with it in the past.

There are two main predictor approaches I've seen: correlation, and logit regression. Correlation looks at the strength of the relationship between two variables (do they always go up and down together? opposite? no relation?) The idea would be to find the variables most strongly correlated with NPS. Logit regression is a technique that takes all the variables you give it and uses them to try to predict the probability of a particular outcome. For example, you can try to predict the probability of each of your three NPS outcomes (Promoter, Neutral, Detractor). From this you can build a model that shows how much variation in different variables can affect NPS up or down, taking the NPS outcome with the highest predicted probability as the definitive outcome. I have also heard of people using Structural Equation Modelling with NPS, although I know little of this approach.

Some limitations of the predictor approach

• Few good independent variables in surveys: In order to be actionable, this approach requires that you think of NPS as a dependent variables and treat other questions as independent. But, even many of the "hard facts" type questions, such as you might use for your key customer segments are not truly independent from NPS. A returning customer is a returning customer in part because they had a positive experience in the past. Most customer satisfaction questions move together in lockstep, unless you have an offering with very distinctive and low-interacting components that can be performing at two different levels of service (a leisure hotel may try to break into the conference business and do so very poorly at first, in which case separate satisfaction questions on bedroom quality and conference room quality may be useful).
• Difficulty communicating: This analytical approach is usually quite far separated from the group that will be implementing recommendations. Because it relies on relatively sophisticated analysis, it's often difficult for implementors to take actions based on these sort of recommendations, because they don't understand what needs to be fixed.

The NPS works well as a performance monitoring or controlling tool. But what you want to find out is something else: You want to identify the true drivers of satisfaction.

The NPS is an aggregate score. If you want to say something about drivers of satisfaction, than you have to develop a model which explains the 10-point-scale "ultimate question".

However, rating scales are contaminated with scale usage heterogeneity (see paper). If you run a regression model, you will find that those sub-questions are highly correlated, causing high VIFs and maybe no significant betas, depending on how many sub-questions you have. You might also be overconfident in the significance of sub-questions, depending on the exact model set-up.

Consider this simple R script:

nobs=1000
p=12
means = rnorm(nobs)
vars = 1/rgamma(nobs,3,3)
data=matrix(,nobs,p)
for(i in 1:nobs){
data[i,]=rnorm(p,means[i],sqrt(vars[i]))
}
cor(data)


Despite the fact that there is no relationship between the 12 variables, the correlations are far from being 0.

If you further reduce the scale level (Promotor yes/no), you throw away information. However, scale-adjusting models are very challenging, especially when you have ordinal and categorical variables, and if you look for a ready-made solution.

You could treat bottom, middle and top box as categorical outcomes and use a Bayesian Network to model the relationships. BNs treat everything categorical, thus providing a nice general purpose tool.

Have you considered a conjoint experiment? Depending on the service or product this is a nice alternative.

Some good answers already here. Here is a much simpler one, which is really an addition to the good answers you have received, not a substitute. Ordered logistic regression is the appropriate tool for modelling an ordinal response variable. There is an implementation - polr() - in Benables & Ripley's library(MASS) in R.

Then you are back in the standard world of model selection, on which there are many good questions and good answers on Cross-Validated.

• Right, this probably the most straightforward approach. The categorical variables would need to be dummied up somehow, which is also easy. Feb 25, 2012 at 20:12