5
$\begingroup$

Here is an explanation of how NPS is calculated:

http://en.wikipedia.org/wiki/Net_promoter_score

I'm interested in testing two net promoter scores to determine if they are statistically different. I read a great answer to calculating margin of error for NPS (see link below), but I'm really interested in testing to see if there is a difference between two scores, because I suspect that our results aren't as "different" from year to year as they appear to be.

How can I calculate margin of error in a NPS (Net Promoter Score) result?

Is this at all possible? I understand t-tests are typically used to test whether two different record sets are statistically different. But is it possible to test Net Promoter Scores, either with a t-test or some other hypothesis test?

Any ideas you have would be a great help. Thank you!

$\endgroup$
  • 3
    $\begingroup$ I think the answer to your question is embedded in @whuber's excellent answer that you have linked to. He effectively suggests what is sometimes called a Z test, but with samples of above 50 or so this will be the same as a t test. NPS will be approximately normally distributed, and you can use his methods of calculating standard errors. $\endgroup$ – Peter Ellis Mar 2 '12 at 20:13
1
$\begingroup$

Whuber's answer is excellent indeed. Another way, which is more straightforward/simple is to :

  1. Calculate the confidence interval of the promoters and for the detractors. Let us denote a P for promoters and D for detractors

  2. Then calculate : (P + error of p) - (D - error of d) for the highest end of NPS confidence interval and (P - error of p) - (D + error of d) for lowest end of the NPS confidence interval

In other words, to get NPS confidence interval, just add the error of promoters and the error of detractors and NPS +/- sum of both errors will be your confidence interval.

Now after you have the confidence interval of your 1st NPS result and the confidence interval of your 2nd NPS result, compare these two to assess if there is a significant difference.

$\endgroup$
  • $\begingroup$ I appreciate you taking the time to reply Sheriff, but I'm not sure that what you're saying is correct. I don't believe that the error calculated on the proportion of promoters added to the error on the proportion of detractors is equal to the error of NPS (which is essentially what you are saying in your example). Each of the respective proportions is not independent of one another. The proportion of the detractors impacts the proportion of promoters, and vice versa. The proportions aren't independent, and neither are the errors. $\endgroup$ – Charles Naughton Dec 6 '12 at 22:22
  • $\begingroup$ I believe the best answer was provided in the post I referenced above, which calculates the error, which you can use to calculate the upper and lower limits of the confidence intervals for the NPS. Also, confidence intervals aren't enough to determine if two Net Promoter Scores are statistically different. To explain further, the NPS of your sample is essentially an estimate of the NPS of your population. You can use confidence intervals to help you determine how reliable that estimate is. $\endgroup$ – Charles Naughton Dec 6 '12 at 22:22
  • 1
    $\begingroup$ Confidence intervals are not really used to determine the whether two separate estimates are statistically different or not. However, a statistical hypothesis test such as a z-test, can tell you within a certain probability whether or not two NPSs are different. I don't mean to "correct" you, so please don't take offense. I just wanted to make sure people who stumble on this thread are aware of all the considerations. $\endgroup$ – Charles Naughton Dec 6 '12 at 22:22
1
$\begingroup$

Unless there is a huge imbalance resulting in almost no Promoters or no Detractors, a t-test should work fine.

Specifically, the NPS method reduces the data to a set of $-1,0,1$ values (representing "Detractors," "Passives," and "Promoters," respectively). In a given dataset $\mathcal{S}$ of $n$ values let the count of the value $x$ be $n_x.$ The NPS is the mean value,

$$NPS_{\,\mathcal{S}} = \frac{1}{n}\left(n_{-1} + n_0 + n_1\right)$$

and its sample variance is an adjusted mean squared difference

$$s_\mathcal{S}^2 = \frac{1}{n-1}\left(n_{-1}(-1-NPS_\mathcal{S})^2 + n_0(0-NPS_\mathcal{S})^2 + n_1(1-NPS_\mathcal{S})^2\right).$$

As explained at https://stats.stackexchange.com/a/18609/919, the square of the standard error (there referred to as "margin of error") is the sample variance divided by the sample size,

$$\operatorname{se}_\mathcal{S}^2 = \frac{s^2_\mathcal{S}}{n}.$$

Given two such sets of data to compare, say $A$ and $B$, the difference in their NPSes is $NPS_A-NPS_B$ and the squared standard error of that difference is $\operatorname{se}_A^2 + \operatorname{se}_B^2.$ The Student $t$ statistic is the ratio of the difference to its standard error,

$$t = \frac{NPS_A - NPS_B}{\sqrt{\operatorname{se}_A^2 + \operatorname{se}_B^2}}.$$

Because we have assumed a situation where there are some promoters or some detractors, the denominator is nonzero, so $t$ is well-defined. The only issue is how to interpret it.

When the size of $t$ is "large," we say the difference in NPS is "significant" and conclude there is some cause for this difference other than sampling error. The only issue concerns the determination of how large is "large." The Student t-test uses quantiles of a Student t distribution with $n_A-1 + n_B-1$ degrees of freedom to determine what is a "large" value of $t$ for any given level of statistical risk $\alpha$ you care to specify. This risk is the chance that two random samples from populations with equal NPSes will produce a "large" value of $t,$ thereby causing you incorrectly to conclude there's a difference in NPS.

The "critical value," or threshold value to determine what "large" means, is the $1-\alpha/2$ quantile of the appropriate Student $t$ distribution.

Let's work an example. Suppose group $A$ has $n_{-1}=2$ Detractors, $n_0=8$ Passives, and $n_1=10$ Promoters for a total of $n=20.$ Its NPS is $NPS_A = (-2 + 10)/20 = 0.4$ (the same as $40\%$ if you prefer to express values as percents) and its variance is $$s^2_A = (2(-1-0.4)^2 + 8(0-0.4)^2 + 10(1-0.4)^2)/19 = 0.463.$$

Similarly, let group $B$ have $5$ Detractors, $20$ Passives, and $5$ Promoters, for a total of $30.$ The balance of Detractors and Promoters shows $NPS_B$ is zero. Its variance is $s_B^2=0.345.$ Thus, the t statistic for comparing these groups is

$$t = \frac{0.4 - 0} {\sqrt{0.463/20 + 0.345/30}} = 2.15.$$

Its size is $|t|=2.15.$ To determine how large this is, we refer to the Student $t$ distribution with $20-1 + 30-1 = 48$ degrees of freedom. It assigns a chance of $3.7\%$ to a value this large. This is the "p-value" of the t-test. If your risk threshold is only, say, $\alpha=5\%,$ then because the p-value is less than the threshold you will conclude this is a significant difference. If your risk threshold is smaller, say $\alpha=1\%,$ then because the p-value is greater than the threshold you will not conclude the observed difference in the samples is significant evidence of a real difference in the population represented by those samples.

Simulation studies indicate the use of the Student $t$ distribution works well when each group has at least 20 people. It also works when there are huge differences in NPS between the groups, where the conclusion to make is obvious. For smaller groups with similar NPSes or where there are extreme imbalances, you should mistrust the p-value. In such circumstances conduct a permutation test or collect more data.


For greater insight, pay attention to the variances: even when the groups have comparable NPSes and those do not differ "significantly," if one of the groups has a much larger variance you might want to take that polarization of your customers into consideration. For instance, a group of $20$ Passives and another group comprised of $10$ Detractors and $10$ Promoters will have identical NPSes of $0,$ whence a $t$ statistic of $0$ (which is never "significant" for any $\alpha$), yet there is a clear difference in how those groups are reacting to your product. This failure to account for the variance in evaluating customers is, IMHO, the chief drawback of using the NPS.

$\endgroup$
  • $\begingroup$ The likelihood is p(S|p1,p2) = p1^(n-1) p2^n0 (1-p1-p2)^n1 where p1 is the probability of detractors, p2 is the probabilty of passives and (1-p1-p2) is the probability of promoters. Given priors p(p1A,p2A) and p(p1B,p2B), we can compute the posteriors. And from both posteriors we can compute the Bayes-Poincaré factor for hypothesis H0: (p1A,p2A) = (p1B,p2B) p((p1A,p2A) = (p1B,p2B)|SA,SB) / p((p1A,p2A) = (p1B,p2B)), which provides the exact probabilistic answer to the problem. So, everything reduces to assigning p(p1,p2).......... $\endgroup$ – Fabrice Pautot Sep 20 '19 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.