I am running an A/B test, and the following are samples of how many questions a user has answered. I am trying to figure out which test is better (A or B), and how confident we are that it is better. For example, in Sample A, first user answered 4 questions, next two users answered 5 questions.

I know how to calculate statistical significance for A/B tests, but am not sure how to do it for a range of numbers. I am trying to get users to answers the maximum questions possible.

Sorry for my incorrect lingo but i'm not a stat guy, so i'm not sure what you refer to these terms. Essentially, I would like to figure out it if my A/B test, A sample or B sample yields a higher number.

Sample A :
4, 5, 5, 9, 11, 14, 15, 15, 16, 19, 27, 30, 31, 32, 58, 65, 67, 79,
98, 99, 100, 103, 106, 204, 232, 341, 354, 359, 360
Med: 58
Avg: 98

Sample B:
1, 3, 4, 4, 4, 5, 9, 11, 12, 15, 19, 28, 37, 48, 50, 54, 59, 72, 74,
78, 80, 81, 89, 91, 99, 101, 103, 103, 104, 120, 121, 174, 203
Med: 59
Avg: 62
  • $\begingroup$ avoiding phrases like "statistical significance percentage" and undefined terms like 'better' (what the heck does better mean here?) - that is, in plain terms - what is it you're trying to do? Imagine you're trying to explain it to an eight year old. $\endgroup$ – Glen_b -Reinstate Monica Aug 24 '13 at 16:10
  • $\begingroup$ What's a 'conversion funnel'? Please clarify what you're trying to achieve when you refer to 'ranges'. Please try to avoid jargon. $\endgroup$ – Glen_b -Reinstate Monica Aug 24 '13 at 16:51
  • $\begingroup$ There is a wikipedia page about conversion funnels: en.wikipedia.org/wiki/Conversion_funnel $\endgroup$ – user25658 Aug 24 '13 at 17:39

I would like to figure out it if my A/B test, A sample or B sample yields a higher number.

This is the sort of question I can answer.

but I see two issues:

1) your data appear ordered from small to large. Is this a feature of the data itself (and then why is it ordered? - is it cumulated??), or did you sort your data for some reason?

2) your responses are effectively counts, so equal-variance assumptions probably wouldn't hold. Nonparametric approaches might have issues with the heavy level of ties. A GLM might be okay.

Edit: Looking at the data on the square root scale:


The data are far too variable to be from a Poisson with constant mean. A negative binomial may be reasonable.

However, I am still quite concerned that there's some facts about this data we should know about but don't, such as something that indicates that the mean is in fact not constant, or that the data are not independent. Here's an index plot for the two samples:

index plot

There's so much here that's left unsaid, still.


just taking a stab, not sure if this is what you want. a simple t-test of these two numeric vectors would indicate that sample A and sample B are not statistically significantly different.

even though the difference in their means is quite large, there's too much variation in your data to detect a significant difference.

a <- c( 4,5,5,9,11,14,15,15,16,19,27,30,31,32,58,65,67,79,98,99,100,103,106,204,232,341,354,359,360 )

b <- c( 1,3,4,4,4,5,9,11,12,15,19,28,37,48,50,54,59,72,74,78,80,81,89,91,99,101,103,103,104,120,121,174,203 )

t.test( a , b )

        Welch Two Sample t-test

data:  a and b
t = 1.5335, df = 37.262, p-value = 0.1336
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -11.63638  84.13377
sample estimates:
mean of x mean of y 
 98.55172  62.30303 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.