3
$\begingroup$

Whenever I release a new feature in my social networking website, I split test it against a control group to measure its effectiveness. If the new feature gets more traction, I deem it to have passed the test. Depending on the context, traction can be anything from number of unique clicks to increase in next-day retention as a result of interacting with the feature.

Currently, I'm passing experiments based on absolute values. E.g. if the experiment group's primary metric is absolutely greater than that of control, I pass it.

I now want to evolve to a more scientific approach. Specifically, I only want to pass the experiment if it's lead over the control group is statistically significant. For instance, imagine group A yielded 1000 unique clicks, whereas group B yielded 900. What's the easiest way to calculate whether this difference is statistically significant (assuming the populations are normally distributed). Sorry for the noob question; I'm a beginner in statistics.

$\endgroup$

1 Answer 1

5
$\begingroup$

You should probably use the t-test. You want to compare the central tendency of two continuous variables.

The test measures whether the average (expected) value differs significantly across samples. If we observe a large p-value, for example larger than 0.05 or 0.1, then we cannot reject the null hypothesis of identical average scores. If the p-value is smaller than the threshold, e.g. 1%, 5% or 10%, then we reject the null hypothesis of equal averages.

In your case, based on only what you've mentioned in the question, I don't think the total number of clicks should be the measure for a group. I believe you should take the mean number of clicks for each group (#clicks / #users). Then if there are 10 users in both group A and group B then by t-test you would want to know if the means 100 and 90 could be considered to be significantly different.

The null hypothesis in this case would be that both the means are same and if you get a larger p-values you could reject the null hypothesis and consider that they are significantly different.

You can learn more about t-test here. If you are using python you can probably use scipy's implementation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.