1
$\begingroup$

Let's say I have two version of an App. Version A goes to 1000 users and Version B goes to 800 users.

A month into testing, I have all of my crash data and crash percentages for both of the versions. Version A has a crash percentage of 2% while Version B has a crash rate 1.8%.

How can I determine the crash rates are statistically significant and if Version B is actually better ?

Should I perform a two tail T test?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Rather than testing for the difference of two means, I would test for the difference of two proportions. This uses a $z$-test, given that Normality holds for your estimates of $\hat{p}$.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks Matt. What if Normality doesn't hold in this case $\endgroup$
    – user82383
    Commented Feb 29, 2016 at 18:49
  • $\begingroup$ $\hat{p}$ is approximately Normal if $n\hat{p}$ and $n(1-\hat{p})$ are at least 15. (This is a guideline which will vary depending on what source you use; I'm relying on my memory of teaching the course here and can't remember which source had the rule of 15.) Thus, you should roughly be okay here, although the crash rate of B doesn't quite meet that threshold. However, if you would feel more comfortable doing so, you could execute a non-parametric test. It would have higher power if Normality doesn't hold. This is a fringe case, so I honestly think you'd be okay either way. $\endgroup$
    – Matt Brems
    Commented Feb 29, 2016 at 18:59
  • $\begingroup$ Thanks again Matt. What if Version A and B goes to the same group of users but at different time frames. For example. Joe was using A and then gets upgraded to B. Should I be performing a One Tail T test? I am trying to rationalize if crash rate % is the right metric. Perhaps Mean Time to Failure is a better metric Or % crash per device. $\endgroup$
    – user82383
    Commented Feb 29, 2016 at 21:59
  • $\begingroup$ Stepping back a bit, what question are you trying to answer? Perhaps there are different metrics that are better-suited to answer your question. First and foremost, as in all statistical work, you should clearly establish what your question is. Only after you know what question you want to ask should you start going into statistical details - one-tailed versus two-tailed tests, etc. You can always tailor your question later to the data you have and the methods that are available but you should have your question clearly defined before beginning any statistical analysis. $\endgroup$
    – Matt Brems
    Commented Mar 1, 2016 at 0:52
  • $\begingroup$ Right now it seems like you're interested in a difference in the performance of A and B, but that's it. Can you provide anything more specific? $\endgroup$
    – Matt Brems
    Commented Mar 1, 2016 at 0:53
1
$\begingroup$

Presumably crashes can occur repeatedly (and you have the data on how many happened), additionally the rate may depend on how long ago the app was installed (e.g. user 1 only installed it 2 weeks ago, user 2 already 4 weeks ago) and crash rates might differ between user (e.g. different mobile phones, problems with different apps other users have). If so, then my first thought would have been a negative binomial model (or more or less equivalently a Poisson model with a random user effect) that has a log(duration app was installed) offset with a class effect for app version.

By the way, were users randomly assigned to app version? If the assignment was non-random, you might want to account for that (e.g. if the first 1000 users got version A, or if users could pick the version themselves or if users from the US got version A and users from India version B, then there could be systematic differences between users, their phones, behaviors and other apps). If you do not have any information on the users you probably cannot account for it, but need to be aware of this issue. If you have information like country, age, other apps, phone etc., you would likely want to account for that. Probably even in the case of a randomized experiment (e.g. as model covariates or factors), but most definitely when it is not a randomized experiment (e.g. stratification by propensity score).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.