Comparing two crash rates and determine if they are statistically significant Let's say I have two version of an App. Version A goes to 1000 users and Version B goes to 800 users.
A month into testing, I have all of my crash data and crash percentages for both of the versions. Version A has a crash percentage of 2% while Version B has a crash rate 1.8%.
How can I determine the crash rates are statistically significant and if Version B is actually better ?
Should I perform a two tail T test?
 A: Rather than testing for the difference of two means, I would test for the difference of two proportions. This uses a $z$-test, given that Normality holds for your estimates of $\hat{p}$.
A: Presumably crashes can occur repeatedly (and you have the data on how many happened), additionally the rate may depend on how long ago the app was installed (e.g. user 1 only installed it 2 weeks ago, user 2 already 4 weeks ago) and crash rates might differ between user (e.g. different mobile phones, problems with different apps other users have). If so, then my first thought would have been a negative binomial model (or more or less equivalently a Poisson model with a random user effect) that has a log(duration app was installed) offset with a class effect for app version.
By the way, were users randomly assigned to app version? If the assignment was non-random, you might want to account for that (e.g. if the first 1000 users got version A, or if users could pick the version themselves or if users from the US got version A and users from India version B, then there could be systematic differences between users, their phones, behaviors and other apps). If you do not have any information on the users you probably cannot account for it, but need to be aware of this issue. If you have information like country, age, other apps, phone etc., you would likely want to account for that. Probably even in the case of a randomized experiment (e.g. as model covariates or factors), but most definitely when it is not a randomized experiment (e.g. stratification by propensity score).
