Whenever I release a new feature in my social networking website, I split test it against a control group to measure its effectiveness. If the new feature gets more traction, I deem it to have passed the test. Depending on the context, traction can be anything from number of unique clicks to increase in next-day retention as a result of interacting with the feature.
Currently, I'm passing experiments based on absolute values. E.g. if the experiment group's primary metric is absolutely greater than that of control, I pass it.
I now want to evolve to a more scientific approach. Specifically, I only want to pass the experiment if it's lead over the control group is statistically significant. For instance, imagine group A yielded 1000 unique clicks, whereas group B yielded 900. What's the easiest way to calculate whether this difference is statistically significant (assuming the populations are normally distributed). Sorry for the noob question; I'm a beginner in statistics.