Determining a Statistically Significant Result

Question

I feel like this is an easy question if I just knew the right words to Google, but apparently I don't, so I'm sorry if it's been asked before.

I'm simply looking to determine how long I need to collect data to be relatively confident in the result.

For example, if I flip a coin 4 times and it comes up heads 3 times, it's a big leap to say the coin is weighted. However, if I flip it a billion times and it comes up heads 75% of the time, I can be pretty confident it is weighted.

At what point between 4 and a billion can I feel confident my results are probably accurate.

My real world use case is I just want to ensure that the base conversion rate I will use to base acceptable advertising costs on is accurate. I know if 4 people visit my site and I sell to 1 of them, I can't be sure of much. But if a billion came, and I sold to 25% of them I'd feel pretty confident in that rate.

I know once you get into A/B testing and multivariate testing there is a lot more you need to know, but I feel like understanding how to determine the initial significance will help me a lot moving forward.

Thanks so much!

Answer 1

The idea you're looking for is Statistical Hypothesis Testing. This is a method of quantifying the amount of evidence you have towards rejecting a 'null' hypothesis that you define (ie. the coin is a fair coin, in this case). This evidence is affirmative evidence towards the opposite of your defined 'null' hypothesis, or the 'alternative' hypothesis. This evidence is usually quantified in the p-value of the statistic you would like to compare (ie. the proportion of heads after multiple coin flips).

Collect as much data as you can reasonably collect. It won't hurt, and the more data you collect, the more evidence towards either the 'null' or 'alternative' hypothesis you will have, and the more confident you can be with your results.