0
$\begingroup$

What is the most appropriate test to find if the differences in conversion rates are statistically significant in the case I'm comparing multiple different groups? e.g.

I have different groups of ages (teenager, young, adult, elderly) and a conversion rate related to each of them in my website. How can I validate the differences in the conversion rates from a statistical point of view?

class      size    conversion_rate (=num_conversions/size)   
teenager   1000        3%
young       700       3.3%
adult       800       4.7%
elderly     300       2.3%

The ultimate question I'd like to answer is if the trend (e.g., grows until adulthood, then drops) can be attributed to chance or not.

$\endgroup$
8
  • 1
    $\begingroup$ Both "validate" and "significance" have technical meanings in statistics that differ from the ordinary meaning of those words. Try rewriting the question in plain english so it is clear what you want to find out. $\endgroup$ Oct 19, 2022 at 14:20
  • 1
    $\begingroup$ For example, do you want to know if there are any differences in conversion rates among your age groups? Are there some specific age groups that you would like to compare? Do you want to perform all 6 pairwise comparisons that are possible with 4 age groups? Please edit the question to clarify what you seek, as comments are easy to overlook and can be deleted. $\endgroup$
    – EdM
    Oct 19, 2022 at 16:47
  • $\begingroup$ Thanks for your comments, I tried to clarify the question $\endgroup$
    – simon
    Oct 19, 2022 at 19:00
  • $\begingroup$ Could you define what is ‘conversion rate’ and it’s measured? $\endgroup$
    – utobi
    Oct 19, 2022 at 19:05
  • $\begingroup$ Is your hypothesis about the trend something you had considered prior to the test, or is this only coming about after seeing the trend? $\endgroup$ Oct 19, 2022 at 19:57

2 Answers 2

1
$\begingroup$

If you want to get one numerical value for the statistical significance of the difference between all groups (rejecting the null hypothesis that all proportions are identical), you might want to take a look at the most trivial $\chi^2$ test for the contingency table:

converted not converted
teenager 30 970
young 23 677
adult 38 762
elderly 7 293
$\endgroup$
0
$\begingroup$

Alex (+1) has the fundamental answer: if you have categorical data the $\chi^2$ test provides a simple test of overall significance. For your data I get a p-value of 0.12, meaning that in 12% of experiments like this you would get at least this large an apparent deviation from independence even if there were no differences among age groups in conversion rates.

Is that far enough from "chance" for your application? It doesn't pass the usual "significance" standard of p < 0.05, but there's nothing magical about that cutoff.

You might do better if you had individual ages corresponding with individual yes/no conversions. Your categorization of a continuous variable like age is generally unwise; it implies unrealistic sudden breaks in behavior at specific ages. Modeling age smoothly as a continuous predctor, for example with a regression spline, could be done in a binomial regression model (e.g., logistic regression). That could be more informative, allowing you to plot estimated conversion rates as a smooth function of age.

If you are stuck with age categories, note that binomial regression will allow you to do analysis among specific categories; the $\chi^2$ test only looks at overall deviations from independence among all categories. For your data, expressed as Alex suggests, you could do the following in R:

glm(cbind(converted,notConverted)~class,data=yourData,family=binomial(link="logit"))

If you pursue this type of work further, look into binomial regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.