Suppose you are trying to determine if 2-door cars sell better than 4-door cars. Note: I turned my actual data into a car analogy in hopes that it will be easier to understand.
Normally, I'd do a two-sample hypothesis test to see for the difference between 2-door and 4-door. That test returns a z-score of
10.825134194943086 and a p-value of
2.6169552470669827e-27. So the conclusion would be, in laymans terms, there is a difference between the two and the 2-door is better since it's higher.
Trials Success Success_Rate Z-score P-value Dimension 2-door 25603 1357 0.0530 4-door 13073 378 0.0289 10.8251 0 Grand Total 38676 1735 0.0449
But let's say I segment my data by more dimensions and attempt to run the same hypothesis tests on a deeper level of dimensions. What I find is 4-door has the higher
Success_Rate in this segmented view, and my hypothesis tests each indicate statistical significance in-favor for 4-door, not 2-door like the above test suggests.
Trials Success Success_Rate Z-score P-value Dimension1 Dimension2 Dimension Car White 2-door 7478 75 0.0100 4-door 2766 75 0.0271 6.391536 1.642280e-10 Black 2-door 7289 1149 0.1576 4-door 429 86 0.2005 2.351559 1.869493e-02 SUV White 2-door 9780 51 0.0052 4-door 9212 124 0.0135 5.944013 2.781270e-09 Black 2-door 1056 82 0.0777 4-door 666 93 0.1396 4.146006 3.383240e-05 Total -- -- 38676 1735 0.0449
Notice that it's still the same data, just segmented.
How do I decide if 2-door or 4-door is better ?