# Two-Sample Hypothesis Test for Proportions Segmented By Dimensions - Conflicting Results

Suppose you are trying to determine if 2-door cars sell better than 4-door cars. Note: I turned my actual data into a car analogy in hopes that it will be easier to understand.

Normally, I'd do a two-sample hypothesis test to see for the difference between 2-door and 4-door. That test returns a z-score of 10.825134194943086 and a p-value of 2.6169552470669827e-27. So the conclusion would be, in laymans terms, there is a difference between the two and the 2-door is better since it's higher.

             Trials  Success  Success_Rate  Z-score P-value
Dimension
2-door        25603     1357        0.0530
4-door        13073      378        0.0289  10.8251       0
Grand Total   38676     1735        0.0449


But let's say I segment my data by more dimensions and attempt to run the same hypothesis tests on a deeper level of dimensions. What I find is 4-door has the higher Success_Rate in this segmented view, and my hypothesis tests each indicate statistical significance in-favor for 4-door, not 2-door like the above test suggests.

                                 Trials  Success  Success_Rate   Z-score       P-value
Dimension1 Dimension2 Dimension
Car        White      2-door       7478       75        0.0100
4-door       2766       75        0.0271  6.391536  1.642280e-10
Black      2-door       7289     1149        0.1576
4-door        429       86        0.2005  2.351559  1.869493e-02
SUV        White      2-door       9780       51        0.0052
4-door       9212      124        0.0135  5.944013  2.781270e-09
Black      2-door       1056       82        0.0777
4-door        666       93        0.1396  4.146006  3.383240e-05
Total       --         --         38676     1735        0.0449


Notice that it's still the same data, just segmented.

How do I decide if 2-door or 4-door is better ?

• This is an example of Simpson's Paradox. en.wikipedia.org/wiki/Simpson%27s_paradox. You might want to consider using a different type of analysis since your data is categorical, unless there is specific some reason you are using a z-score? – Robert Montgomery Aug 10 '16 at 18:33
• I'm so grateful for your answer. I didn't know there was a theory that described this scenario (Simpson's Paradox). I'm using z-score and proportions because I simply don't know what other options of analysis exist for this type of problem. Any alternative recommendations or ideas? – Jarad Aug 10 '16 at 18:53
• I just added an answer that addresses a few of the concerns about this question. I think you should nail down your hypothesis first. But I would use logistic regression to answer this question. CDA by Alan Agresti is an excellent reference, and if you don't have a hard copy it's available online at mathdept.iut.ac.ir/sites/mathdept.iut.ac.ir/files/AGRESTI.PDF – Robert Montgomery Aug 10 '16 at 19:08
• I noticed in another recent question that you were dealing with a similar situation of segmenting your data and noticing the different results you get. I would recommend reading parts (most focus on 2,3,5) of Chapters 1,2,3,5 of Agresti's book. I think it would give you a better framework to address these types of questions. – Robert Montgomery Aug 10 '16 at 19:14
• This question actually doesn't have to do with cars. I just used that analogy in hopes it would be easier for a reader to understand. Ultimately, I'm trying to decide which website landing page URL is better. Dimension1 and Dimension2 could be thought of as Traffic Source and Device (mobile, computer for example). The high-level concept though that I'm trying to figure out is how to statistically determine to stop all traffic to one page in favor of another. That means either "2-door" or "4-door" but as my data shows, the segmented data contradicts the high-level data conclusion. – Jarad Aug 10 '16 at 19:51

I decided to turn my comment into an answer.

What you are experiencing is an example of Simpson's Paradox, where a marginal result can be in the opposite direction of a conditional result. In this case having so much more data on 2-door cars than 4-door cars masks the difference based on color.

Before you can answer the question of "How do I decide if 2-door or 4-door is better?" you need to address the following questions.

1) What is your research hypothesis, are you interested in the difference between 2-door and 4-door in general or do you care about how the relationship changes for specific things like color (or price range, or gas mileage, or leather vs cloth interior, etc..).

Because you could probably always play with your data in such a way that you could find evidence that 2-door or 4-door sold better.

2) Do you really care about the difference between black and white cars specifically, or is this phenomenon something you just happened to notice while messing around with the data.

I would be concerned about this one because you are only looking at two colors. This could be an issue because it's not unreasonable to assume that differences between 2-door and 4-door cars lead people to choose different colors. (i.e. 2-door cars are considered sportier and therefore have a higher proportion of red.) If that is true then when you break down this proportion into 2-door vs 4-door based on red, white, black etc this could change.

Answering these questions will help you decide whether you need more data(maybe more colors included) or maybe you only care about the very general 2-door vs 4-door.

After you've settled on your Hypothesis I would consider using a Categorical technique. If you're interested in describing the data you could start with Odd's Ratios. Personally I would probably include color because you don't want to be accused of fishing for a result. I would try to include more colors than just black and white, and it looks like you've also broken up your data by car type.

So to analyze this I would recommend looking at a logistic regression model. A logistic regression model takes into account the binomial nature (success vs failure) of your data and will allow you to account for the different variables affecting the overall result.

A good reference for everything concerned with Categorical Data is Alan Agresti's Categorical Data Analysis.