# Analyzing the difference between two datasets where one is a subset of the other

I apologize in advance for the vague title, but I couldn't think of anything better.

I have two datasets, where one is a very small subset of the other. The percentage of people who have a specific attribute in the large dataset is x%. The percentage of people who have the same attribute in the subset is y%. The subset contains people most likely to do some action X.

For instance: I have a list of people who have bought my software. I have a list of people who have renewed. The attribute % for the bought is 20, for the renewed is 60%.

The buyer list size is say 100,000, the people who have renewed is 500.

My questions are:

1. Is the likelihood of that attribute affecting the user's behaviour simply 3x?
2. I know correlation is not causation, but at what point does this change? I.e, in the above case, there's a 3x difference, is that statistically significant enough?
3. Does the huge difference between buyers/renewed introduce some sort of bias into the dataset tha makes the attribute percentage useless as a form of determining people who will renew?

Thank you.

-

Make a table:

              No renewal Renewal  Total
---------- ------- ------
Attribute  No      79800     200  80000
Yes      19700     300  20000
---------------------------------------
Total              99500     500 100000


The computations are:

• Number of non-renewers = 100,000 - 500 = 99,500.

• "Attribute" is 20% of all buyers = 0.20 * 100,000 = 20,000.

• Non-"attribute" is therefore 100,000 - 20,000 - 80,000 of all buyers.

• "Attribute" is 60% of all renewers = 0.60 * 500 = 300.

• Therefore, 20,000 - 300 = 19,700 of all non-renewers have the attribute.

• Non-"attribute" for renewers is 500 - 300 = 200.

• Therefore, 80,000 - 200 = 79,800 of all non-renewers do not have the attribute.

Conduct a chi-square test of independence. This is valid because each of the cells in your table has a large count. ("Large" typically means 5 or greater.) Here are the calculations:

• 80% of all buyers are "no attribute" and 99.5% do not renew. Therefore we expect .80 * .995 * 100,000 = 79,600 to be non-renewers with "no attribute".

• Similarly, the rest of the table of expected values is

               No renewal  Renewal
---------- -------
Attribute  No      79600     400
Yes      19900     100

• The residuals (differences between the tables) are 79800 - 79600 = 200, 200 - 400 = -200, 19700 - 19900 = -200, and 300 - 100 = 200.

• The chi-square terms are the squared residuals divided by the expectations. Specifically, these equal $200^2/79600 = 0.5025$, $(-200)^2/400 = 100$, $(-200)^2/19900 = 2.0101$, and $200^2/100 = 400$. The chi-square statistic is their sum, 0.5025 + 100 + 2.0101 + 400 = 502.5126.

• There is only one degree of freedom: given the marginal percentages (80% and 20% for rows, 99.5% and 0.5% for columns), the entire table is determined by the number of buyers (100,000). That's a single value, so there's one DoF.

• The chance of a chi-square variate exceeding 502.5126 is astronomically small (less than 2.3E-111). It's best to use good statistical software to compute this value, but even Excel's calculation =CHIDIST(1, 502.5126) is close enough.

Now we can answer the questions.

1. Your data say nothing about the "attribute" affecting behavior. This is an observational study. All it finds is that there is an association between the "attribute" and renewal and (thanks to the chi-square calculation) we cannot attribute that association to randomness.

2. If 55.65% of all buyers had the "attribute" and 60% of all renewers continued to have the attribute, the chi-square test would still be (barely) significant at the 5% level.

3. It is not clear what you mean by "huge difference." Is it the proportion having the "attribute"? Their actual numbers? And what do you mean by "bias"? That would depend on how you plan to use these results. If you want to predict future renewal rates, there are all kinds of potential biases resulting from ways in which these particular 100,000 customers could differ from future customers. But that question is not settled with a statistical analysis: it's really a matter of (marketing) faith. What the data say is that renewal is convincingly associated with the "attribute" within this particular population of 100,000 people.

-
Thank you for the great answer! I do have a couple of questions: is it correct to assume that "chance of chi-square variate exceeding X" is essentially the chance that a random selection between 1-2.3E-111 would produce the number? or? And, is there any software that I can use to quickly calculate these metrics, or is doing it by hand the best option? Thank you! – Sam Thropton Jan 6 '11 at 10:56
@Sam The chi-square statistic is a measure of association of the two variables ("attribute" and renewing). When there is no association, the chi-square statistic should act like the 500 renewers occurred at random in the population of buyers, regardless of the values of "attribute." In this model, the number 3E-111 estimates the chance that chi-square could be as large as it turned out to be. You can do all the calculations in Excel, although stats software like R (free), Stata, SAS, JMP, Mystat (free), etc., is easier and more reliable. – whuber Jan 6 '11 at 15:48

I interpret "has attribute" as "the value for this attribute is not missing". As you already pointed out, the question now is whether the value "missing" has a importance regarding the target "renewed" (to add another (fictional, but plausible) example: missing entries for attribute "income" in a customer database may occur with a higher probability when the corresponding customers have high income).

Let's say $q$ the missing ratio and $p=1-q$ the not-missing ratio.

1. No, not plainly 3. This is because the other value of the attribute may have some influence on the outcome ("renewed"), too. To measure the importance one can e.g. use InformationGainRatio. I recommend to perform a discretization first (if the attribute we are talking about is numerical) and (most important) treat the value "missing" as an additional value, don't just ignore it. Now you can calculate the InformationGainRatio for the whole set and the subset to see the difference in importance. Another option is do apply a Logistic Regression and check the coefficient of the attribute (but I do not have that much experience in this area).
2. See the answer of Freya Harrison. You can perform a statistical test using the normal distribution approximation to the binomial distribution with the Null-Hypothesis $H_0:=p<=0.2$ and Alternative-Hypothesis $H_1:= p>0.2$. The result will tell you whether the difference is statistical significant.
3. I don't think so, because a set of 500 is not small (beside: it is not necessarily true that the whole set would have less information if you restrict it to a sample of 20000). There is still some variance, but useless is the attribute (given that the difference is, even if not significant, quite big) definitively not. In such cases a good old repetitive crossvalidation tells us (I'd choose k around 5 in your case) whether one can perform a prediction based on the attribute or not.
-

For question 2 (significance), could you simply use a randomised resampling procedure and calculate the percentage of subsets of n=500 where the attribute % is ≥60?

-
Depending on how this suggestion is interpreted, it may amount to either Fisher's Exact Test or Barnard's Test. See en.wikipedia.org/wiki/Fisher%27s_exact_test . – whuber Jan 5 '11 at 20:06