Analyzing the difference between two datasets where one is a subset of the other I apologize in advance for the vague title, but I couldn't think of anything better.
I have two datasets, where one is a very small subset of the other. The percentage of people who have a specific attribute in the large dataset is x%. The percentage of people who have the same attribute in the subset is y%. The subset contains people most likely to do some action X.
For instance: I have a list of people who have bought my software. I have a list of people who have renewed. The attribute % for the bought is 20, for the renewed is 60%.
The buyer list size is say 100,000, the people who have renewed is 500.
My questions are:


*

*Is the likelihood of that attribute affecting the user's behaviour simply 3x?

*I know correlation is not causation, but at what point does this change? I.e, in the above case, there's a 3x difference, is that statistically significant enough?

*Does the huge difference between buyers/renewed introduce some sort of bias into the dataset tha makes the attribute percentage useless as a form of determining people who will renew?


Thank you.
 A: I interpret "has attribute" as "the value for this attribute is not missing". As you already pointed out, the question now is whether the value "missing" has a importance regarding the target "renewed" (to add another (fictional, but plausible) example: missing entries for attribute "income" in a customer database may occur with a higher probability when the corresponding customers have high income). 
Let's say $q$ the missing ratio and $p=1-q$ the not-missing ratio.


*

*No, not plainly 3. This is because the other value of the attribute may have some influence on the outcome ("renewed"), too. To measure the importance one can e.g. use InformationGainRatio. I recommend to perform a discretization first (if the attribute we are talking about is numerical) and (most important) treat the value "missing" as an additional value, don't just ignore it. Now you can calculate the InformationGainRatio for the whole set and the subset to see the difference in importance. Another option is do apply a Logistic Regression and check the coefficient of the attribute (but I do not have that much experience in this area). 

*See the answer of Freya Harrison. You can perform a statistical test using the normal distribution approximation to the binomial distribution with the Null-Hypothesis $H_0:=p<=0.2$ and Alternative-Hypothesis $H_1:= p>0.2$. The result will tell you whether the difference is statistical significant.

*I don't think so, because a set of 500 is not small (beside: it is not necessarily true that the whole set would have less information if you restrict it to a sample of 20000). There is still some variance, but useless is the attribute (given that the difference is, even if not significant, quite big) definitively not. In such cases a good old repetitive crossvalidation tells us (I'd choose k around 5 in your case) whether one can perform a prediction based on the attribute or not.

A: For question 2 (significance), could you simply use a randomised resampling procedure and calculate the percentage of subsets of n=500 where the attribute % is ≥60?
