Determining the significance of a rule

Question

Let's say I establish the following rules from a data set:

Rule A: If you exercise daily, you have a 70% chance of having a BMI under 28, based on 100 cases from the data.

Rule B: If you eat fast food less than once per week, you have a 90% chance of having a BMI under 28, based on 10 cases from the data.

The challenge for me is that Rule B seems to show a stronger correlation between a factor and a BMI under 28, but it is observed in much fewer cases than in Rule A, and therefore has a higher variance. How can I mathematically determine which rule is more significant?

Answer 1

If you want to know which effect is stronger, then it makes sense to just look at your point estimates and choose case b.

However, your instinct, that there is considerable uncertainty in rule b, is right. So, it is good practice to also consider the uncertainty in these estimates. One common way of doing so is to look at the confidence intervals. So in case a you have 70 successes out of 100 tries, leading to a 95% confidence interval [60.0, 78.8], while in case b you have 9 successes out of 10 tries leading to a 95% confidence interval of [55.5, 99.7].

These confidence intervals are based on the binomial distribution. Alternatives exist.

Answer 2

You can look at these rules as two proportions (namely the proportion of people with a BMI under 28 in two different groups). Then you can test if those two proportions are significantly different. The fact that the group sizes are different is not necessarily a big problem, the test can take care of that.

On the other hand, having a very small number of cases means that there is a lot of uncertainty in your estimates. Consequently, it will be more difficult to establish that the rule holds in the population your sample comes from but it does not mean that the relationship is smaller. Conversely, if you had a very large sample you would know these proportions precisely but it could very well be that they are very similar. The main thing to understand is that the strength of the relationships (represented by the proportion in your case) is distinct from the uncertainty about them that results from the limited sample size.

There are two unrelated problems in the way the rules are presented though:

• The thresholds seem arbitrary. What about people who exercise twice weekly? What a BMI of 30 or 25? Presumably there is some sort of quantitative relationship between exercise and BMI that is not fully captured by arbitrary thresholds.
• There will be some overlap between people who exercise and people who eat fast food and those two variables are also likely to be correlated.

For all these reasons, it's probably more fruitful to use the original data to build some model of the relationship between exercise, diet and BMI rather than reduce it to simple rules.