# How does one create a confidence interval for the ratio of the means of two non-normal bounded distributions?

Suppose $X$ and $Y$ are known to take values in the interval $[a,b]$ and $[c,d]$ respectively, but that beyond that nothing is known about their distributions. If $(X_i, Y_i)_{i = 1}^n$ is an i.i.d. sequence drawn from $X \times Y$, what is the best confidence interval for $\bar{X} / \bar{Y}$, containing the estimate $\sum X_i / \sum Y_i$?

I am interested more in conservative confidence intervals (in the sense that we know for sure that they are right) rather than practical advice for approximating confidence intervals (although, indeed, that might be what I end up doing). Conservative confidence intervals that converge are possible, as I outline below. However, as I describe, the simplistic approach I have taken makes confidence intervals that are too wide to be useful.

To give some more context, I have a situation in which I want to estimate the fraction of a population that has some property, P. Unfortunately, the data comes grouped in pairs $(X_i, Y_i)$, where $Y_i$ is the size of a sub-population and $X_i$ is the number from that sub-population that has P. $\sum X_i / \sum Y_i$ will converge to the fraction that I am interested in, given enough samples. Let's assume that $X_i$ is i.i.d. and that $Y_i$ is as well. Neither $X_i$ nor $Y_i$ is normally distributed. However, I can place bounds on each of them. Beyond that, I don't know much about either distribution. Obviously, $X_i \leq Y_i$, so $X_i$ and $Y_i$ are not independent. I also suspect that $X_i/Y_i$ and $Y_i$ are not independent. So in particular, $\frac{1}{n}\sum X_i/Y_i$ is not a good estimate of the ratio we are interested in.

To give a concrete problem that has the same issues, imagine that the $Y_i$ were the sizes of US counties and the $X_i$ were the number of people from the county voting for the Democrat in the last election (this is not my actual use case). Then $X_i/Y_i$ and $Y_i$ are not independent (large cities, for example, tend to vote Democrat) and we are interested in the quantity $\sum X_i/\sum Y_i$ (the fraction of people voting democrat).

It is possible to create a conservative confidence interval, but it seems too conservative. Suppose I want a confidence level of $\gamma$. Using the bounds, I can create confidence intervals for $\bar{X}$ and $\bar{Y}$ at the $1/2 + \gamma/2$ confidence level. I then know with confidence $\gamma$ that both $\bar{X}$ and $\bar{Y}$ are in their respective intervals. You can then compute the smallest and largest possible ratios when drawing numbers from these intervals. The interval from this smallest to this largest ratio serves as a level $\gamma$ confidence interval.

The problem with this approach is that for the number of samples I currrently have (around 60) this approach gives the entire interval from [0,1] as a confidence interval. Part of the problem is that both $X_i$ and $Y_i$ have occasional very large values. So the mean is very different from the bounds on the variable, so you need large sample size. However, that can't be the whole story, since surely the data that I have gathered so far has produced SOME information, so $[0,1]$ for a confidence interval seems too conservative.

I have seen a suggestion to use Feiller's theorem in the case where $X_i$ and $Y_i$ are normally distributed, but they are not. Of course, by the central limit theorem, their sampling distributions will be well approximated by normal distributions. I wonder if I could somehow use that fact.

• Are you aware of this paper that explains the difficulty with this setup? – Xi'an Jan 16 '15 at 17:08
• Have you done a plot of data to examine your suspicion "that $X_i/Y_i$ and $Y_i$ are not independent"? If they are independent, this seems to be a fairly straightforward analysis of a binomial process; each of your observations can be thought of as providing a fraction between 0 and 1. If they are not independent, then it would help to know more about the nature of the sub-populations, in particular why they differ in size and whether a member of the overall population may be a member of more than one sub-population. – EdM Jan 16 '15 at 17:28
• @EdM : I could get into the specific details of the proportion I am trying to estimate, and I would not be surprised if they would lead to improvements in the confidence intervals. However, I suspect that progress on that front would come from modeling the distribution on $X_i$ and $Y_i$ based on specific domain knowledge. I am mostly interested in knowing if there are good ideas that don't require knowing/guessing/assuming much about the data. – Josh Brown Kramer Jan 16 '15 at 19:17
• @Xi'an : The paper seems relevant, but then I feel like I have outlined a process for getting a finite length confidence interval, so I'm not sure where the disconnect it. – Josh Brown Kramer Jan 16 '15 at 19:33

What it seems like you're really trying to do is:

1. Prove that there is a linear relationship between $X_i$ and $Y_i$
2. Determine the constant proportion relating the two with some degree of certainty.

I think the separation is important because if your assumption "$X_i/Y_i$ and $Y_i$ are not independent" was meant to mean you believe the ratio of the two is truly dependent on $Y_i$, then this is equivalent to saying that there is NOT a linear relationship between $X$ and $Y$. This also means that the target quantity $\sum{X_i} / \sum{Y_i}$ may not actually represent a quantity that exists in the system being sampled.

In other words, if you can't prove #1 above then there's really no sense in trying to tackle #2 since the estimated quantity might be a poor reflection of what's going on.

Hopefully this example taken from your comment will help:

Let's say that $X_i$ is the number of democrats in a US county with population $Y_i$ and that the $X$ and $Y$ values are not independent. Then the relationship could be nice and straightforward like this, where the estimate you seem to be after, $\hat{p} = \frac{1}{n}\sum_{i=1}^n{X_i/Y_i}$, would be pretty accurate for most countries and would increase slightly with the overall population:

## Scenario 1

Slight linear relationship between $X_i/Y_i$ and $Y_i$

Above, a confidence interval for $\hat{p}$ would be useful since it could be used to predict $Y$ for a given $X$ pretty reliably since there is not a strong linear relationship between $X_i/Y_i$ and $Y_i$. On the other hand, in a scenario like the one below, the confidence interval for $\hat{p}$ is really kind of useless and since it spans such a large number of the possible proportions:

## Scenario 2

Strong linear relationship between $X_i/Y_i$ and $Y_i$

My point then is that if you think $X_i/Y_i$ increases or decreases with $Y_i$, then I would be very surprised if there is any theoretical bound for the confidence interval given that it's pretty straightforward to come up with ways that interval can span nearly all of [0, 1].

Overall, I think it would be smart to take a closer look at how the fraction $X_i/Y_i$ changes with $Y_i$ and if you can't convince yourself that the dependence between the two is small (possibly with a linear regression of $X$ to $Y$ and examination of the residuals), then I don't see how the estimate you're pulling now will be representative of much.

OR, you could ignore all of this and jump straight to computing bootstrap confidence intervals for the statistic $\hat{p} = \frac{1}{n}\sum_{i=1}^n{X_i/Y_i}$, but again, if there is a strong dependence between this $\hat{p}$ and $Y_i$, any inference made with this estimate may not be very accurate.

• Imagine that the $Y_i$ were the sizes of US counties and the $X_i$ were the number of people from the county voting for the Democrat in the last election (this is not my actual use case). Then $X_i / Y_i$ and $Y_i$ are not independent (large cities, for example, tend to vote Democrat), but the quantity $\sum X_i / \sum Y_i$ certainly exists and is extremely important. And there is, BTW, a strong correlation between $X_i$ and $Y_i$ in this case as well. So I feel like the reasoning in the paragraph following your enumeration of 1. and 2. isn't right. – Josh Brown Kramer Jan 16 '15 at 20:03
• By the way, I like the idea of using bootstrap confidence intervals. They don't quite have the mathematical correctness that I am looking for. On the other hand, I think they will be useful for convincing people that we need significantly more data before our estimates start to converge. – Josh Brown Kramer Jan 16 '15 at 22:01
• Ah, I see what you mean @JoshBrownKramer -- the $\sum{X_i}/\sum{Y_i}$ certainly will exist. What I meant to say is that the effect it is meant to estimate may not. I tried incorporating your example into another take at an explanation so I hope that helps. – Eric Czech Jan 17 '15 at 14:57

For your application, where what you care about is ultimately the probability that an individual has property P, it seems that one of two simpler types of analysis should suffice. These analyses do not depend on distributions of the $X_i$ or $Y_i$, but simply whether an individual has property P and, possibly, to which sub-population it belongs.

If all of your sampled sub-populations are independent random samples of the population as a whole, then what you describe is simply a Bernouilli trial, like flipping a biased coin many times. Your estimate of the overall proportion is simply the ratio of all $X$ cases to the total number of cases observed. This page on Bernouilli Confidence Intervals describes how to proceed for confidence intervals.

You fear, however, that the probability of having property P may differ among sub-populations depending on some characteristic of the specific sub-populations, such as their size. Then you need to know which sub-population an individual belongs to in order to estimate the individual's chance of having property P.

This latter possibility seems like a good candidate for logistic regression, where you examine how the probability of an individual having property P relates to one or more covariates associated with the individual. I don't know of a reason why the size of the sub-population containing an individual can't be such a covariate. Statistical programs that perform logistic regressions will provide confidence intervals for odds ratios, which can be translated back to the probability scale. The model you use to set up the regression should ideally be based on domain knowledge. If you instead use your data set to set up the model (like deciding whether you examine a relation to the absolute size versus the log of the size of the sub-population), your confidence intervals will be too optimistic.

• I don't yet know if $X_i/Y_i$ and $Y_i$ are independent, but there are strong theoretical reasons to believe that they are not. I think logistic regression idea is a good one. However, there are still some difficult to answer questions : how much data do I need before the logistic regression is good? Is $X_i$ a binomial distribution with $p$ determined by logistic regression and $n = Y_i$? We also have to model the distribution on $Y_i$. All of that has value, but I know it is possible to create confidence intervals with much weaker assumptions about the data. – Josh Brown Kramer Jan 16 '15 at 21:44
• See stats.stackexchange.com/questions/11724/… for power considerations in logistic regression. In general, the fewer the assumptions you make and can validate about underlying distributions, the wider the confidence intervals will be. Much depends here on the particular purposes for which you will use the confidence intervals. – EdM Jan 19 '15 at 15:24