How to construct confidence intervals for sum of dependent random variables. Specifically, I want a high probability claim regarding the difference between the empirical mean and the true mean of the sum.
There are two general approaches to construction of CI I'm aware of:
- based on finite sample probability bounds for measure concentration, such as Chernoff and Hoeffding bounds.
- based on Central Limit Theorem arguments, in which case the problem translates to estimating/bounding the variance.
Yet, these hold for independent r.v. (I'm aware of concentration bounds that assume some type of dependency, like Azuma-Hoeffding, but I don't have this type of dependency).
How do I approach the problem for dependent r.v? A reference to a relevant book that discusses the subject would be great.
edit - My specific mean is the Leave-One-Out-CV loss (not the zero-one loss) of a binary classifier. So, there is a dependency between the losses, because of overlapping training sets. I know the variation of this loss around the true loss (i.e. generalization loss) is usually analyzed by stability of the learning alg. I wanted to check if I can bound this deviation without stability using CI.