Is it correct to say that the nonparametric BCa (bootstrap confidence intervals) is assumptions-free?

Question

I'd like to use the BCa algorithm to compute 95% confidence limits (maybe let's not call it intervals) for the medians of two population samples. I'd like to visually inspect and conclude whether the intervals overlap to say something about the significance of the difference. btw As far as I know visually comparing the differences like this with confidence intervals is a necessary condition but not sufficient condition to assess whether there is significant difference between the statistic of the two groups i.e. if the confidence intervals don't overlap then they are significantly different; if however, they overlap we need more specific tests to be sure ... I need to find where I learned that before.

Is it ok to state that BCa is assumptions-free and rely on it for inference i.e. generating reliable confidence limits given the following points?

1. The two unpaired groups are very departed from a normal distribution i.e. positively skewed.
2. The within-group samples were chosen to be the top N 1.5k and not randomly (the independence of the samples assumption required to resort to CLT doesn't hold). 1.5k samples is simply an arbitrarily large N for inference, it could have been 200 or 1k or 2k doesn't matter.
3. I can't use the mean but the median due to how the data is distributed ... the median is a distribution-free statistic and standard errors can not be calculated in this case.

UPDATE my use-case is the following I have millions of users, they reach some categories e.g. $C \subset B \subset A$. Meaning all users in C are in B and A and all users in B are in A but not the other way around. Since what I am doing is exploratory analysis and inference I would like to create relevant samples of users within each group in a short time.

The final process should first select all users in A, then excluding the ones in A, find all users in B and so on. This exclusion process is computationally very expensive to do in all the data so I needed a way to ensure exclusivity so that A users don't slip randomly into the B group which would be the case if I simply random sample from A, B and C.

The best way I found to do that is choosing the top N users, because it is guaranteed that the top users in B excluding the top users in A, will not contain top users in A slipping into B randomly.

Answer 1

As @whuber notes in a comment, you do need to deal first with what seem to be incorrect premises in your approach. Most important, if your samples aren't truly random then "you can't use any of these methods to make inferences," as he put it. Fix that first.

In terms of mean versus median as a measure of central tendency, the choice is yours based on your understanding of the subject matter unless your data are from a distribution like the Cauchy distribution, which has a median but not a mean. Skewness per se has nothing to do with it; the Cauchy distribution is symmetric about its median.

As @carlo put it in a comment, non-overlap of 95% confidence intervals is much too stringent a test for a difference between two values at p < 0.05. See this classic answer from @whuber for details in the case of t-tests. The associated suggestion to use bootstrap directly to test your hypotheses about differences is worth pursuit.

This answer goes into detail about several ways to use bootstrapping to estimate confidence intervals. You clearly can compute BCa confidence intervals for any data set regardless of underlying distribution, but the question is whether 95% confidence intervals determined that way represent true 95% confidence intervals. BCa is designed to deal with bias and skew, and might perform the best of common ways to estimate confidence intervals via bootstrapping, but you can't necessarily be sure.

The problem is that confidence intervals are based on a null hypothesis: there is no difference in the value of the parameter of interest between the two populations. In contrast, the data from which you are sampling might actually represent two different populations with respect to that parameter. For bootstrapping to provide reliable confidence intervals in that case you should be evaluating a parameter that is pivotal. As @AdamO has put it: "This means that if the underlying parameter changes, the shape of the distribution is only shifted by a constant, and the scale does not necessarily change. This is a strong assumption!" In that sense, the BCa confidence intervals are not assumption-free. It might be impossible in some cases to get reliable bootstrapped estimates of confidence intervals, as when sampling from a lognormal distribution without transformation. The answer linked at the beginning of the previous paragraph provides several references on these matters.