I'd like to use the BCa algorithm to compute 95% confidence limits (maybe let's not call it intervals) for the medians of two population samples. I'd like to visually inspect and conclude whether the intervals overlap to say something about the significance of the difference. btw As far as I know visually comparing the differences like this with confidence intervals is a necessary condition but not sufficient condition to assess whether there is significant difference between the statistic of the two groups i.e. if the confidence intervals don't overlap then they are significantly different; if however, they overlap we need more specific tests to be sure ... I need to find where I learned that before.
Is it ok to state that BCa is assumptions-free and rely on it for inference i.e. generating reliable confidence limits given the following points?
- The two unpaired groups are very departed from a normal distribution i.e. positively skewed.
- The within-group samples were chosen to be the top N 1.5k and not randomly (the independence of the samples assumption required to resort to CLT doesn't hold). 1.5k samples is simply an arbitrarily large N for inference, it could have been 200 or 1k or 2k doesn't matter.
- I can't use the mean but the median due to how the data is distributed ... the median is a distribution-free statistic and standard errors can not be calculated in this case.
UPDATE my use-case is the following I have millions of users, they reach some categories e.g. $C \subset B \subset A$. Meaning all users in C are in B and A and all users in B are in A but not the other way around. Since what I am doing is exploratory analysis and inference I would like to create relevant samples of users within each group in a short time.
The final process should first select all users in A, then excluding the ones in A, find all users in B and so on. This exclusion process is computationally very expensive to do in all the data so I needed a way to ensure exclusivity so that A users don't slip randomly into the B group which would be the case if I simply random sample from A, B and C.
The best way I found to do that is choosing the top N users, because it is guaranteed that the top users in B excluding the top users in A, will not contain top users in A slipping into B randomly.