1
$\begingroup$

I have data about storage consumption for a lot of users (N > 50k+). The raw data distribution would look something like this:

enter image description here

Given the outliers, to get an estimate of how much storage each user consumes I'd like to build a confidence interval for the median memory consumption (I guess I could build it for the mean if I removed outliers first.. am I correct?).

If I follow my most intuitive understanding of a, say, 95% confidence interval, I would go as follows:

  • bootstrap many samples (e.g. 10_000) of some size (e.g. 1000);
  • compute the median for each sample;
  • take any interval that contains 95% of the samples' medians, by default the equally-tailed one.

As an example, in Python:

medians = [
    df['Total storage used (GB)']
    .sample(n=1000, replace=True)
    .median()
    
    for _ in range(100_000)
]
sorted_medians_srs = pd.Series(medians).sort_values()

alpha = 0.05
left_ci, right_ci = (
    sorted_medians_srs
    # take the extremes
    .iloc[[int(100_000 * alpha/2), -int(100_000 * alpha/2)]]
    .values
)

Which gives a credible CI:

enter image description here

That is, I didn't assume any underlying distribution and I just sampled from the "actual" distribution I have at hand. How incorrect is this approach?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Bootstrapping is a computationally (very) expensive way to approximate the nonparametric CI described in the duplicate. In light of some of your comments here, you might want to ask about a nonparametric prediction limit or possibly a nonparametric tolerance limit (which is a confidence limit for an upper percentile, exactly as given in the duplicate). Prediction limits can be obtained with a similar combinatorial analysis and will be effective with such a large dataset. Hahn & Meeker, Statistical Intervals, describes this and provides tables for small datasets. $\endgroup$
    – whuber ♦
    Commented Oct 14, 2022 at 16:50

2 Answers 2

Reset to default
3
$\begingroup$

Yes, this is an appropriate use of bootstrapping, except that the bootstrap sample size should be the same as the actual sample size (df['Total storage used (GB)'].sample(n=len(df), replace=True)).

However, you always need to ask yourself whether the median is actually the quantity you should care about. For instance, if you're trying to predict future storage needs, and there may be more outliers in the future, median(storage) * n_users will grossly underestimate the actual amount needed.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you! Well, in this case I am more interested in what the majority of users does in order to determine a valid upper bound for the allowed storage. In this case it is more a matter of setting a reasonable limit than being prepared for exceptional needs. Computing the mean with such extreme values felt like it was distorting the reality of things. Makes sense? $\endgroup$
    – rusiano
    Commented Oct 14, 2022 at 16:05
  • 1
    $\begingroup$ I would stick to a prespecified quantile that doesn't have to be the 50th quantile (the median), because you may want to meet the demands not of exactly 50% of your clients (leaving the other 50% unhappy), but of, let's say, 80% or 95% or your clients. $\endgroup$
    – Alex
    Commented Oct 14, 2022 at 16:11
  • $\begingroup$ @Alex So you're saying I should compute a CI for the 80th or 95th quantile? $\endgroup$
    – rusiano
    Commented Oct 14, 2022 at 17:06
  • $\begingroup$ @rusiano, it all depends on what you want to do, what questions you want to answer, what risks you want to account for, what compromises you are willing to make etc. I think your main questions regarding non-parametric estimation of statistical quantities has been thoroughly answered, but most of the statisticians here are are trying to point you to the more important questions. $\endgroup$
    – bdeonovic
    Commented Oct 14, 2022 at 17:31
  • $\begingroup$ You may not need the confidence interval if your practical goal is to find the threshold to limit the storage. The confidence interval is a range estimate of the quantity you want to find. The sample quantile is the point estimate. $\endgroup$
    – Alex
    Commented Oct 14, 2022 at 20:36
2
$\begingroup$

I have doubts the median is the quantity that might be of interest in this case. But more importantly seems to be that calculation of the confidence interval for the true median assumes that that true median isn't going to move in the future. This assumption seems to be too bold, since the storage cost tends to decrease over time, leading to higher and higher storage consumption. Therefore this calculated true median's range is expected to become false pretty soon.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.