# MAE Confidence Intervals: how many bootstraps?

I have predictions and actuals for a dataset. There is a categorical feature in this dataset. Call it state. I want to know the average MAE at the state level with confidence intervals. (Like a Mean MAE, not just MAE of the entire dataset)

I'm assuming the answer involves bootstrapping, but my question is which level(s) to perform the bootstrap on. To lay out the options:

State Level

1. Select a state randomly (with replacement)
2. Calculate MAE from all predictions within that state
3. Repeat many times, use quantiles to estimate CIs

State and Prediction Level

1. Select a state randomly (with replacement)
2. Take a bootstrap sample from within that state
3. Calculate MAE
4. Repeat many times, use quantiles to estimate CIs

Prediction Level

1. Take a bootstrap sample of predictions within each state, exactly once per state
2. Calculate MAE
3. Repeat many times, use quantiles to estimate CIs

Note: The state variable contains all states, the entire population. So can we just do the Prediction level and ignore the state level?

• Looks like a similar question has been asked and answered here, favoring the state and prediction level approach: stats.stackexchange.com/q/1399/389281 Commented Jul 6, 2023 at 21:39

## Answer: Option 4 - Bootstrap all predictions, regardless of state

I attempted to answer this question using simulation:

• 35 groups/states
• Number of samples follows neg binomial with mean ~130
• Predictions and actuals drawn from separate normal distributions for each group
• Parameters for individual group prediction and actual normal distributions themselves drawn from normal distributions
• 100 bootstrap samples used to calculate intervals
• 500 trials of dataset creation and bootstrap estimates run

On this particular simulation, bootstrapping across all predictions was the only method that adequately covered the intervals it was intended to.

I don't have good intuition at this point why the others overestimate/underestimate.

Full code for simulation: https://github.com/robert-robison/Bootstrap-Simulations/blob/main/bootstrap_simulations.ipynb