2
$\begingroup$

I have predictions and actuals for a dataset. There is a categorical feature in this dataset. Call it state. I want to know the average MAE at the state level with confidence intervals. (Like a Mean MAE, not just MAE of the entire dataset)

I'm assuming the answer involves bootstrapping, but my question is which level(s) to perform the bootstrap on. To lay out the options:

State Level

  1. Select a state randomly (with replacement)
  2. Calculate MAE from all predictions within that state
  3. Repeat many times, use quantiles to estimate CIs

State and Prediction Level

  1. Select a state randomly (with replacement)
  2. Take a bootstrap sample from within that state
  3. Calculate MAE
  4. Repeat many times, use quantiles to estimate CIs

Prediction Level

  1. Take a bootstrap sample of predictions within each state, exactly once per state
  2. Calculate MAE
  3. Repeat many times, use quantiles to estimate CIs

Note: The state variable contains all states, the entire population. So can we just do the Prediction level and ignore the state level?

$\endgroup$
1

1 Answer 1

0
$\begingroup$

Answer: Option 4 - Bootstrap all predictions, regardless of state

I attempted to answer this question using simulation:

  • 35 groups/states
  • Number of samples follows neg binomial with mean ~130
  • Predictions and actuals drawn from separate normal distributions for each group
  • Parameters for individual group prediction and actual normal distributions themselves drawn from normal distributions
  • 100 bootstrap samples used to calculate intervals
  • 500 trials of dataset creation and bootstrap estimates run

On this particular simulation, bootstrapping across all predictions was the only method that adequately covered the intervals it was intended to.

I don't have good intuition at this point why the others overestimate/underestimate.

Full code for simulation: https://github.com/robert-robison/Bootstrap-Simulations/blob/main/bootstrap_simulations.ipynb

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.