1
$\begingroup$

Consider models such as DeepAR, ngboost and other frameworks to the general problem of predicting the parameters of some parametric distribution with some black-box function, call it f(X). The predicted parameters are chosen to minimize the negative loglikelihood of the assumed parametric distribution.

These models, specifically ngboost, seem to generate accurate probabilistic forecasts, at least according to the paper. However, even after talking with the package creators on this topic, I still don't understand how we can call these intervals "prediction intervals" when parameter uncertainty is ignored.

In a Bayesian setting, we integrate over our posterior uncertainty in order to derive the posterior predictive. This is quite intuitive.

In a frequentist setting, defining prediction intervals is harder in general, but in cases such as ordinary linear regression, there is a closed form solution that clearly captures both uncertainty around the mean (i.e. parameter uncertainty) and the inherent uncertainty in the process.

For other frequentist models such as GAMs and GLMs, we can produce prediction intervals by bootstrapping the sampling distribution (or utilizing the CLT by assuming a multivariate normal distribution for the parameters), and then effectively recreate the Bayesian definition of the posterior predictive distribution by integrating these bootstrap samples against the likelihood of the model. See here as an example, along with this.

However, it is clear that in both DeepAR and ngboost, neither of these probabilistic models do anything like this. It would appear that both of these models just predict the parameters of some parametric distribution via. black box learners f(X), and then prediction intervals are defined as the quantiles of the implied distributions.

This would be, in my mind, the exact same thing as plugging in the predicted mean and estimated sigma parameters from an ordinary least squares model, and using the quantiles of this distribution as a "prediction interval". This clearly ignores uncertainty in estimating both mu and sigma.

I've had this dilemma for quite some time now, where I don't really understand how some of these state of the art probabilistic machine learning models supposedly produce "prediction intervals", even though they ignore the inherent uncertainty in estimating the parameters of the distribution they assume. Any thoughts?

$\endgroup$
6
  • 1
    $\begingroup$ Having read the ngboost stuff in the link, I'm inclined to agree with you. I don't see it either. $\endgroup$
    – jbowman
    Oct 12, 2022 at 20:52
  • $\begingroup$ @jbowman If it helps, here is my public discussion I had with one of the authors of that paper regarding this "issue". Perhaps the intent is to use these models on large datasets, and therefore ignore parameter uncertainty? That's the gist of what I got from this conversation but in hindsight, I am still unsatisfied with the response: github.com/stanfordmlgroup/ngboost/discussions/261 $\endgroup$
    – aranglol
    Oct 12, 2022 at 21:07
  • $\begingroup$ Well, that link seems pretty definitive, doesn't it? They don't incorporate parameter estimation uncertainty in their predictive distributions, and seemingly rely on lots of data to make that a small, therefore ignorable, contribution to the predictive uncertainty. Along those lines, remember Gelman's dictum: the most important formula in statistics is $12^2 + 5^2 = 13^2$, implying that effects with relatively small(ish) standard deviations contribute very little to overall variability relative to effects with relatively large standard deviations. $\endgroup$
    – jbowman
    Oct 12, 2022 at 21:28
  • $\begingroup$ @jbowman Doesn't that imply that in these models, if we were able to incorporate parameter uncertainty in these models that we would naturally get better prediction intervals, however small the contribution ends up being? I just find the answer a bit hand wavy and unsatisfying. Furthermore, there is no proof given that the base estimators f(X) are consistent for the parameters (at a fast enough rate) in general use cases such that with large enough data, the parameter uncertainty can be ignored. Maybe I am just misunderstanding or over interpreting. $\endgroup$
    – aranglol
    Oct 12, 2022 at 21:59
  • 1
    $\begingroup$ No, I don't think you are misunderstanding or overinterpreting, and I too think the response is a bit handwavy and unsatisfying. It would be good to have seen how well-calibrated those predictive confidence intervals were on their test datasets, for example. $\endgroup$
    – jbowman
    Oct 12, 2022 at 22:11

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.