1
$\begingroup$

The problem: I'm looking to emit prediction intervals for each predicted value (the mean) in regression. I need that these intervals cover say 90% of true values and be as narrow as possible. In other words I want to learn and emit variance (or noise) which can't be explained by features of the model in each region of data - each sample would have different intervals determined by input vector. For example: Predicting income by number of education years. Given there is no additional data we have I would expect lower variance of income for lower education and higher variance for higher education. Another example - predicting how much years left to leave, by age and health data of a person. Young would have larger variance, while old lower, and old and unhealthy even more lower.

There are two main methods I'm aware of to do it: - non-parametric methods - using Quantile Loss, where trying to get 90% of coverage we can train three models: Quantile:alpha=0.5 for the median and alpha=0.1 and 0.9 for the lower and higher bounds (quantiles) respectfully. - parametric method - suppose some distribution of the noise, and learn it's parameters. For example to get 90% coverage suppose Normal distribution and train models for mean and stdev. Then calculate the interval using stdev. - bayesian methods, essentially parametric as well, but the modeling is done using probabilistic methods.

The question: As mentioned I want the interval be as narrow as possible but still satisfy the needed coverage, which means learning separate models for quantiles or parameters for parametric methods wouldn't provide optimal solution in terms of coverage and width. I'm looking for the loss function which can optimize two things simultaneously. Is there something builtin already in some library and if not what would be the simplest way to implement it? Both parametric and non parametric methods are accepted. Thanks in advance! Alexander

Example The data in example is simulated: 2 independent variables - stage (categorical) and age (axis X), axis Y is the predicted value. The bounds and mean created by 3 separate quantile models. But real data is much more complex, so separate models approach not creating nice results. image

$\endgroup$
2
  • $\begingroup$ Note that $\alpha=0.5$ will give you the median, which will be different from the mean if your distribution is not symmetrical. $\endgroup$ Commented Jan 16, 2020 at 10:24
  • $\begingroup$ thanks, you completely right, I meant for mean in terms of loss optimization but things mixed me up. Fixed in question. $\endgroup$
    – sashaostr
    Commented Jan 16, 2020 at 12:10

1 Answer 1

0
$\begingroup$

To make the prediction interval for a particular forecast point , one needs to efficiently separate the data into signal and noise thus providing the "smallest model error variance" . Now the model might contain not only memory but latent deterministic structure ,, for example pulses.

Now to make a forecast prediction interval one needs to take into account the precise arima structure and the possibility that anomalies might occur on the future. This requires a monte-carlo (boot-strapping) approach.

The resultant is a probability distribution for each point on the future providing the smallest interval.

$\endgroup$
3
  • $\begingroup$ actually this is connected to forecast but I'm not talking about time series data here, rather predicting value for particular sample using regression. For example - predicting income by number of education years. Given I don't have additional data I would expect lower variance of income for lower education and higher variance for higher education. Another example - predicting how much years left to leave, by age and health data. $\endgroup$
    – sashaostr
    Commented Jan 16, 2020 at 12:21
  • $\begingroup$ the question is "what does an analysis of the model's RESIDUALS tell you" $\endgroup$
    – IrishStat
    Commented Jan 16, 2020 at 14:48
  • $\begingroup$ for my best understanding model residuals would give me GENERAL interval for all samples, while I trying to get it depending on the sample values $\endgroup$
    – sashaostr
    Commented Jan 17, 2020 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.