3
$\begingroup$

I am currently working with time series data. My objective is to predict the a certain value at time t given some other variables that we will know the same day ( but prior to our objective variable). After trying several models I have managed to obtain a relatively good prediction using lasso regression.

However, given the importance of the problem, I would like to have some kind of confidence interval for my predictions, it would be very important to understand how accurate my prediction would be given a certain probability.

One solution I have though about is using a certain number of past MAE to compute the standard deviation and with that, and assuming they errors have a normal distribution compute a confidence interval at 95% with +-2 s.d.

One important consideration is that my dependent variable does not behave the same through the years, it is not stationary.

Would this be a robust way of computing this intervals or are there better alternatives?

$\endgroup$
5
  • $\begingroup$ Check RMSEP (Root Mean Standard Error of Prediction). This value tracks your model's ability to predict on out-of-sample values. You can calculate RMSEP over different time periods to determine how well you are predicting, which will give you a "standard error" of prediction which could be used in the calculation of a rough confidence interval. $\endgroup$ – ERT Aug 1 '18 at 11:57
  • $\begingroup$ There's not really any reason you need to assume a normal distribution here. Collect out-of-sample prediction errors at the horizon you're interested in, then look at their empirical distribution. $\endgroup$ – Chris Haug Aug 1 '18 at 12:05
  • $\begingroup$ Chris, sorry but when I said I would assume normality, I meant it to illustrate the +-2 s.d with 95% confidence. My intention is to apply you suggestion. ERT thanks for the suggestion, I didn't know that measure. $\endgroup$ – surface Aug 1 '18 at 13:19
  • $\begingroup$ @ERT: your comment sounds like a good answer, do you want to turn it into one? Chris' suggestion is also good, perhaps you want to work it in. surface: you are looking for a prediction-interval, not a confidence-interval. Note that there is a difference. $\endgroup$ – Stephan Kolassa Aug 1 '18 at 15:23
  • $\begingroup$ A potentially useful article is Ziel & Liu (2016, IJF). I can't tell whether it is relevant, but at least it uses the lasso and calculates probabilistic predictions, so you may be able to get something out of it. $\endgroup$ – Stephan Kolassa Aug 1 '18 at 15:28
0
$\begingroup$

As suggested in the comments, I am turning my comment into an answer so the question will have an answer in the system.

Original:

Check RMSEP (Root Mean Standard Error of Prediction). This value tracks your model's ability to predict on out-of-sample values. You can calculate RMSEP over different time periods to determine how well you are predicting, which will give you a "standard error" of prediction which could be used in the calculation of a rough confidence interval.

I suppose a bit of an explanation of this answer is needed, since only a certain number of characters are allowed per comment. First, as @StephanKolassa noted in his comment, there is a difference between prediction intervals and confidence intervals. This is an important distinction. In RMSEP calculation, one is attempting to understand how well a model can predict, and can compare different models by relative magnitudes of RMSEPs. The RMSEP by itself may not be extremely useful (like $R^2$ for simple regression would be), but it can be enlightening for model comparison.

Additionally, building on @ChrisHaug's comment, you may want to look at the out-of-sample errors you use to calculate your RMSEP measure. They may provide you with an empirical distribution which could shed some light on how well your model is predicting. For example, if your errors have an extremely heavy tail, it may indicate that your model does not do a great job if you are looking to avoid extremely unlikely but costly tail-events, (like an asset manager attempting to avoid exposure to recession-level events).

$\endgroup$
1
  • $\begingroup$ Thanks for the more in depth answer. Indeed, my errors are quite asymmetric but I am able to explain this asymmetry so it is not a problem(I suppose) as I can account for it ex-ante.. $\endgroup$ – surface Aug 1 '18 at 16:20
0
$\begingroup$

It's not exactly your question (?) but you may want to look at the R function selectiveInference::fixedLassoInf which computes confidence intervals for LASSO coefficients. The documentation refers to arXiv:1311.6238 and arXiv:1602.07358

$\endgroup$
6
  • $\begingroup$ "I would like to have some kind of confidence interval for my predictions, it would be very important to understand how accurate my prediction would be..." OP implies they are interested in understanding the accuracy of predictions made by the model, not CIs for coefficients. $\endgroup$ – ERT Aug 1 '18 at 16:34
  • $\begingroup$ But if you know CIs on coefficients and the probability distribution of the error term (maybe normal), then you can derive confidence intervals on predictions. $\endgroup$ – paf Aug 1 '18 at 16:55
  • $\begingroup$ That sounds a little more complicated than what OP was looking for... $\endgroup$ – ERT Aug 1 '18 at 16:58
  • 1
    $\begingroup$ The procedure described in the answer by @ERT measures the actual, realized error distribution, without any assumptions. What you are suggesting requires the distribution of the error term to be normal AND for the model to be correct and stable over time. You will in all likelihood underestimate the uncertainty in your predictions. $\endgroup$ – Chris Haug Aug 1 '18 at 17:24
  • 1
    $\begingroup$ @paf I should have said "requires the model form to be correct" (the description of the model aside from the specific values of the parameters). Most naïve ways of computing PI's are too narrow because they don't account for model uncertainty/instability (actually, in the time series literature, PI's often don't even account for parameter uncertainty either, so they are even worse). $\endgroup$ – Chris Haug Aug 2 '18 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.