# Machine Learning - Prediction Interval - Cheating?

I work at a company that is trying to use machine learning methods in particular gradient boosting and neural networks to make predictions on stock market data, so using historical data to predict what the price of a stock/asset will be $$x$$ time periods from the present. We are using these methods for regression as opposed to classification, and it is my habit being trained in experimental sciences to always give a regression prediction in terms of a $$\pm$$, giving an interval of prediction, rather than just one number. My manager (who doesn't seem very technical) told me that this is unacceptable/cheating, as I'm using an interval to cover the fact that my algorithm is unable to produce a single correct number. I'm a bit confused about this attitude, as coming from laboratory sciences (chemistry) we always cite every result in terms of a $$\pm$$.

So, I was wondering what the stats experts on here thought? Just out of curiosity, I checked my Machine Learning textbook by Hastie, Witten, et. al., and they use the MSE on the test set to give a $$\pm$$ on the predictions from an example they use on gradient boosting, so it seems standard to do this...

Thanks.

• It isn't cheating, it is being honest about your accuracy. Imagine you make a prediction which turns out to be wildly inaccurate. If you include a margin of error, you have covered your ass by saying "This is my prediction but I am really uncertain". Sounds like your boss doesn't really understand. Commented Nov 16, 2018 at 22:24
• i think you want to change company. you might want to go to the website nuclear phynance which has experts in this field. Commented Nov 16, 2018 at 22:31

## 2 Answers

In general, prediction intervals are considered better than point estimates. While it's great to have a good estimate for what a stock price will be tomorrow, it's much better to be able to be able to give a range of values that the stock price is very likely to be in.

That being said, it's generally more difficult to produce reliable prediction intervals than merely produce point estimates that have good prediction properties. For example, in many cases we can show that with non-constant variance, we can still produce a consistent estimator of the mean of the new value even if we ignore the non-constant variance issue. However, we definitely need a reliable estimate of the variance function to produce prediction intervals.

I've heard of people just treating this as another level of a machine learning problem: the first level is to produce a function $$\hat f(x_i) = E[\hat y_i | x_i]$$, the estimates of the values and the second level is to produce a function $$\hat V(x_i) = E[(y_i - \hat y_i)^2 | x_i]$$, the estimates of the variance of the function given the inputs. In theory, this should work (given enough data with a stable function), but in practice, it must be handled with a lot of care, as variance estimates are inherently much less stable than mean estimates. In short, you should expect to need much more data to accurately estimate $$\hat V(x_i)$$ than $$\hat f(x_i)$$.

So there's definitely nothing about prediction intervals that is "cheating" compared with just producing point estimates. It's just harder to do. As an empirical example, in the M4 forecasting competition, only 2 of the 15 methods that produced 95% prediction intervals had nearly correct coverage; most of the other prediction intervals had coverage in the 80-90% range (see slide 35 in the link).

• You can get marginally-valid prediction intervals with exactly correct coverage using conformal inference, using essentially any arbitrary machine learning algorithm. Estimating the variance may help, but won’t affect the coverage at least.
– guy
Commented Nov 18, 2018 at 21:01
• @guy: thanks! Conformal inference is a new topic for me and I've started here to learn about it. Commented Nov 27, 2018 at 21:59

I don't understand your manager's attitude. If the model predicts that the stock will be 173.56, and it's actually 173.55, will they consider that a "failure"? If you're trying to make money from the stock market, you shouldn't be depending on getting the price exactly right. Stock investment is all about reducing variance, so knowing which predictions have the smallest error bars is key.

The basic regression model is that $$Y = mX+b+\epsilon$$, where $$\epsilon$$ is a normally distributed error term with mean 0 and standard deviation $$\sigma$$. In training the model, we find the $$m$$ and $$b$$ that minimize $$\sigma$$. When we use the model for prediction, we predict that given a particular $$x$$, the $$y$$ will be normally distributed with mean $$mx+b$$ and standard deviation $$\sigma$$. Thus, we are not predicting the value of $$y$$, we are predicting the distribution of $$y$$.

In this model, the exact value $$y$$ does not deterministically depend on $$x$$. If the depedance were both deterministic and linear, one wouldn't need linear regression to begin with; one could simply solve for $$m$$ and $$b$$. One can have different models, and those models could be nonlinear and/or deterministic. But generally speaking, regression models are built around the idea that you're going to have some error, and there should be some loss function quantifying how much a particular error matters, and that loss function is then minimized over the training set. If your manager is trying to create a model that incorporates all the factors that deterministically determine stock prices, that is incredibly ambitious.