Difference between confidence interval and RMSE

Question

For a long time I have been doing chemometrics using Partial Least Squares regression to predict the composition of samples based on their spectra, and calculating the root mean squared error (RMSE) as an estimate of the accuracy of the model.

I am now looking into using gaussian process regression to do something similar. As a Bayesian method, GPR results automatically come with a confidence interval, but I could also calculate an RMSE just as I do with PLS.

So, my question is: If I want to report the accuracy of a prediction using GPR, do I report the RMSE or the confidence interval (or both)?

Answer 1

By confidence-interval, you probably mean a prediction-interval. The two terms are often confused, but yes, there is a difference.

Point predictions and prediction intervals offer different kinds of information.

• Point predictions predict a single number. You can assess their quality using rms, or a lot of other accuracy KPIs, like the mse, the mae or even the mape. Which KPI is most meaningful really depends on your loss function - the mse penalizes large errors much more than the mae.

  If your Bayesian method gives you an entire posterior distribution, you can extract a point forecast from it in various ways, e.g., by taking the mean, or the median, or the mode. Fun fact: if you use the rmse or the mse, take the mean - and if you use the mae, take the median. There won't be much of a difference if the posterior distribution is symmetric, but if it's asymmetric, this can indeed make a difference.

• Prediction intervals give interval-valued predictions, which aim at covering the actual value, say, 80% or 90% of the time if you run "many" predictions. The simplest way to assess the quality of a prediction-interval would be to count whether they indeed cover your prespecified proportion of actuals. Maybe test the resulting contingency table or run a binomial test if you want to see whether the deviation from your prespecified coverage is statistically significant.

Whether you actually want a point prediction or a prediction interval is up to what you want to do with your predictions. Point predictions are easier for non-statisticians to grasp. PIs allow you to do scenario analyses; they tell you how sure you are about a prediction (if they are well calibrated).

Answer 2

There are multiple techniques, Juš Kocijan in the paper "Output-Error Model Training for Gaussian Process Models" explain why and what method also can be applied. In the paper, he suggests the use of the log predictive density error(he reference the book of Rasmussen).