I am getting real confused now, What is the difference between, RMSEP (Root Mean Square Error of Prediction), RMSECV(Root Mean Square Error of Cross Validation), RMSEC (Root Mean Square of Calibration), RMSEE (Root Mean Swuare Error of Estimation)

Are they the square root of their corresponding MSE*.

And what about PRESS (Prediction Residual Error Sum of Square)?

Need some help please!

  • It would help (immensely) to (1) spell out these acronyms and (2) provide definitions. Several of these (RMSECV, RMSEC) are used only specialized areas but most readers, given definitions, will be able to explain them to you. – whuber Feb 14 '15 at 15:16
  • I'm very sorry, I should have given the acronyms. I will edit the post. – TheRimalaya Feb 14 '15 at 16:09
up vote 4 down vote accepted
  • yes, the R* (root...) versions are the square root of the corresponding MSEs (mean squared errors)
  • They differ in the type of cases that are used to measure them:

    • RMSEC: calibration error, i.e. the residuals of the calibration data.
      (R)MSEC measures goodness of fit between your data and the calibration model. Depending on the type of data, model and application this can be subject to a huge optimistic bias due to overfitting compared to the (R)MSE observed for real cases when applying the calibration. If the model suffers from not being complex enough (underfitting), calibration error approximates prediction error. But it cannot indicate overfitting.

    • RMSECV: errors are calculated on test/train splits using a cross validation scheme for the splitting.
      If the splitting of the data is done correctly, this gives a good estimate on how the model built on the data set at hand performs for unknown cases. However, due to the resampling nature of the approach, it actually measures performance for unknown cases that were obtained among the calibration cases. I.e. it does not measure how well the model works for cases that are measured months after calibration is done. For that, you need

    • MSEP/RMSEP: prediction error, i.e. measured on real cases and compared to reference values obtained for these.
      RMSEP can measure e.g. how performance deteriorates over time (e.g. due to instrument drift), but only if the validation experiments have a design that allows to measure these influences.

General remarks:

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.