2
$\begingroup$

I would like to understand the difference between Root Mean Squared Error and the Standard Prediction Error.

The SEP formula is simillar to the RMSE, but with an aditional term called bias inside the root squared.

Is expected to give similar values for my model?

$$RMSE = \sqrt {{\sum_{i=1}^n(y_i -\hat y_i)^2} \over n}$$

$$SEP = \sqrt {{\sum_{i=1}^n(y_i -\hat y_i-bias)^2} \over {n-1}}$$

$$bias = {{\sum_{i=1}^n(y_i -\hat y_i)} \over n} $$

$\endgroup$
3
  • 1
    $\begingroup$ Welcome to StackExchange! Please add the formulas you are referring to, to avoid misunderstandings. $\endgroup$
    – user289381
    Jul 25 '20 at 12:35
  • 1
    $\begingroup$ Sorry, i added the pictures for the two formulas. thanks! $\endgroup$
    – Tiago Dias
    Jul 27 '20 at 10:03
  • $\begingroup$ Can you provide a link to an explanation of the SEP term? Never heard of it and could hardly google it $\endgroup$ Jul 27 '20 at 10:20
1
$\begingroup$

SEP functions similarly to RMSE, but the bias term acts to adjust the mean of the predictions to match the mean of the actuals. That is, if you were to add a constant term to all of your predictions, you would degrade your RMSE would would not affect your SEP.

Another way to express SEP is:

$$ SEP(y, \hat{y}) \equiv RMSE(y, \hat{y} + \bar{y} - \bar{\hat{y}}) $$

I can think of a few cases where predicting the mean accurately is less relevant and you might prefer a metric like SEP:

  • You care more about the rank order of your predictions than their absolute magnitude (in this case there are other metrics like normalized gini that are also useful).
  • You care more about the relative difference between predictions than their absolute magnitude.
  • You expect that the mean varies noisily and is hard to get right, though that it is still be possible to predict the differences accurately, hence you favour a metric that places less emphasis on accurately predicting the mean.
  • Some other process over which you have no control will adjust the mean of your predictions later.
  • Your particular holdout set has a large shift in mean versus your training set and you do not want to favour models that skew in that direction.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.