2
$\begingroup$

Say I have a set of measurement values $y_\text{m} = (y_{\text{m},1}, \dots y_{\text{m},N}) $, and compare these with some ground truth $y = (y_1, \dots y_N)$. Then, if I understood correctly, I can estimate the (sample) variance of the error $(y_\text{m} - y)$ in my measurement values as

\begin{equation} \sigma^2 = \frac{\sum_{i=1}^N (y_{\text{m},i} - y_i)^2}{N-1} \end{equation}

(and, if I'm working with a fit instead of ground truth, this changes to the mean square error, with $N-2$ in the denominator).

According to the explanation here, with a sufficiently well-behaved error, I could even compute a confidence interval for $\sigma^2$.

Now, if I'm interested in the relative error, expressed as a percentage, $100 \times (y_\text{m} - y)/y$, how do these estimates for $\sigma^2$ and MSE change? And can I still compute a confidence interval in that case?

EDIT: Sorry, I just realised that in the ground truth case the variance doesn't change, because ground truth is not a stochastic variable. So my question only pertains to the MSE case.

EDIT 2: Doing a further search based on @Demetri Pananos's comments, it looks like the answers here, here, and here may help me get started

Improve this question
$\endgroup$
4
  • $\begingroup$ You could use the delta method, but you would need to estimate the covariance between the model predictions and the data. $\endgroup$
    – Demetri Pananos
    Commented Apr 6, 2023 at 15:24
  • $\begingroup$ Thanks @DemetriPananos, where do I find more information about the delta method? $\endgroup$
    – Sita
    Commented Apr 7, 2023 at 6:13
  • $\begingroup$ you could start with Wikipedia. Most graduate level books on stats should also cover the topic. $\endgroup$
    – Demetri Pananos
    Commented Apr 7, 2023 at 13:23
  • $\begingroup$ @DemetriPananos, got it: en.wikipedia.org/wiki/Delta_method. It's not quite what I was looking for I think, but your hint is clear enough, I'll go and look elsewhere for an answer to my question. Thanks for your help. $\endgroup$
    – Sita
    Commented Apr 11, 2023 at 5:48

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.