3
$\begingroup$

The chi-square statistic is a fairly standard goodness-of-fit test defined as:

$$\chi^2 = \sum_k^N \left ( \frac{d_k-m_k}{\Delta d_k}\right )^2$$

where $\mathbf{d}$ is a vector of data with error $\Delta \mathbf{d}$, and $\mathbf{m}$ is some modelled data. All vectors have length $N$.

This definition ensures that if, on average, the modelled data are within the error bars, then $\chi^2 < N$ and if the modelled data are outside the error bars then $\chi^2 > N$.

The problem with this test is that there is an inherent ambiguity to the test. See this example:

enter image description here

In the above figure, the data are shown as black dots with errors bars and there are two different models shown as red and blue dots. As you can see, both of these models will return $\chi^2 = N$.

What is another type of goodness-of-fit test which will show that both of these models fit the data equally well while simultaneously showing that the models themselves are different?

Any help or ideas are appreciated.

$\endgroup$
1
  • 1
    $\begingroup$ You would need a statistic that returned more than one value at once. COnsider two further models one which alternated being above and below and the other which alternated the other way. There are may ways to both differ from the data and from each other. $\endgroup$
    – Glen_b
    Commented Nov 23, 2018 at 4:02

1 Answer 1

2
$\begingroup$

You need to decide what exactly are you measuring. $\chi^2$ test measures the squared errors. Obviously, after squaring it loses the information about the direction of an error. If the direction was important then you shouldn't be squaring and using this particular test. For instance, you could simply measure the mean error, and in both cases it would be different than zero. For one model it would negative, and positive for another. You could meausue whether the biases are significant, then make a conclusion about it.

However, you seem to pose the question in a general setting, using the biases as an example. I don't think it is possible to have a general test like you want.

$\endgroup$
2
  • $\begingroup$ If I don't square the residuals, what test would I use to test if "the biases are significant"? $\endgroup$
    – Darcy
    Commented Nov 22, 2018 at 22:48
  • $\begingroup$ look at ANOVA comparing the errors $\endgroup$
    – Aksakal
    Commented Nov 23, 2018 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.