Obtaining the p-value of a chi-square test in Matlab

Question

I am currently fitting a model to a dataset. To measure the goodness-of-fit I am using the chi-square test with

$H_0$: The model fits the data ($\chi^2 \lt \chi^2_{critical}$)

$H_1$: The model does not fit the data ($\chi^2 \gt \chi^2_{critical}$)

The figure below shows the datapoints in black as well as the fitted model in orange.

The datapoints in black are occurences of an event (disclosure of a security vulnerability) along the time axis. At time 1 there have been 2 events in total. At time 101 there have been 160 events in total.

Thus, $\text{df} = 101-1 = 100$

The orange curve is obtained by fitting the alhazmi malaiya logistics model (a known model for modelling the vulnerability discovery process) given by the equation

$\Omega(t) = \frac{B}{B\times C\times e^{-A\times B\times t} + 1}$

Parameters A, B and C are selected during the fitting process such that the model describes the datapoints as good as possible. Therefore the parameter combination that results in the lowest $\chi^2$ has been selected.

$\chi^2$ is calculated by using the datapoints (black) as $o_i$ and the values obtained by solving the equation at time t as $e_i$ in the formula

$\chi^2 = \sum\frac{(o_i - e_i)^2}{e_i}$

This gives me in my case $\chi^2 = 111.8410$ and by selecting $\alpha = 5\%$ I get a critical value of $124.3421$. As $111.8410 \lt 124.3421$ I accept my $H_0$ which states that the datapoints are distributed according to the model or, in other words, the model describes the data reasonably well.

In the above case, what exactly is the P-Value of this $\chi^2$-Test?

According to Pearson's chi-square test, the P-Value is calculated by

$ \text{P-Value} = 1 - \text{chi2cdf}(\chi^2, \text{df})$

This would in this case result in a P-Value of $0.0967$ which would suggest a significant fit of the model. But shouldn't the P-Value for a good fit (a fit that has been especialy made for the data) be close to 1?

Is this calculation correct?

Unfortunately, the literature I am working with does not properly explain the used methodology and I can't seem to reproduce the results. One of the papers using the approach above is http://www.cs.colostate.edu/~malaiya/pub/issre05.pdf. Their goodness-of-fit test is described on page 5, last section.

Answer 1

Several points:

• If all parts of the model and assumptions are correct, the p-value should be uniform in (0,1), not close to 1 (except by chance); a value less than 0.01 should be no less likely than a value greater than 0.99

• The very small p-value suggests that - for the amount and variability of data you have - the data are distinguishable from the fitted model.

• I have concerns that several of the assumptions under which the chi-square might apply may not hold in this situation, but there's not enough details in the question to be sure either way.

  Edit^2: Note that cumulative counts are not independent. Let $Y_t=X_1+X_2+...+X_t$ be the current cumulative count, where $X_t$ is the additional count of notifications in period $t$.

  $\text{Cov}(Y_{t-1} ,Y_{t}) = \text{Cov}(X_1+X_2+X_3+...+X_{t-1} ,X_1+X_2+X_3+...+X_{t}) = \text{Var}(X_1+X_2+X_3+...+X_{t-1}) + \text{Cov}(X_1+X_2+X_3+...+X_{t-1}, X_{t})$

  That's not going to be zero, so they'll certainly be correlated.

  As a result I don't believe this chi-square calculation makes sense.

  A chi-square on the increments may be much closer to being reasonable, but I don't know that it necessarily makes sense to assume complete independence there either (that may take some domain-knowledge).

• Even if the assumptions do hold (or are close enough that it doesn't matter much), the fitted curve may nevertheless actually be "good" in several senses. For example, with enough data you can reject fairly trivial deviations from the model, where the model -- even though you can be fairly sure the deviations from the data are not just due to random variation under the model -- may be perfectly adequate for some purposes. (Your purposes are not made clear, so again, this is hard to discuss.)

Answer 2

The chisquare statistic you discuss is usually used to test the goodness of fit for contingency table data. This is based on the assumption that the distribution of cell counts is multinomial, and the $(O_i - E_i)^2/E_i$ for each cell has an expected $\chi^2_1$ distribution (which is then added up). The problem is that this distribution is not expected with a general continuous distribution of $E$ and $O$.

If you are interested in a global goodness of fit test, try binning your predicted values according to their deciles and calculating the same test. This is the Hosmer Lemeshow test of calibration. This is described in their book "Logistic Regression".