2
$\begingroup$

I am currently fitting a model to a dataset. To measure the goodness-of-fit I am using the chi-square test with

$H_0$: The model fits the data ($\chi^2 \lt \chi^2_{critical}$)

$H_1$: The model does not fit the data ($\chi^2 \gt \chi^2_{critical}$)

The figure below shows the datapoints in black as well as the fitted model in orange.

enter image description here

The datapoints in black are occurences of an event (disclosure of a security vulnerability) along the time axis. At time 1 there have been 2 events in total. At time 101 there have been 160 events in total.

Thus, $\text{df} = 101-1 = 100$

The orange curve is obtained by fitting the alhazmi malaiya logistics model (a known model for modelling the vulnerability discovery process) given by the equation

$\Omega(t) = \frac{B}{B\times C\times e^{-A\times B\times t} + 1}$

Parameters A, B and C are selected during the fitting process such that the model describes the datapoints as good as possible. Therefore the parameter combination that results in the lowest $\chi^2$ has been selected.

$\chi^2$ is calculated by using the datapoints (black) as $o_i$ and the values obtained by solving the equation at time t as $e_i$ in the formula

$\chi^2 = \sum\frac{(o_i - e_i)^2}{e_i}$

This gives me in my case $\chi^2 = 111.8410$ and by selecting $\alpha = 5\%$ I get a critical value of $124.3421$. As $111.8410 \lt 124.3421$ I accept my $H_0$ which states that the datapoints are distributed according to the model or, in other words, the model describes the data reasonably well.

In the above case, what exactly is the P-Value of this $\chi^2$-Test?

According to Pearson's chi-square test, the P-Value is calculated by

$ \text{P-Value} = 1 - \text{chi2cdf}(\chi^2, \text{df})$

This would in this case result in a P-Value of $0.0967$ which would suggest a significant fit of the model. But shouldn't the P-Value for a good fit (a fit that has been especialy made for the data) be close to 1?

Is this calculation correct?

Unfortunately, the literature I am working with does not properly explain the used methodology and I can't seem to reproduce the results. One of the papers using the approach above is http://www.cs.colostate.edu/~malaiya/pub/issre05.pdf. Their goodness-of-fit test is described on page 5, last section.

$\endgroup$
  • $\begingroup$ 1. Can you clarify whether the data are counts, or continuous observations with known standard deviations (or perhaps something else)? 2. I note that there seems to be not a single downward movement. Are these values cumulative sums of some process or is the noise just extremely low? $\endgroup$ – Glen_b Oct 14 '15 at 15:47
  • $\begingroup$ The data is cummulative occurences of a random event along the time axis. So there is few events in the beginning and the rate increases after around 60 time units. $\endgroup$ – sge Oct 14 '15 at 16:01
  • 1
    $\begingroup$ @AdamO The ans output was only to show the values i am working with. The actual call for chi2cdf looks like chi2cdf(118.8032, 159). The result is 0.0073. $\endgroup$ – sge Oct 14 '15 at 16:10
  • 1
    $\begingroup$ @Glen_b I am performing the chi square by calculating x2 as $\chi^2 = \sum \frac{(o_i - e_i)^2}{e_i}$ where $o_i$ and $e_i$ are the observed and expected values. This gives me $\chi^2 = 118.8032$ (which is the smallest value for all tested parameter combinations for this model). $\endgroup$ – sge Oct 14 '15 at 16:12
  • 1
    $\begingroup$ @Lorunification and you calculated 118.8032 as an $(O-E)^2/E$ type statistic? Are you binning your continuous distribution into deciles? There is no $E$ for continuous distributions. $\endgroup$ – AdamO Oct 14 '15 at 16:14
4
$\begingroup$

Several points:

  • If all parts of the model and assumptions are correct, the p-value should be uniform in (0,1), not close to 1 (except by chance); a value less than 0.01 should be no less likely than a value greater than 0.99

  • The very small p-value suggests that - for the amount and variability of data you have - the data are distinguishable from the fitted model.

  • I have concerns that several of the assumptions under which the chi-square might apply may not hold in this situation, but there's not enough details in the question to be sure either way.

    Edit^2: Note that cumulative counts are not independent. Let $Y_t=X_1+X_2+...+X_t$ be the current cumulative count, where $X_t$ is the additional count of notifications in period $t$.

    $\text{Cov}(Y_{t-1} ,Y_{t}) = \text{Cov}(X_1+X_2+X_3+...+X_{t-1} ,X_1+X_2+X_3+...+X_{t}) = \text{Var}(X_1+X_2+X_3+...+X_{t-1}) + \text{Cov}(X_1+X_2+X_3+...+X_{t-1}, X_{t})$

    That's not going to be zero, so they'll certainly be correlated.

    As a result I don't believe this chi-square calculation makes sense.

    A chi-square on the increments may be much closer to being reasonable, but I don't know that it necessarily makes sense to assume complete independence there either (that may take some domain-knowledge).

  • Even if the assumptions do hold (or are close enough that it doesn't matter much), the fitted curve may nevertheless actually be "good" in several senses. For example, with enough data you can reject fairly trivial deviations from the model, where the model -- even though you can be fairly sure the deviations from the data are not just due to random variation under the model -- may be perfectly adequate for some purposes. (Your purposes are not made clear, so again, this is hard to discuss.)

$\endgroup$
2
$\begingroup$

The chisquare statistic you discuss is usually used to test the goodness of fit for contingency table data. This is based on the assumption that the distribution of cell counts is multinomial, and the $(O_i - E_i)^2/E_i$ for each cell has an expected $\chi^2_1$ distribution (which is then added up). The problem is that this distribution is not expected with a general continuous distribution of $E$ and $O$.

If you are interested in a global goodness of fit test, try binning your predicted values according to their deciles and calculating the same test. This is the Hosmer Lemeshow test of calibration. This is described in their book "Logistic Regression".

$\endgroup$
  • $\begingroup$ The problem is, that the chi-square test as i am using it is used in all of the papers in the same research area. I added one of the papers in the question as a reference. $\endgroup$ – sge Oct 15 '15 at 9:10
  • 1
    $\begingroup$ @Lorunification It's somewhat well known that such tests applied to continuous distributions require some form of grouping. Either this detail was omitted from the writing or the test results are dubious. $\endgroup$ – AdamO Oct 15 '15 at 17:38
  • $\begingroup$ Ok. So my application of the test per se is not correct since my continuos data does not allow such test. I will read again into the literature and maybe contact the authors and report back as soon as i have found the proper way to perform the test. Thank you very much already! $\endgroup$ – sge Oct 15 '15 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.