0
$\begingroup$

I have mean level spacing as input data. With this data I plot histogram using 10 bins. x ranges from 0 to 9 because bins are 10, f is the bin counts(means how many values in each bin). Then I calculate mean of the distribution using formula $\mu = \frac{\sum_{i}^{10}(f\times x)}{\sum_{i=1}^{10}{f}}$ . After this I used formula of Poisson distribution $P=\frac{\mu^{x}\exp(-\mu)}{x!}$ and then calculated the expected frequency as $E=x \times \sum_{i=1}^{10}{f}$. After measuring expected frequency of Poisson distribution I used formula $\chi^2 = \sum_{1}^{10}\frac{(O -E)^2}{E}$. But my answer is 3000 that is extremely large as compared to the values given in Chisquare distribution table. Its hard for me to figure out that why it is too large. Where I'm wrong? I also want to menton that I'm working on machine learning project. Here is the histogram of my input data. For plotting this histogram I choosed 4000 samples and each single sample has 256 features so total values use to plot histogram are $4000 \times 256$. If I increase number of bins say 30,50 then the values in the last bins are too small approximately zero. So I preferred to choose only 10 bins to get meaningful result. There is one more thing that I want to mention that in each sample I measure measure nearest neighbor spacing $E_{N}-E_{N-1}$. After measuring nearest neighbout spacing in each sample 255 features will be left. After collecting 255 features of all the 4000 samples in a single list I divide it y mean value. What I want to say is that I used normalized level spacing to plot this histogram. My input data has both positive and negative values and the level spacing is also too small. My final $X^2 = 6.445144657597967\exp^{28}$ and p_value= 0.0

enter image description here

Here is the plot for more bins. But here I plotted probability. The pupose of this graph is to show that if I increase number of bins values become zero and I read that chisquare test is not suitable for small values. enter image description here

$\endgroup$
4
  • 1
    $\begingroup$ When it's so large, the null hypothesis is blatantly bad: you don't need the table to tell you that. Another possibility is your calculation is incorrect. You would need to supply more information to determine what's going on. $\endgroup$
    – whuber
    Commented Jul 10, 2019 at 14:34
  • $\begingroup$ Can you edit your post to include your data and/or your histogram? Why do you believe a Poisson distribution would be appropriate for your binned data? $\endgroup$ Commented Jul 10, 2019 at 14:35
  • $\begingroup$ Sure i edit this $\endgroup$
    – jerry
    Commented Jul 10, 2019 at 14:36
  • $\begingroup$ What else should I need to provide? $\endgroup$
    – jerry
    Commented Jul 10, 2019 at 14:52

1 Answer 1

1
$\begingroup$

How do you condense $4,000\times 256$ two-dimensional data points into ten bins? And why do you believe the result should be Poisson distributed?

It seems like the answer simply is that it isn't, and that with your huge sample size, even small deviations from a posited distribution yield a very large test statistic (and correspondingly, a tiny p value).

$\endgroup$
2
  • $\begingroup$ Data is not 2 dimensional. I collected the $4000 \times 256$ values in a single list to plot histogram. When I plot the histogram with normalized values it seems similar to Poisson. In my model I'm sure that at high temperature data it follows Poisson distribution. Above 4000 samples are from high temperature . But there is also a probability of following semi-poisson distribution. Two distributions are quite similar. I want to proof with chisquare test that there is more evidence that it follows Poisson distribution. $\endgroup$
    – jerry
    Commented Jul 10, 2019 at 15:10
  • $\begingroup$ Yes 10 bins are too small. But in higher bins data points are close to zero. I can show that figure. $\endgroup$
    – jerry
    Commented Jul 10, 2019 at 15:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.