1
$\begingroup$

I need to fit a 7-parameter function (actually 6 since one of them is fixed) on a histogram using ROOT. The function is basically an asymetric gaussian curve. The algorithm I'm using actually makes between 20 and 1000 fits and keeps the parameters that produces the highest p-value.

After each try, the code makes a chi-squared goodness-of-fit test to check whether or not the p-value is at least 0.05; and it stops trying when enough fits have produced a p-value over this limit. My problem is that the p-values I get are all way smaller than 0.05 (sometimes as low as 10$^{-200}$), even though the resulting curve fits correctly to my data. Here is an example :

Example of fit that looks OK, but gives a p-value of 10$^{-7}$.

As you can see, the fit seems to be OK, but the p-value for this particular set was around 5$\times$10$^{-7}$ (ROOT indicates that $\chi^2$ = 168.8 and that there are 88 degrees of freedom, since only about 90% of the histogram was used for the fit). With such p-values, the algorithm always tries to make 1000 fits even though the results do not get better, and I have over 60,000 datasets to fit, so the computation time is needlessly horribly long.

I thought maybe there are too many degrees of freedom, or maybe my histogram is too "noisy" for the goodness-of-fit test to be relevant. I thought I could rebin my histogram, maybe regroup bins in pairs as to lower the importance of random fluctuations. However, I wonder if this method is legitimate? Isn't it some form of p-hacking, ie presenting the data in a way that will make my fit more valid than in reality? Of course, I will remake the fit on the rebinned histogram (and not keep the initial parameters on the new histogram), but I expect the parameters to be only slightly modified (except for the amplitude, which is not important for my study). The point is that it should lower the chi-squared (faster than the number of degrees of freedom I hope) and increase the p-value, thus making the fit "more valid". Hopefully this way the algorithm won't have to make 1000 unnecessary tries to fit the data.

Is rebinning the best solution here? How much would be enough; and more importantly, how much is too much? Of course, I could present my data in a 3-bin histogram and the chi-squared would be pretty low, but this seems like cheating (and I really want to get the best parameters for my data, not just "a good fit"). Should I limit my test to a smaller interval of the histogram? Or should I try another goodness-of-fit test?

Any suggestion is welcome. Thanks a lot.


I'm editing to bump the question, as many details have been added in the comments below, and to include more people in the discussion. As noted by @Kodiologist, there might be a better method to fit the distribution than to put it in a histogram, I would like to get opinions on the matter. I thought I could fit the CDF instead of the PDF, but since the function I'm fitting is only a surrogate function (read: approximate), I'm not sure the CDF would give better results. Also, the exact form of the CDF has not been described anywhere. It could be derived, but perhaps it is not easy to do.

$\endgroup$
  • $\begingroup$ "I need to fit a 7-parameter function (actually 6 since one of them is fixed) on a histogram" — Why? Provide context. $\endgroup$ – Kodiologist May 25 '17 at 2:07
  • $\begingroup$ There are objective criteria for determining bin width. See the references on this page that should let you avoid the problem of p-hacking. $\endgroup$ – David Lane May 25 '17 at 2:21
  • $\begingroup$ @Kodiologist I believe context is irrelevant to my question as it could apply to any statistical study (ie what is the best goodness-of-fit test given the data above, and/or is it legitimative to modify the bin width). Anyway, since you asked, I'll just say I'm doing a research internship that could possibly lead to a publication. So this is why I don't want to do anything too shady with my data like p-hacking. $\endgroup$ – Jasmeru May 25 '17 at 11:42
  • $\begingroup$ @DavidLane Thank you. I found some rules of thumb like the Freedman-Diaconis rule which seems to be an objective solution. However, I find that my optimal bin size should be around 9 times its initial width, so the number of bins is reduced to about 10-11 (by comparison with 100 initially). It seems pretty extreme to get 6 parameters (4 of which are very important for my study, so it has to be extremely precise) out of a 10-bin histogram fit. Could I justify a smaller bin size objectively, say in a scientific article? Thanks. $\endgroup$ – Jasmeru May 25 '17 at 11:49
  • 1
    $\begingroup$ "I believe context is irrelevant to my question" — Certainly not. Statistics isn't a machine that takes numbers in one end and produces APA-style results sections out the other. The right tool for the job depends on things like where the data came from, what the variables measure, and what you're trying to accomplish. In particular, I doubt that fitting a function to a histogram will do you any good, but unless you provide these details, nobody can give you specific advice. $\endgroup$ – Kodiologist May 25 '17 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.