1
$\begingroup$

I have circular data -- observations each of which falls between -180 degrees and +180 degrees -- divided into a 15-bin histogram.

I'd like to see how well a continuous PDF -- specifically a mixture of a von Mises distribution and a uniform distribution, with particular parameters -- fits the observed histogram.

And to determine this fit, I'd like to use the r-squared statistic. I'd like to leave aside the question of whether I should be using r-squared or something else. My choice of r-squared is based on Zhang & Luck, Nature, 2008 and Zhang & Luck, Psychological Science, 2009, work I'm trying to replicate. (These papers did exactly what I'm describing I want to do -- compute the r-squared between a 15-bin histogram of circular data and the mixture model.) But if you'd like to suggest a better method, and can describe it clearly, I'd be happy to try it out.

My question is, how should I compute the r-squared? Should I bin the continuous function, and then compare the PDF bin heights to the observed bin heights? Should I take the mean of the PDF over the range spanned by each bin of the observed data? Should I compare the bin centers to the corresponding points in the continuous function?

Improve this question
$\endgroup$
16
  • 2
    $\begingroup$ You may feel obliged to use R-squared but that does not oblige us to feel that compulsion. My own view is that R-squared makes no sense unless you have observed and expected values of a variable (not a density). If anything it makes even less sense here without modifying predictions modulo 360 degrees. This is not an attack and you should not feel that you have to inject personal comments in defence or justification. Nevertheless the onus really is on you to show that the idea makes sense. $\endgroup$
    – Nick Cox
    Commented Jul 26, 2013 at 19:03
  • $\begingroup$ Nick, thanks for the reply. I have two responses: 1) I think it's legitimate for me to ask how something should be done, without regard for whether it should be done. Those are two separate questions: People can disagree about which method is best, but agree on the proper way to perform a certain method, whether or not they think it's the best one. 2) You say "R-squared makes no sense unless you have observed and expected values of a variable." This is exactly what I have. I have observed frequencies in my histogram, and my PDF provides expected values for these frequencies. $\endgroup$
    – user28511
    Commented Jul 26, 2013 at 19:12
  • $\begingroup$ Your 1) is your rule about what you think is a fair question, but others may have their own ideas about what is fair technical comment. On 2), a density is continuously defined, and only in very weak sense analogous to a response with a finite set of measurements for which $R^2$ can be calculated. You'll have to hope that others are more inclined to answer your question in your own terms. Certainly you can bin into frequencies, but any fit measure will depend arbitrarily on those bins, and needs to take the circular scale into consideration. $\endgroup$
    – Nick Cox
    Commented Jul 26, 2013 at 19:20
  • 1
    $\begingroup$ Consider the case where you have equal counts in all 15 bins, assumed of equal widths. Obviously the uniform distribution is a perfect fit. But then (upon comparing the counts to their expected values with the uniform distribution) the R^2 is not even defined. Worse, the slightest variation in the counts will allow R^2 to be calculated, but its value will be practically arbitrary, somewhere between -1 and 1. This alone shows that using R^2 is inappropriate and likely to lead to erroneous results. $\endgroup$
    – whuber ♦
    Commented Jul 26, 2013 at 19:41
  • 1
    $\begingroup$ I believe Z&L are referring to something other than a distributional fit. They are assessing a different relationship when they are trying to "explain variance." Using $r^2$ to assess a fit of a distribution would be a gross, elementary error, one that I doubt would get past the reviewers or readers of Nature. $\endgroup$
    – whuber ♦
    Commented Jul 28, 2013 at 21:35

3 Answers 3

Reset to default
2
$\begingroup$

You can plot empirical data vs your distribution with fitted parameters. Please see the answer of this link and plot histogram of your original data and smooth histogram of your fitted data as data1 is the original data and data2 is your fitted data using MLE estimates.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ The idea of visualizing the data is good, but comparing histograms is inadvisable: there are much better ways to compare distributions. Please see stats.stackexchange.com/questions/51718/… for a critical evaluation. $\endgroup$
    – whuber ♦
    Commented Jul 28, 2013 at 21:32
  • 1
    $\begingroup$ I'm also a strong believer in the importance of visualizing data, but was hoping for a quantitive means of assessing the goodness of fit. I should have made that explicit in my question. $\endgroup$
    – user28511
    Commented Jul 29, 2013 at 1:14
  • $\begingroup$ We should use both statistic (goodness of fit, BIC, AIC etc.) and visualizing, because we have certain problem in which statistic shows it is good fit but visualization of observed and expected is not close. Both gives strong evidence for a good fit. @whuber my answer is related after using a distribution and the link you provided is about pre selection of finding approximate distribution. $\endgroup$
    – SAAN
    Commented Jul 29, 2013 at 6:13
  • 1
    $\begingroup$ Good point, Azeem. On the other hand, you wouldn't want to weight visual fit too highly, per the concern of over-fitting the data. The beauty of BIC and AIC is they weight models by the number of parameters, and so can deem a model with a worse fit (discerned visually), but fewer parameters, the more likely source of the data. $\endgroup$
    – user28511
    Commented Jul 30, 2013 at 19:01
2
$\begingroup$

When computing the $r^2$ between the histogram and the continuous PDF, one should use the normalized values of the histogram (indicative of frequencies, such that the area of the histogram is 1) and the mean of the PDF in each bin (the integral over the range of each bin, normalized by the width of the bin). See this webpage for a description of how to obtain a continuous function's mean within a given range.

Note, however, that other statistics may be more appropriate for assessing goodness of fit between the observed distribution and the PDF. Two suggested in the comments to my question are the $\chi^2$ statistic (thanks, @AdamO) and the Kolmogorov-Smirnov (KS) test. Zhang and Luck (2008, 2009) calculated these in addition to $r^2$. Specifically, $r^2$ is inappropriate when the PDF being compared to the data is the uniform function, for the reasons @whuber states in one of his comments. @NickCox claims in one of his comments that all three of these measures are inappropriate for circular data.

@NickCox offers the Kuiper statistic as the best measure for testing the goodness of fit in the present scenario (thanks for that!), but claims it is unlikely to be available in software. This pessimism is unfounded: The KS test is available in the SciPy package for Python as scipy.stats.kstest (see the documentation here), and the results from this function can be used to calculate the Kuiper statistic (see this webpage for the latter statistic's definition). It appears at least one user of Python has written code that directly calculates the Kuiper statistic.

As most reviewers of scientific papers are unlikely to be satisfied with the use of a single measure of goodness of fit, I recommend using several, perhaps all, of the suggested statistics. Another option, vaguely alluded to by @NickCox, is to compare the fits of different models with something like the Bayesian Information Criterion (see van den Berg et al. [PNAS, 2012] for an example of its application) or the Akaike Information Criterion (see Fougnie et al. [Nat Commun, 2012]).

Improve this answer
$\endgroup$
3
  • $\begingroup$ I was referring to a Kuiper-type calculation comparing the observed distribution function and the continuous fitted distribution function integrated from the fitted density function. Is that what the Python code you've unearthed does? Using the means of observed and fitted density functions is at best only an indirect approximation to that. I am at a loss to see why you think any of my comments allude, vaguely or otherwise, to BIC or AIC; for the record, that was not being suggested by me. Other people replying or commenting on your question may want to add their own remarks. $\endgroup$
    – Nick Cox
    Commented Jul 28, 2013 at 0:38
  • $\begingroup$ "Using the means of observed and fitted density functions is at best only an indirect approximation to that." Please re-read my answer and you'll see that I never suggest calculating the mean of anything when computing the Kuiper statistic. My description of how to calculate the mean refers specifically to the calculation of $r^2$ per my original question. In an earlier comment you said "I just compare two or more model fits and see which is closer." The BIC and AIC are specific ways of making such a comparison. I don't think I have any more patience for this thread! $\endgroup$
    – user28511
    Commented Jul 28, 2013 at 3:25
  • 2
    $\begingroup$ You can follow through by accepting your answer formally; you will gain reputation points thereby. I've not studied the Python code you cite to see what it does to calculate a Kuiper statistic; I do understand that your discussion of calculations with bins refers only to an $r^2$ calculation. It appears that no discussant could accept all your premises here, let alone what was off-limits in discussion, but you get to be judge and jury on what answers your question. When I say "which is closer" I meant graphically; as said I don't favour single measures of fit much. $\endgroup$
    – Nick Cox
    Commented Jul 28, 2013 at 9:19
1
$\begingroup$

This is just a very simple test of calibration. By using the binning approach, you need not do any more than calculate the $\chi^2$ fit statistic and be done with it. It is very limited in scope of what it addresses, but you have not indicated whether you're interested in anything else based on your problem description.

To calculate expected frequencies, you integrate the continuous DF for your referent population distribution model over bounds you've defined by binning. The choice of which values to bin is very important indeed and should not be guided by $p$-values but rather meaningful cut points. The $\chi^2$ statistic is then $\chi^2_{(k-1)} = \sum_{i=1}^k \frac{\left(O_i - E_i\right)^2}{E_i}$ with $k$ being the number of bins.

Improve this answer
$\endgroup$
6
  • $\begingroup$ This is the right approach, but some consideration of the degrees of freedom is in order. A mixture of a Von Mises distribution and a uniform has three parameters; if all three are fit from the data and the bins are determined independently of the data, then the DF should be close to $k-3$ instead of $k-1$. I say "close to" instead of "equal to" because of the considerations described at stats.stackexchange.com/questions/16921/…. $\endgroup$
    – whuber ♦
    Commented Jul 26, 2013 at 19:46
  • $\begingroup$ @whuber, I've just read through the explanation at the link you posted, and I'm thinking it's important that I mention that I've arrived at my PDF parameters via maximum likelihood estimation, using the original observations, not binned counts. My goal is to get a measure of how well the MLE-derived parameters fit my distribution of observations, and I've chosen to construct the histogram and measure the fit of the model to the histogram because it's something I saw done in the papers I cited above. But maybe this is a bad idea? $\endgroup$
    – user28511
    Commented Jul 26, 2013 at 21:11
  • $\begingroup$ It's a good idea: use either the chi-squared test proposed by AdamO here or the K-S test. $\endgroup$
    – whuber ♦
    Commented Jul 26, 2013 at 21:20
  • $\begingroup$ The Kuiper procedure mentioned in my comments is the extension of Kolmogorov-Smirnov to circular set-ups. K-S is wrong in spirit. Chi-square is popular but pays no attention to circular scale. $\endgroup$
    – Nick Cox
    Commented Jul 26, 2013 at 21:27
  • 1
    $\begingroup$ Roughly, each term $(O-E)^2/E$ behaves (asymptotically in the maximum likelihood theory) like the square of a Standard Normal distribution. There are $k$ of them, so if they were independent their sum would approximately have a $\chi^2_{(k)}$ distribution (because that's what a Chi-squared distribution is!) and if they were subject to $p$ linearly independent linear restrictions, they would have a $\chi^2_{(k-p)}$ distribution (because such sums act like sum of $k-p$ independent squared Standard Normals). So, asymptotically, it is obvious what the degrees of freedom need to be. $\endgroup$
    – whuber ♦
    Commented Jul 28, 2013 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.