1
$\begingroup$

Disclaimer: I'm not a statistician but a software engineer. Most of my knowledge in statistics comes from self-education, thus I still have many gaps in understanding concepts that may seem trivial for other people here.

I would like to fit the following data, as a beta distribution if that makes sense, but I am getting the following error:

x = c(0.038, 0.017, 0.08, 0.013, 0.01, 0.031, 0.021, 0.029, 0.02, 
0.005, 0.013, 0.027, 0.019, 0, 0.042, 0.02, 0.016, 0.004, 0.022, 
0.003, 0.022, 0.025, 0.043, 0.033, 0.021, 0.006, 0.009, 0.031, 
0.006, 0.037, 0.035, 0.015, 0.028, 0.018, 0.014, 0.012, 0.022, 
0.026, 0.08, 0.034, 0.007, 0.018, 0.02, 0.03, 0.045, 0.026, 0.009, 
0.012, 0.01, 0.012, 0.02, 0.019, 0.019, 0.007, 0.024, 0.008, 
0.031, 0.028, 0.017, 0.011, 0.039, 0.012, 0.03, 0.002, 0.027, 
0.021, 0.003, 0.057, 0.019, 0.025, 0.007, 0.021, 0.004, 0.027, 
0.013, 0.004, 0.01, 0.031, 0.009, 0.045, 0.008, 0.013, 0.02, 
0.008, 0.024, 0.013, 0.007, 0.015, 0.048, 0.025, 0.047, 0.027, 
0.025, 0.023, 0.007, 0.018, 0.023, 0.014, 0.024, 0.021, 0.007, 
0.021, 0.005, 0.008, 0.029, 0.026, 0.002, 0.021, 0.001, 0.001, 
0.026, 0.025, 0.008, 0.004, 0.005, 0, 0.01, 0.045, 0.004, 0.035, 
0.038, 0.02, 0.015, 0.035, 0.028, 0.027, 0.042, 0.034, 0.028, 
0.024, 0.019, 0.033, 0.033, 0.033, 0.014, 0.026, 0.012, 0.019, 
0.035, 0.019, 0.017, 0.005, 0.015, 0.024, 0.044, 0.008, 0.011, 
0.012, 0.031, 0.01, 0.076, 0.019, 0.035, 0.003, 0.041, 0.023, 
0.005, 0.038, 0.013, 0.005, 0.043, 0.01, 0, 0.026, 0.009, 0.015, 
0.014, 0.023, 0.021, 0.005, 0.002, 0.006, 0.003, 0.014, 0.057, 
0.054, 0.017, 0.031, 0.063, 0.02, 0.006, 0.02, 0.045, 0.035, 
0.013, 0.009, 0.019, 0.033, 0.028, 0.018, 0.012, 0.007, 0, 0.023, 
0.04, 0.009, 0.039, 0.021, 0.006, 0.019)
m <- MASS::fitdistr(x, dbeta, start = list(shape1 = 1, shape2 = 10))
Error in stats::optim(x = c(0.038, 0.017, 0.08, 0.013, 0.01, 0.031, 0.021,  : 
  non-finite finite-difference value [1]

I am not sure that a beta distribution makes sense. Here is a plot of the distribution (histogram) of the whole data (above is just a sample):

enter image description here

Can this be fitted with a beta distribution? And if yes, how?

The reason I want to fit it as a beta distribution is to be able to use this distribution as a prior for a Bayesian estimate as explained here.

Improve this question
$\endgroup$
3
  • $\begingroup$ Some of your data values are equal to 0, which isn't valid for the beta distribution. They must be in $(0,1)$. Either make them small, say, equal to 0.0005 (half the three-digit accuracy you have) or remove them; the former is probably better. $\endgroup$
    – jbowman
    Commented May 2, 2016 at 21:04
  • $\begingroup$ First do what @jbowman suggests. The zero data points may manifest themselves in the optimizer trying negative parameter values, resulting in your error. You should use the 'lower' option to fitdistr to specify a lower bound (perhaps 1e-6 or so) to keep the optimizer from venturing into negative parameter values. Note that the worse the fit, the more difficulty you may encounter in the fitting process, as for instance due to your zeros. If fitdistr were written in a smarter way, lower bounds would be provided to the optimizer on parameters which must satisfy them, as with beta distribution. $\endgroup$
    – Mark L. Stone
    Commented May 2, 2016 at 21:16
  • 1
    $\begingroup$ This isn't even remotely like any Beta distribution. (Those with nonzero limiting density at zero will have a mode at zero.) What is the reason you want to fit a Beta to these data? Or fit any mathematical distribution, for that matter? $\endgroup$
    – whuber ♦
    Commented May 2, 2016 at 21:36

1 Answer 1

Reset to default
1
$\begingroup$

These data are not at all close to a beta -- beta distributions don't "squish up" at the end unless the density is monotonic. If the mode is not at the end, then there should be zero density there.

You also have exact zeros.

Assuming the values themselves are in fact a reasonable source for a prior, and assuming you really want to use beta priors, one possibility you could consider is a mixture of beta priors (plus, presumably a component for the atom at 0)

The nice thing about mixtures (besides their flexibility) is that conjugacy carries through -- you end up with a corresponding mixture of posteriors in the same proportions.

Leaving aside the proportion of exact zeros for a second, for the rest of it you might get a more-or-less adequate approximation from a 2- or 3-component mixture of betas. Note that if you carry through the degenerate component, its component of the posterior will be degenerate.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.