2
$\begingroup$

I am trying to model data that looks like this:

Data I am trying to graph

The data reflects frequencies of items at 0, 25, 50, 75, and 100. The majority of the time, values are concentrated at the 0, and 100 points. I believe a beta distribution would be the best model for this. I am having trouble trying to calculate the alpha and beta values to fit this data. Any advice on how to do this would be appreciated.

Edit: For some more context, the data points of 0, 25, 50, 75, 100, reflect continuous data points. To give some more perspective, let's say those values reflect percentages, 0% to 100%. We can assign it to an online course, where many students may sign-up, but never touch the course, resulting in many 0% users, and many end up finishing the course as well (100%).

$\endgroup$
  • $\begingroup$ To clarify: the only values the data take on are 0, 25, 50, 75, and 100, correct? $\endgroup$ – Sheridan Grant Aug 9 '19 at 23:08
  • $\begingroup$ @SheridanGrant - Yes, that is correct. I do want to mention that those values are reflecting continuous data though. The system which records those values don't store values any more granular than 0, 25, 50, 75, 100. $\endgroup$ – Shazbots Aug 10 '19 at 21:35
6
$\begingroup$

If the only values the data take on are $\{0,25,50,75,100\}$ then the Beta isn't an appropriate model since it's continuous and the data are highly discrete. If this is the case, and you don't want to make assumptions about the distribution, then just use the empirical distribution (i.e. if the data are $X_1,\ldots,X_n$, then $\hat{p}(k) = \frac{1}{n} \sum_{i=1}^n \mathbb{1}(X_i = k)$) and a bootstrap to assess confidence intervals/sets since your sample size is reasonable but not huge.

If the data take on continuous values from 0 to 100, or integers $0,\ldots,100$, then a Beta may be a good model. Divide all the data by 100 so that they lie in $[0,1]$, and then find the Beta MLE (this R package will do it for you, for example). In the analysis after this, just multiply predictions, means, standard errors, etc. by 100 to get back to the original data scale.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ could also use method of moments ... $\endgroup$ – Ben Bolker Aug 10 '19 at 0:29
  • $\begingroup$ I want to mention that the real data is continuous, from 0 to 100%. The values that are stored in the system are 0, 25, 75, 100. That's why I think beta is an acceptable model. (it's kinda funny that I'm going from continuous data, saved as discrete data, and I want to "uncompress" it back to continuous again.) For example: let's say the 0 to 100% represents the competition percentage of an online course. A lot of people register and don't ever take the course (0%-ers), and a good number of people complete the course (100%-ers). There are fewer people who quit in the middle. $\endgroup$ – Shazbots Aug 10 '19 at 21:36
  • $\begingroup$ Good answer (+1), but I think you can just say "discrete"; there is no "highly discrete". $\endgroup$ – Ben Aug 10 '19 at 23:28
  • $\begingroup$ @Shazbots that's unfortunate that the system isn't storing the true percentages. If you had those percentages, then the Beta could probably be pretty useful, but unfortunately it is NOT good for your data, even if the underlying data are continuous-valued, because your data are highly discrete. Ben, I say "highly discrete" because clearly with only 5 possible values a continuous distribution is a bad idea, whereas if the data were integer percents (101 possible values) then the continuous approximation would be acceptable. $\endgroup$ – Sheridan Grant Aug 12 '19 at 6:14
2
$\begingroup$

I do not disagree with @SheridanGrant's Answer. However, I will directly address your question about using a beta distribution. Even if you are only observing values "rounded" (or otherwise transformed) to $0, .25, .50, .75, 1,$ there may be a beta distribution determining the probabilities with which these values are observed.

Following up on @BenBolker's Comment about estimating beta parameters by the method of moments, I digitized (approximately) the counts in the histograms you provided, to obtain $\mu \approx 0.5$ and $\sigma^2 \approx 0.2.$

Because $\mu = \alpha/(\alpha + \beta),$ we observe that $\mu = 0.4$ imples $\alpha = \beta.$ Then because $\sigma^2 = \frac{\alpha\beta}{(\alpha + \beta)^2( \alpha+\beta+1)}$ we can set $\alpha = \beta$ to get $\alpha=\beta=1/8).$

We can quickly verify this in R by looking at a million observations from $\mathsf{Beta}(1/8, 1/8),$ to get $\mu \approx 1/2, \sigma^2 \approx 0.2.$ (With a million observations, we can expect 2 or 3-place accuracy.) Even if I didn't digitize your 'data' correctly, the same method would work with the correct data.

x = rbeta(10^6,1/8,1/8)
mean(x); var(x)
[1] 0.5003867
[1] 0.1999466

Then looking at a smaller sample $(n = 500)$ from this beta distribution we get a histogram that is not much different than the ones you show.

set.seed(2019)  # for reproducibility
x = rbeta(500, 1/8, 1/8)
hist(x, br=5, prob=T, ylim=c(0,3), col="skyblue2", label=T)
  curve(dbeta(x,1/8,1/8), add=T, col="red", n=10001)

enter image description here

Because the width of each bin in this histogram is 0.2, we can multiply the 'density' labels atop the histogram bars by 0.2 to get the probability represented by each bar (for a total of $1).$

Theoretical probabilities (to 4 places) in these five intervals are as follows:

round(diff(pbeta(c(0,.2,.4,.6,.8,1), 1/8, 1/8)), 4)
[1] 0.4269 0.0513 0.0435 0.0513 0.4269

With sample sizes of only $n = 100$ observations, results are more variable as illustrated by the four histograms below.

enter image description here

Note: The density function of $\mathsf{Beta}(1/8, 1/8)$ has vertical asymptotes at $0$ and $1.$ So there is more area under the density curve near $0$ and $1$ than may be apparent in the plots above.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You've just fit a Beta to the data with MoM and then discretized the results, right? I agree that the results look plausible but I would still hesitate to use the Beta. $\endgroup$ – Sheridan Grant Aug 12 '19 at 6:18
  • $\begingroup$ @SheridanGrant. Right about MoM, but I did not discretize beta samples, just showed histograms with bins matching those in the Question. $\endgroup$ – BruceET Aug 12 '19 at 7:02
  • 1
    $\begingroup$ This is an interesting approach. Your examples suggest it may have a tendency to estimate overly extreme values of the parameters (closer to zero for both), which is what one might expect from a method that relies on the moments. Perhaps a good use of your solution would be for constructing a starting solution for an MLE. $\endgroup$ – whuber Aug 12 '19 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.