4
$\begingroup$

I want to fit a distribution to some data to sample from it in a subsequent simulation. There are I got a dataset that looks somehwat like this:

dat <- data.frame(num=1:10, value=c(6000, 2800, 1000, 230, 142, 53,
                                    32, 21, 10, 110))

Every entry of "num" refers to a non-negative discrete observation, except the last one (in this case num=10) which is a category of "10 or more". The column "value" refers to the frequency of that observation. The data is extremly right skewed, but an exponential distribution seems to be a decent fit if you ignore the last row of the data. (note: this isn't the real data)

I happen to know the maximum value that can occur, let's say that one is 500. So in other words I know the area under the curve between 10 and 500, which is the frequency of the last category, and have exact values before that. How do I fit an exponential (or other fitting) distribution to this data that satisfies these constraints?

So far I only managed to fit an exponential distribution to the first 9 columns and predict values for the subsequent values of 10-500 like this:

m.exp <- nls(value ~ I(a * exp(b * num)),
             data = dat, start = list(a = 1, b = 0), trace = T)

new_dat <- predict(m.exp, newdata = data.frame(num=c(9:500)))

This does work somewhat, but the area under the curve between 9 and 500 obviously is not correct (e.g. it is not equal to 110). Another solution I implemented is fitting a triangle, which does satisfy the area under curve and the maximum value, but the distribution would be far off from the prior distribution of values.

Edit: Is it possible to fit a distribution that has 112 (the frequency of "10 and more") as the area under the curve between the values 9 and 500, which roughly follows the same distribution as the values before it?

$\endgroup$
  • 1
    $\begingroup$ So the first vector is of observations, the last item being a bin for counts between 10 & a known upper bound of 500; the second of their frequency? The exponential distribution is for a continuous random variable with no upper bound. $\endgroup$ – Scortchi - Reinstate Monica Nov 29 '19 at 10:11
  • 1
    $\begingroup$ Exactly correct. I know that exponential distributions don't have an upper bound, what I am trying to ask is: Is it possible to fit a distribution that has 112 (the frequency of "10 and more") as the area under the curve between 9 and 500, which roughly follows the same distribution as the values before it? $\endgroup$ – friendlystatsguy Nov 29 '19 at 10:17
  • 5
    $\begingroup$ The general idea is maximum-likelihood fitting of censored observations - when you have a parametric distribution in mind, of course. See e.g. stats.stackexchange.com/q/133347/17230. $\endgroup$ – Scortchi - Reinstate Monica Nov 29 '19 at 10:52
  • 1
    $\begingroup$ Thank you for that reference. I see that this works for survival data, but it's unclear to me how I could use this for the present case. $\endgroup$ – friendlystatsguy Nov 29 '19 at 12:57
  • 1
    $\begingroup$ Take a look at the fistdistrplus package. $\endgroup$ – Dave2e Nov 29 '19 at 13:08
0
$\begingroup$

The data you showed seem to be quite well modelled by a geometric distribution, expect by the truncation (which seems to be expected).

dat <- data.frame(num=1:10, value=c(6000, 2800, 1000, 230, 142, 53,
                                                                        32, 21, 10, 110))

suppressPackageStartupMessages(library(tidyverse))


avg <- dat %>% summarise(avg = weighted.mean(num, value)) %>% pull(avg)

p <- 1/avg

dat <- dat %>% mutate(expected =  dgeom(num - 1 , prob=p), prop = value/sum(value))

dat %>% 
    gather(measure, value, expected, prop) %>% 
    ggplot(aes(x = num, value, fill=measure)) +
    geom_col(position = 'dodge') +
    labs(title = 'Expected and actual counts oberved for the num variable')

Created on 2019-11-29 by the reprex package (v0.3.0)

$\endgroup$
  • 1
    $\begingroup$ Could you explain how you determine the data are "quite well modeled" by a geometric distribution? Please don't forget that you must account for the above-10 group: if its count isn't very accurately reproduced by the fitted geometric distribution, then using that distribution to extrapolate into the above-10 group (in a simulation, for instance) would be highly suspect. $\endgroup$ – whuber Nov 29 '19 at 20:42
  • $\begingroup$ The data I posted here was completely arbitrary and does not really represent the data I actually have. I was just trying to illustrate the problem I am facing. I sadly can't share the actual data here. The truncation is the problem here. I can easily fit a distribution to the other values $\endgroup$ – friendlystatsguy Nov 29 '19 at 23:00
  • $\begingroup$ I was waiting for me to get home to perform the requested simulations. If these data are not representative of your case I can try and delete the answer. $\endgroup$ – Guilherme Marthe Nov 29 '19 at 23:07
  • $\begingroup$ If you can show me how you wanted to draw values from the "10 or more" category that might still be very usefull to me. If you wanted to ignore that part I probably failed to formulate the question correctly. Either way: Thanks for your help! $\endgroup$ – friendlystatsguy Nov 29 '19 at 23:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.