Skip to main content
added 355 characters in body
Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k

You have interval censoredinterval censored data (as you seem to be aware).

https://en.wikipedia.org/wiki/Censoring_(statistics)#Types

You could fit either distribution by maximum likelihood; it's a matter of writing the likelihood for the censored data and maximizing it (see the section on likelihood at that page above). Obvious start values to use would be the ML estimates you get by just placing all the data at the center of each interval. The full ML estimates will be very likely to only differ slightly from those so convergence should be only a step or two.

The likelihood could be written in grouped form using the counts, so grouped or ungrouped should not be a big issue.

If one observation is between $b_i$ and $b_{i+1}$ (with $b_{i+1}>b_i$), its likelihood is $p_i=F(b_{i+1})-F(b_i)$ (the cdf, $F$ is a function of the parameters). You can write the likelihood for all the observations from the multinomial, in terms of those $p_i$ values and the counts in each bin ($n_i$), and so in turn as a function of the parameters. Note that if you're just optimizing, the combinatorial term in the multinomial is just an additive constant in the log-likelihood, so you could optimize $\sum_i n_i\log(p_i(\theta))$, where $\theta$ is the vector of parameters.

You will then need some optimizer (some will require derivatives, some will not). One specifically designed for ML estimation (there are some in R for example), should give you standard errors as well.

Alternatively to this direct approach, you may be able to use parametric survival regression models to get ML estimates without even having to write that censored likelihood at all (at the cost of having some small additional setup for the data). Not every implementation of such models will work with just the counts in each bin though; some may assume you have individual data (though it shouldn't be hard to check whatever implementations you have access to).


If you would be happy with an approximate (asymptotically efficient) answer, it's simple enough to write the goodness of fit for either model in terms of a chi-squared statistic and use minimum chi-squared estimation.

https://en.wikipedia.org/wiki/Minimum_chi-square_estimation

It works directly with the counts, so it's already grouped in the way you want.

Berkson's paper (which is listed in the Wikipedia article) does a good job of explaining its usefulness and simplicity; in this instance you will still end up performing the same task - calling some optimizer to take you from the initial estimates to the 'optimized' values.

If you replace that multinomial Pearson chi-squared statistic with the multinomial G-statistic you should get back to to the interval-censored ML estimates.


None of this is to suggest that either model will be a good choice (i.e. will 'fit' these data).

One thing that seems a particular issue is you seem to have a possible detection threshold at the left end (no values between 0 and 0.125, though the detection threshold may differ from 0.125); this will tend to make any model like the Weibull, lognormal, gamma, etc unsuitable unless you account for that apparent truncation.

You have interval censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)#Types

You could fit either distribution by maximum likelihood; it's a matter of writing the likelihood for the censored data and maximizing it (see the section on likelihood at that page above). Obvious start values to use would be the ML estimates you get by just placing all the data at the center of each interval. The full ML estimates will be very likely to only differ slightly from those so convergence should be only a step or two.

The likelihood could be written in grouped form using the counts, so grouped or ungrouped should not be a big issue.

If one observation is between $b_i$ and $b_{i+1}$ (with $b_{i+1}>b_i$), its likelihood is $p_i=F(b_{i+1})-F(b_i)$ (the cdf, $F$ is a function of the parameters). You can write the likelihood for all the observations from the multinomial, in terms of those $p_i$ values and the counts in each bin ($n_i$), and so in turn as a function of the parameters. Note that if you're just optimizing, the combinatorial term in the multinomial is just an additive constant in the log-likelihood, so you could optimize $\sum_i n_i\log(p_i(\theta))$, where $\theta$ is the vector of parameters.

You will then need some optimizer (some will require derivatives, some will not). One specifically designed for ML estimation (there are some in R for example), should give you standard errors as well.

Alternatively to this direct approach, you may be able to use parametric survival regression models to get ML estimates without even having to write that censored likelihood at all (at the cost of having some small additional setup for the data). Not every implementation of such models will work with just the counts in each bin though; some may assume you have individual data (though it shouldn't be hard to check whatever implementations you have access to).


If you would be happy with an approximate (asymptotically efficient) answer, it's simple enough to write the goodness of fit for either model in terms of a chi-squared statistic and use minimum chi-squared estimation.

https://en.wikipedia.org/wiki/Minimum_chi-square_estimation

It works directly with the counts, so it's already grouped in the way you want.

Berkson's paper (which is listed in the Wikipedia article) does a good job of explaining its usefulness and simplicity; in this instance you will still end up performing the same task - calling some optimizer to take you from the initial estimates to the 'optimized' values.

If you replace that multinomial Pearson chi-squared statistic with the multinomial G-statistic you should get back to to the interval-censored ML estimates.


None of this is to suggest that either model will be a good choice (i.e. will 'fit' these data).

You have interval censored data (as you seem to be aware).

https://en.wikipedia.org/wiki/Censoring_(statistics)#Types

You could fit either distribution by maximum likelihood; it's a matter of writing the likelihood for the censored data and maximizing it (see the section on likelihood at that page above). Obvious start values to use would be the ML estimates you get by just placing all the data at the center of each interval. The full ML estimates will be very likely to only differ slightly from those so convergence should be only a step or two.

The likelihood could be written in grouped form using the counts, so grouped or ungrouped should not be a big issue.

If one observation is between $b_i$ and $b_{i+1}$ (with $b_{i+1}>b_i$), its likelihood is $p_i=F(b_{i+1})-F(b_i)$ (the cdf, $F$ is a function of the parameters). You can write the likelihood for all the observations from the multinomial, in terms of those $p_i$ values and the counts in each bin ($n_i$), and so in turn as a function of the parameters. Note that if you're just optimizing, the combinatorial term in the multinomial is just an additive constant in the log-likelihood, so you could optimize $\sum_i n_i\log(p_i(\theta))$, where $\theta$ is the vector of parameters.

You will then need some optimizer (some will require derivatives, some will not). One specifically designed for ML estimation (there are some in R for example), should give you standard errors as well.

Alternatively to this direct approach, you may be able to use parametric survival regression models to get ML estimates without even having to write that censored likelihood at all (at the cost of having some small additional setup for the data). Not every implementation of such models will work with just the counts in each bin though; some may assume you have individual data (though it shouldn't be hard to check whatever implementations you have access to).


If you would be happy with an approximate (asymptotically efficient) answer, it's simple enough to write the goodness of fit for either model in terms of a chi-squared statistic and use minimum chi-squared estimation.

https://en.wikipedia.org/wiki/Minimum_chi-square_estimation

It works directly with the counts, so it's already grouped in the way you want.

Berkson's paper (which is listed in the Wikipedia article) does a good job of explaining its usefulness and simplicity; in this instance you will still end up performing the same task - calling some optimizer to take you from the initial estimates to the 'optimized' values.

If you replace that multinomial Pearson chi-squared statistic with the multinomial G-statistic you should get back to to the interval-censored ML estimates.


None of this is to suggest that either model will be a good choice (i.e. will 'fit' these data).

One thing that seems a particular issue is you seem to have a possible detection threshold at the left end (no values between 0 and 0.125, though the detection threshold may differ from 0.125); this will tend to make any model like the Weibull, lognormal, gamma, etc unsuitable unless you account for that apparent truncation.

added clarifying detail, plus some minor clarifying edits
Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k

Let me leave aside your wish to keep the data 'grouped' for the moment (I will return to it).

You have interval censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)#Types

You could fit either distribution by maximum likelihood; it's a matter of writing the likelihood for the censored data and maximizing it (see the section on likelihood at that page above). Obvious start values to use would be the ML estimates you get by just placing all the data at the center of each interval. The full ML estimates will be very likely to only differ slightly from those so convergence should be only a step or two.

The likelihood could be written in grouped form using the counts, so grouped or ungrouped should not be a big issue.

If one observation is between $b_i$ and $b_{i+1}$ (with $b_{i+1}>b_i$), its likelihood is $p_i=F(b_{i+1})-F(b_i)$ (the cdf, $F$ is a function of the parameters). You can write the likelihood for all the observations from the multinomial, in terms of those $p_i$ values and the counts in each bin ($n_i$), and so in turn as a function of the parameters. Note that if you're just optimizing, the combinatorial term in the multinomial is just an additive constant in the log-likelihood, so you could optimize $\sum_i n_i\log(p_i(\theta))$, where $\theta$ is the vector of parameters.

You will then need some optimizer (some will require derivatives, some will not). One specifically designed for ML estimation (there are some in R for example), should give you standard errors as well.

Alternatively to this direct approach, you may be able to use parametric survival regression models to do get ML estimates without even having to write that censored likelihood at all (at the cost of having some small additional setup for the data). Not every implementation of such models will work with just the counts in each bin though; some may assume you have individual data (though it shouldn't be hard to check whatever implementations you have access to).


If you would be happy with an approximate (asymptotically efficient) answer, it's simple enough to write the goodness of fit for either model in terms of a chi-squared statistic and use minimum chi-squared estimation.

https://en.wikipedia.org/wiki/Minimum_chi-square_estimation

It works directly with the counts, so it's already grouped in the way you want.

Berkson's paper (which is listed in the Wikipedia article) does a good job of explaining its usefulness and simplicity; in this instance you will still end up performing the same task - calling some optimizer to take you from the initial estimates to the 'optimized' values.

If you replace that multinomial Pearson chi-squared statistic with the multinomial G-statistic you should get back to to the interval-censored ML estimates.


None of this is to suggest that either model will be a good choice (i.e. will 'fit' these data).

Let me leave aside your wish to keep the data 'grouped' for the moment (I will return to it).

You have interval censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)#Types

You could fit either distribution by maximum likelihood; it's a matter of writing the likelihood for the censored data and maximizing it (see the section on likelihood at that page above). Obvious start values to use would be the ML estimates you get by just placing all the data at the center of each interval. The full ML estimates will be very likely to only differ slightly from those so convergence should be only a step or two.

The likelihood could be written in grouped form using the counts, so grouped or ungrouped should not be a big issue.

If one observation is between $b_i$ and $b_{i+1}$ (with $b_{i+1}>b_i$), its likelihood is $p_i=F(b_{i+1})-F(b_i)$ (the cdf, $F$ is a function of the parameters). You can write the likelihood for all the observations from the multinomial, in terms of those $p_i$ values and the counts in each bin ($n_i$), and so in turn as a function of the parameters. Note that if you're just optimizing, the combinatorial term in the multinomial is just an additive constant in the log-likelihood, so you could optimize $\sum_i n_i\log(p_i(\theta))$, where $\theta$ is the vector of parameters.

You will then need some optimizer (some will require derivatives, some will not). One specifically designed for ML estimation (there are some in R for example), should give you standard errors as well.

Alternatively to this direct approach, you may be able to use parametric survival regression models to do get ML estimates without even having to write that censored likelihood at all (at the cost of having some small additional setup for the data). Not every implementation of such models will work with just the counts in each bin though; some may assume you have individual data (though it shouldn't be hard to check whatever implementations you have access to).


If you would be happy with an approximate (asymptotically efficient) answer, it's simple enough to write the goodness of fit for either model in terms of a chi-squared statistic and use minimum chi-squared estimation.

https://en.wikipedia.org/wiki/Minimum_chi-square_estimation

It works directly with the counts, so it's already grouped in the way you want.

Berkson's paper (which is listed in the Wikipedia article) does a good job of explaining its usefulness and simplicity; in this instance you will still end up performing the same task - calling some optimizer to take you from the initial estimates to the 'optimized' values.

If you replace that multinomial Pearson chi-squared statistic with the multinomial G-statistic you should get back to to the interval-censored ML estimates.

You have interval censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)#Types

You could fit either distribution by maximum likelihood; it's a matter of writing the likelihood for the censored data and maximizing it (see the section on likelihood at that page above). Obvious start values to use would be the ML estimates you get by just placing all the data at the center of each interval. The full ML estimates will be very likely to only differ slightly from those so convergence should be only a step or two.

The likelihood could be written in grouped form using the counts, so grouped or ungrouped should not be a big issue.

If one observation is between $b_i$ and $b_{i+1}$ (with $b_{i+1}>b_i$), its likelihood is $p_i=F(b_{i+1})-F(b_i)$ (the cdf, $F$ is a function of the parameters). You can write the likelihood for all the observations from the multinomial, in terms of those $p_i$ values and the counts in each bin ($n_i$), and so in turn as a function of the parameters. Note that if you're just optimizing, the combinatorial term in the multinomial is just an additive constant in the log-likelihood, so you could optimize $\sum_i n_i\log(p_i(\theta))$, where $\theta$ is the vector of parameters.

You will then need some optimizer (some will require derivatives, some will not). One specifically designed for ML estimation (there are some in R for example), should give you standard errors as well.

Alternatively to this direct approach, you may be able to use parametric survival regression models to get ML estimates without even having to write that censored likelihood at all (at the cost of having some small additional setup for the data). Not every implementation of such models will work with just the counts in each bin though; some may assume you have individual data (though it shouldn't be hard to check whatever implementations you have access to).


If you would be happy with an approximate (asymptotically efficient) answer, it's simple enough to write the goodness of fit for either model in terms of a chi-squared statistic and use minimum chi-squared estimation.

https://en.wikipedia.org/wiki/Minimum_chi-square_estimation

It works directly with the counts, so it's already grouped in the way you want.

Berkson's paper (which is listed in the Wikipedia article) does a good job of explaining its usefulness and simplicity; in this instance you will still end up performing the same task - calling some optimizer to take you from the initial estimates to the 'optimized' values.

If you replace that multinomial Pearson chi-squared statistic with the multinomial G-statistic you should get back to to the interval-censored ML estimates.


None of this is to suggest that either model will be a good choice (i.e. will 'fit' these data).

added clarifying detail, plus some minor clarifying edits
Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k

Let me leave aside your wish to keep the data 'grouped' for the moment (I will return to it).

You have interval censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)#Types

You could fit either distribution by maximum likelihood; it's a matter of writing the likelihood for the censored data and maximizing it (see the section on likelihood at that page above). Obvious start values to use would be the ML estimates you get by just placing all the data at the center of each interval. The full ML estimates will be very likely to only differ slightly from those so convergence should be only a step or two.

The likelihood could be written in grouped form using the counts, so grouped or ungrouped should not be a big issue.

If one observation is between $b_i$ and $b_{i+1}$ (with $b_{i+1}>b_i$), its likelihood is $p_i=F(b_{i+1})-F(b_i)$ (the cdf, $F$ is a function of the parameters). You can write the likelihood for all the observations from the multinomial, in terms of those $p_i$ values and the counts in each bin ($n_i$), and so in turn as a function of the parameters. Note that if you're just optimizing, the combinatorial term in the multinomial is just an additive constant in the log-likelihood, so you could optimize $\sum_i n_i\log(p_i(\theta))$, where $\theta$ is the vector of parameters.

You will then need some optimizer (some will require derivatives, some will not). One specifically designed for ML estimation (there are some in R for example), should give you standard errors as well.

Alternatively to this direct approach, you may be able to use parametric survival regression models to do get ML estimates without even having to write that censored likelihood at all (at the cost of having some small additional setup for the data). Not every implementation of such models will work with just the counts in each bin though; some may assume you have individual data (though it shouldn't be hard to check whatever implementations you have access to).


If you would be happy with an approximate (asymptotically efficient) answer, it's simple enough to write the goodness of fit for either model in terms of a chi-squared statistic and use minimum chi-squared estimation.

https://en.wikipedia.org/wiki/Minimum_chi-square_estimation

It works directly with the counts, so it's 'inherently'already grouped in the way you want.

Berkson's paper (which is listed in the Wikipedia article) does a good job of explaining its usefulness and simplicitysimplicity; in this instance you will still end up performing the same task - calling some optimizer to take you from the initial estimates to the 'optimized' values.

If you replace that multinomial Pearson chi-squared statistic with the multinomial G-statistic I think you should get back to to the interval-censored ML estimates.

Let me leave aside your wish to keep the data 'grouped' for the moment (I will return to it).

You have interval censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)#Types

You could fit either distribution by maximum likelihood; it's a matter of writing the likelihood for the censored data and maximizing it (see the section on likelihood at that page above). Obvious start values to use would be the ML estimates you get by just placing all the data at the center of each interval. The full ML estimates will be very likely to only differ slightly from those so convergence should be only a step or two.

The likelihood could be written in grouped form using the counts, so grouped or ungrouped should not be a big issue.

If one observation is between $b_i$ and $b_{i+1}$ (with $b_{i+1}>b_i$), its likelihood is $p_i=F(b_{i+1})-F(b_i)$ (the cdf, $F$ is a function of the parameters). You can write the likelihood for all the observations from the multinomial, in terms of those $p_i$ values and the counts in each bin ($n_i$), and so in turn as a function of the parameters.

You will then need some optimizer (some will require derivatives, some will not). One specifically designed for ML estimation (there are some in R for example), should give you standard errors as well.

Alternatively to this direct approach, you may be able to use parametric survival regression models to do get ML estimates without even having to write that censored likelihood at all (at the cost of having some small additional setup for the data). Not every implementation of such models will work with just the counts in each bin though; some may assume you have individual data (though it shouldn't be hard to check whatever implementations you have access to).


If you would be happy with an approximate (asymptotically efficient) answer, it's simple enough to write the goodness of fit for either model in terms of a chi-squared statistic and use minimum chi-squared estimation.

https://en.wikipedia.org/wiki/Minimum_chi-square_estimation

It works directly with the counts, so it's 'inherently' grouped in the way you want.

Berkson's paper (which is listed in the Wikipedia article) does a good job of explaining its usefulness and simplicity.

If you replace that Pearson chi-squared with the G-statistic I think you should get back to to the interval-censored ML estimates.

Let me leave aside your wish to keep the data 'grouped' for the moment (I will return to it).

You have interval censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)#Types

You could fit either distribution by maximum likelihood; it's a matter of writing the likelihood for the censored data and maximizing it (see the section on likelihood at that page above). Obvious start values to use would be the ML estimates you get by just placing all the data at the center of each interval. The full ML estimates will be very likely to only differ slightly from those so convergence should be only a step or two.

The likelihood could be written in grouped form using the counts, so grouped or ungrouped should not be a big issue.

If one observation is between $b_i$ and $b_{i+1}$ (with $b_{i+1}>b_i$), its likelihood is $p_i=F(b_{i+1})-F(b_i)$ (the cdf, $F$ is a function of the parameters). You can write the likelihood for all the observations from the multinomial, in terms of those $p_i$ values and the counts in each bin ($n_i$), and so in turn as a function of the parameters. Note that if you're just optimizing, the combinatorial term in the multinomial is just an additive constant in the log-likelihood, so you could optimize $\sum_i n_i\log(p_i(\theta))$, where $\theta$ is the vector of parameters.

You will then need some optimizer (some will require derivatives, some will not). One specifically designed for ML estimation (there are some in R for example), should give you standard errors as well.

Alternatively to this direct approach, you may be able to use parametric survival regression models to do get ML estimates without even having to write that censored likelihood at all (at the cost of having some small additional setup for the data). Not every implementation of such models will work with just the counts in each bin though; some may assume you have individual data (though it shouldn't be hard to check whatever implementations you have access to).


If you would be happy with an approximate (asymptotically efficient) answer, it's simple enough to write the goodness of fit for either model in terms of a chi-squared statistic and use minimum chi-squared estimation.

https://en.wikipedia.org/wiki/Minimum_chi-square_estimation

It works directly with the counts, so it's already grouped in the way you want.

Berkson's paper (which is listed in the Wikipedia article) does a good job of explaining its usefulness and simplicity; in this instance you will still end up performing the same task - calling some optimizer to take you from the initial estimates to the 'optimized' values.

If you replace that multinomial Pearson chi-squared statistic with the multinomial G-statistic you should get back to to the interval-censored ML estimates.

added 917 characters in body
Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k
Loading
added 917 characters in body
Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k
Loading
added 917 characters in body
Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k
Loading
Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k
Loading