Skip to main content
edited body
Source Link
Ben
  • 133k
  • 7
  • 255
  • 588

For many one-parameter probability distributions, the variance in the distribution is a function of the mean. When you fit data to a statistical model using these distributions, the estimator will tend to give you a reasonable estimate of the mean, but the estimated variance will just be a function of that, so it will not generally fit to the data very well. This happens with certain one-parameter probability distributions, most notably the Poisson distribution. In this case, it is common for the data to be more variable than the estimated variance coming out of your model, in which case we say that there is a problem of "overdispersion".

Both of the descriptions you have given for this are correct. Overdispersion is indeed the presence of greater variability in the data than predicted by the model. This generally occurs because the variance in the distribution used in the model is a function of the mean, so the estimation procedure can't estimate them both well (and mean estimation is generally more important than variance estimation when fitting data to a model).

Roughly speaking, if you have $k$ parmaters in a statistical distribution, and you fit it to data, it will allow you to accurately estimate $k$ moments of the distribution (often the first $k$ moments, but not always$^\dagger$). So, for example, some one-parameter distributions allow you to accurately estimate the mean but not the variance, some two parameter-parameter distributions allow you to accurately estimate the mean and variance but not the skewness, some three parameter-parameter distributions allow you to accurately estimate the mean, variance and skewness, but not the kurtosis, and so on.

If you want to avoid overdispersion in your modelling, you should use statistical models that use an underlying two-parameter distribution that can fit the mean and variance (e.g., use a negative binomial model instead of a Poisson model). The same basic principle also applies if you want to accurately fit higher-order moments --- e.g., if you want to accurately fit skewness you might generalise to a three-parameter distribution, and so on.


$^\dagger$ For example, the Student's T-distribution has a single parameter that affects the variance and kurtosis but not the mean or skewness.

For many one-parameter probability distributions, the variance in the distribution is a function of the mean. When you fit data to a statistical model using these distributions, the estimator will tend to give you a reasonable estimate of the mean, but the estimated variance will just be a function of that, so it will not generally fit to the data very well. This happens with certain one-parameter probability distributions, most notably the Poisson distribution. In this case, it is common for the data to be more variable than the estimated variance coming out of your model, in which case we say that there is a problem of "overdispersion".

Both of the descriptions you have given for this are correct. Overdispersion is indeed the presence of greater variability in the data than predicted by the model. This generally occurs because the variance in the distribution used in the model is a function of the mean, so the estimation procedure can't estimate them both well (and mean estimation is generally more important than variance estimation when fitting data to a model).

Roughly speaking, if you have $k$ parmaters in a statistical distribution, and you fit it to data, it will allow you to accurately estimate $k$ moments of the distribution (often the first $k$ moments, but not always$^\dagger$). So, for example, some one-parameter distributions allow you to accurately estimate the mean but not the variance, some two parameter distributions allow you to accurately estimate the mean and variance but not the skewness, some three parameter distributions allow you to accurately estimate the mean, variance and skewness, but not the kurtosis, and so on.

If you want to avoid overdispersion in your modelling, you should use statistical models that use an underlying two-parameter distribution that can fit the mean and variance (e.g., use a negative binomial model instead of a Poisson model). The same basic principle also applies if you want to accurately fit higher-order moments --- e.g., if you want to accurately fit skewness you might generalise to a three-parameter distribution, and so on.


$^\dagger$ For example, the Student's T-distribution has a single parameter that affects the variance and kurtosis but not the mean or skewness.

For many one-parameter probability distributions, the variance in the distribution is a function of the mean. When you fit data to a statistical model using these distributions, the estimator will tend to give you a reasonable estimate of the mean, but the estimated variance will just be a function of that, so it will not generally fit to the data very well. This happens with certain one-parameter probability distributions, most notably the Poisson distribution. In this case, it is common for the data to be more variable than the estimated variance coming out of your model, in which case we say that there is a problem of "overdispersion".

Both of the descriptions you have given for this are correct. Overdispersion is indeed the presence of greater variability in the data than predicted by the model. This generally occurs because the variance in the distribution used in the model is a function of the mean, so the estimation procedure can't estimate them both well (and mean estimation is generally more important than variance estimation when fitting data to a model).

Roughly speaking, if you have $k$ parmaters in a statistical distribution, and you fit it to data, it will allow you to accurately estimate $k$ moments of the distribution (often the first $k$ moments, but not always$^\dagger$). So, for example, some one-parameter distributions allow you to accurately estimate the mean but not the variance, some two-parameter distributions allow you to accurately estimate the mean and variance but not the skewness, some three-parameter distributions allow you to accurately estimate the mean, variance and skewness, but not the kurtosis, and so on.

If you want to avoid overdispersion in your modelling, you should use statistical models that use an underlying two-parameter distribution that can fit the mean and variance (e.g., use a negative binomial model instead of a Poisson model). The same basic principle also applies if you want to accurately fit higher-order moments --- e.g., if you want to accurately fit skewness you might generalise to a three-parameter distribution, and so on.


$^\dagger$ For example, the Student's T-distribution has a single parameter that affects the variance and kurtosis but not the mean or skewness.

Source Link
Ben
  • 133k
  • 7
  • 255
  • 588

For many one-parameter probability distributions, the variance in the distribution is a function of the mean. When you fit data to a statistical model using these distributions, the estimator will tend to give you a reasonable estimate of the mean, but the estimated variance will just be a function of that, so it will not generally fit to the data very well. This happens with certain one-parameter probability distributions, most notably the Poisson distribution. In this case, it is common for the data to be more variable than the estimated variance coming out of your model, in which case we say that there is a problem of "overdispersion".

Both of the descriptions you have given for this are correct. Overdispersion is indeed the presence of greater variability in the data than predicted by the model. This generally occurs because the variance in the distribution used in the model is a function of the mean, so the estimation procedure can't estimate them both well (and mean estimation is generally more important than variance estimation when fitting data to a model).

Roughly speaking, if you have $k$ parmaters in a statistical distribution, and you fit it to data, it will allow you to accurately estimate $k$ moments of the distribution (often the first $k$ moments, but not always$^\dagger$). So, for example, some one-parameter distributions allow you to accurately estimate the mean but not the variance, some two parameter distributions allow you to accurately estimate the mean and variance but not the skewness, some three parameter distributions allow you to accurately estimate the mean, variance and skewness, but not the kurtosis, and so on.

If you want to avoid overdispersion in your modelling, you should use statistical models that use an underlying two-parameter distribution that can fit the mean and variance (e.g., use a negative binomial model instead of a Poisson model). The same basic principle also applies if you want to accurately fit higher-order moments --- e.g., if you want to accurately fit skewness you might generalise to a three-parameter distribution, and so on.


$^\dagger$ For example, the Student's T-distribution has a single parameter that affects the variance and kurtosis but not the mean or skewness.