Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Hot answers tagged

88

Here is an example of application. Poisson regression is typically used to model count data. But, sometimes, it is more relevant to model rates instead of counts. This is relevant when, e.g., individuals are not followed the same amount of time. For example, six cases over 1 year should not amount to the same as six cases over 10 years. So, instead of ...


50

Recall that an offset is just a predictor variable whose coefficient is fixed at 1. So, using the standard setup for a Poisson regression with a log link, we have: $$\log \mathrm{E}(Y) = \beta' \mathrm{X} + \log \mathcal{E}$$ where $\mathcal{E}$ is the offset/exposure variable. This can be rewritten as $$\log \mathrm{E}(Y) - \log \mathcal{E} = \beta' \...


24

Offsets can be used in any regression model, but they are much more common when working with count data for your response variable. An offset is just a variable that is forced to have a coefficient of $1$ in the model. (See also this excellent CV thread: When to use an offset in a Poisson regression?) When used correctly with count data, this will let ...


19

You can use an offset: glm with family="binomial" estimates parameters on the log-odds or logit scale, so $\beta_0=0$ corresponds to log-odds of 0 or a probability of 0.5. If you want to compare against a probability of $p$, you want the baseline value to be $q = \textrm{logit}(p)=\log(p/(1-p))$. The statistical model is now \begin{split} Y & \sim \...


16

Offsets in Poisson regressions Lets start by looking at why we use an offset in a Poisson regression. Often we want to due this to control for exposure. Let $\lambda$ be the baseline rate per unit of exposure and $t$ be the exposure time in the same units. The expected number of events will be $\lambda \times t$. In a GLM model we are modelling the ...


15

If you are interested in the probability of an incident given N days of patients on ward then you want a model either like: mod1 <- glm(incident ~ 1, offset=patients.on.ward, family=binomial) the offset represents trials, incident is either 0 or 1, and the probability of an incident is constant (no heterogeneity in tendency to generate incidents) and ...


12

It is correct you to get cases instead of rates since you are predicting cases. If you want to obtain the rates you should use the predict method on a new data set having all columns equal to data but the population column identically equal to 1, so to have log(populaton)=0. In this case you will get the number of cases of one unit of population, i.e. the ...


11

There are several issues here: You need to use the observed counts as your response variable. You should not use the densities (g_den). If the observed counts are from differing areas, you need to take the log of those areas as a new variable: larea = log(area) You can control for the differing areas for the observations in two different ways: ...


9

This answer comes in two parts, the first a direct answer to the question and the second a commentary on the model you're proposing. The first part relates to the use of Numbers as an offset along with having it on the r.h.s. of the equation. The effect of doing this will simply be to subtract 1 from the estimated coefficient of Numbers, thereby ...


8

1) What is the best way of determining when to use neg. binom. vs. poisson? A common way (not necessarily the best --- what's 'best' depends on your criteria for bestness) to decide this would be to see if there's overdispersion in a Poisson model (e.g. by looking at the residual deviance. For example, look at summary(glm(count~spray,InsectSprays,family=...


8

I don't know why glm() doesn't blow up. To figure that out, you'll have to unpack all of the underlying code. (In addition, if your only question is how the R code works, this question is off topic here.) What I can say is that you are not modeling the rates correctly. If you want to model rates instead of counts, you need to include an offset in the ...


8

I don't know where you heard that a Poisson or negative binomial with an offset is preferable to a binomial model for a number of individuals surviving out of an initial number; I would normally prefer a binomial as it is closer to the actual stochastic process we think is going on. Note that the binomial model would be a binomial GLM, $$ n_{\textrm{surv}} \...


8

This also confused me. I thought, "what is the point of explicitly including an offset instead of just pretending that the response divided by the offset / exposure is the $y$ value?". You actually get two different loss functions if you do so. The correct way (use an exposure/offset $s_i$) Model $\log \lambda_i = \log s_i + \theta^T x$ so that $\lambda_i ...


7

You can always include an offset in any GLM: it's just a predictor variable whose coefficient is fixed at 1. Poisson regression just happens to be a very common use case. Note that in a binomial model, the analogue to log-exposure as an offset is just the binomial denominator, so there's usually no need to specify it explicitly. Just as you can model a ...


7

Time offsets can usually be viewed as your model estimating the rate an event occurs per unit time, with the offset controlling for how long you observed different subjects. In poisson models you are always estimating a rate that something happens, but you never get to observe this rate directly. You do get to observe the number of times that an event ...


7

According to the answer in: https://stackoverflow.com/questions/34896004/xgboost-offset-exposure xgboost can handle offset term as in glm or gbm using setinfo, but this method is not documented very well. In your example, the code would be: setinfo(xgbMatrix,"base_margin",log(Insurance$Holders))


6

I think that you want offset(log(population)) in your models above. The offset is just a term included in the model without estimating a coefficient for it (fixing the coefficient at 1). Since the standard transformation in poisson regression is log, you can think of incuding the offset of log(population) as a rough equivalent (though mathematically ...


6

First, from the help-page of gam (bold font added by me): offset: Can be used to supply a model offset for use in fitting. Note that this offset will always be completely ignored when predicting, unlike an offset included in formula: this conforms to the behaviour of lm and glm. So for predicting, you should use the formula-specification. Further, ...


6

One way would be to adopt a formal model-based tree. The glmtree() function in the partykit package implements the general MOB algorithm for model-based recursive partitioning (Zeileis et al. 2008, Journal of Computational and Graphical Statistics, 17(2), 492-514). This supports Poisson responses and also allows for the inclusion of offsets. Furthermore, ...


6

It looks like you divided the fish counts by the volume (or perhaps area) of water surveyed. In that case an offset is indeed appropriate, you should use the log of whatever you divided by. Perhaps model1 <- glm(g_den ~ method + site + depth + offset(log(area)), poisson) (edited from earlier incorrect version, missing the log) The reason for the error ...


6

Look at confidence interval for parameters of your GLM: > set.seed(1) > x = rbinom(100, 1, .7) > model<-glm(x ~ 1, family = "binomial") > confint(model) Waiting for profiling to be done... 2.5 % 97.5 % 0.3426412 1.1862042 This is a confidence interval for log-odds. For $p=0.5$ we have $\log(odds) = \log \frac{p}{1-p} = \log 1 = 0$. ...


5

Rounding your response variable to an integer is NOT OK. For simplicity, lets assume you're conducting a Poisson regression. What you're modeling is the following: $ \begin{align*} E(Y|x) &= \beta^{T}x + \beta_{0} \\[0.5em] \log \left( \frac{\mbox{No. of Deer}}{\mbox{Area}} \right) &= \beta^{T}x + \beta_{0} \\[0.5em] \log(\mbox{No. of Deer}) - \...


5

So the response you want to model is "Number of calls per bird" and the troublesome lines are where you didn't observe any birds? Just drop those rows. They add no information to the thing you are trying to model.


5

$R^2$ is computed in terms of the sum of squares of fitted values $\text{MSS}=\sum (\hat{y}_i - \bar y_i)^2$ (assuming an intercept term is present) and sum of squares of residuals $\text{RSS} = \sum \left(y_i - \hat{y}_i\right)^2$ as $$R^2 = \frac{\text{MSS}}{\text{MSS} + \text{RSS}}:$$ it is the fraction of the total sum of squares "explained" by the fit....


5

While I don't recommend looking at the source code for glm for those that wish to preserve their mental health, I looked at the source code to glm. The reason R doesn't blow up seems to be that it just doesn't bother to do the kind of defensive checks it probably should. The main iteratively re-weighted least squares loop works by using the methods ...


5

If you are going to model using the Poisson you have to have integer values for your response variable. You then have two options Use area or some other suitable denominator as an offset. This would usually need to be logged first Include area or etc as a predictor variable. Again this would usually be included as a log because you are modelling the log ...


5

An offset is generally just a coefficient set to a specific value. To get more than one offset, in general you just need to combine the different variables in a way that is consistent to get that fixed value. In a Poisson equation if you set $Z$ as the offset (or exposure its sometimes called): $$\log(\mathbb{E}[Y]) = \beta_0 + \beta_1X + 1\cdot Z$$ And ...


5

(with your R code, you could replace "poisson" with "quasipoisson" to avoid all the warnings that get generated. Nothing else of import will change. See (*) below). Your reference use the term "multiplicative glm" which I think just means a glm with log link, since a log link can be thought of as a multiplicative model. Your own example shows that the ...


4

Here is the source code of pscl:::pR2.glm: function (object, ...) { llh <- logLik(object) objectNull <- update(object, ~1) llhNull <- logLik(objectNull) n <- dim(object$model)[1] pR2Work(llh, llhNull, n) } <environment: namespace:pscl> If the offset is specified in the formula, it gets lost in the second line (update ...


4

The point of the offset is that you do not explicitly transform the response. The rate resulting from the standardization would typically not be an integer and a Poisson model would not fit well then. Instead one keeps the count response for which a count distribution like Poisson is appropriate and includes log(exposure) as an offset. Then you get $$ \log(...


Only top voted, non community-wiki answers of a minimum length are eligible