I would like some advice on modeling rare events that arises from a continuous variable crossing a threshold.

I have a data set consisting of a continuous response $Z_t$ and continuous regressors $A_t, B_t, \dots$ and would like to model when $Z_t$ crosses some threshold $k$ (i.e., when $Z_t > k$). A logistic regression works reasonably well, but there are a few problems: the events are rare (I have about 1200 observations with less than a dozen events) and I would like this to back test well (this means estimating the model on observations where $t_1<t<t_2$ and then sliding $t_1,t_2$ forward in time). Given the rarity of events, during back testing there will be many $t_1,t_2$ with few, if any, events. Furthermore, given that there is a continuous variable, $Z_t$, underlying the events, it seems that focusing on just the events results in some information being discarded (such as a "near miss", where $Z_t$ is only slightly less than $k$).

A natural approach would to directly model just $Z_t$ using a linear model. This does not work very well, probably because away from $k$, $Z_t$ behaves in a way that is not well modeled by a linear model. So, loosely speaking, it seems that what I want is a model that is similar to a linear model but is "tuned" to pay attention to $Z_t\approx k$.

For example, here is one concrete approach: convert $Z_t$ into a value between 0 and 1 (via a logit function, say; map $Z_t=k$ to $0.5$) and then use a fractional logistic regression.

What are some modeling strategies for a situation such as this?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.