Approach to Analyzing Semi-Rare Events

Question

I am often faced with analyzing data that follow a pattern as shown in a mock example in the image below. Key data characteristics:

1. for any value of the predictor (e.g. temperature), the most frequently observed outcome is always Zero. If I were to calculate the median for example at any predictor value, I would get zero. Thus the outcome (e.g. crack area) is highly skewed toward zero or exponentially distributed.

2. often I have many more observations available in a certain predictor range.

My interest here is not so much Prediction as finding associations between effects and outcomes.

I have read on this forum (for example here) about the dangers of turning continuous variables into discrete variables.

My question: in a case such as this, would it be acceptable to classify the predictor and outcome and then analyze the observed frequencies in a Contingency Table using the Chi-Square statistic. For example, in my view there is a clear conceptual difference between "No Cracks" and "Some Cracks" or "High Cracks".

If this approach (Contingency tables) is not appropriate, can you please suggest other approaches to analyze this type of data, bearing in mind I am primarily interested to find associations between variables, and secondly only in predictive modelling. My analysis outcomes need to be explained to non-statisticians.

Answer 1

Your second example (the air conditioners one) makes a bit more sense to me, though ideally the time scale used for cost of repair would be the same as that for the usage. To make it simpler, I will assume that you have n air conditioners you monitor over the duration of a year. For each air conditioner, you would record the following information:

1. Usage amount in hours (for the entire year);
2. Whether or not the air conditioner needed any repairs during that year;
3. The total repair cost for the year.

The total repair cost would be 0 dollars for air conditioners not needing any repairs for the year.

The resulting data could be modelled via a two-step model.

In the first step, you would fit a binary logistic regression model relating Y to X for all n air conditioners, where:

• Y = 1 if the air conditioner did need repairs and 0 if it didn't;
• X = usage amount.

This way you can model the probability of needing repairs to the usage amount.

In the second step, you could use something like log-linear regression to relate Y to X only for those air conditioners that need repairs, where this time:

• Y = total repair cost for the year;
• X = usage amount.

The reason you would need to log-transform cost is because cost variables tend to be right-skewed. (Another option here would be gamma regression.)

This kind of two-step model tends to be used in the medical literature to model health care utilization costs - you can do a Google search to find some references.

If you have issues with rare events in the binary logistic regression model (where event would be akin to an air conditioner needing repairs), then you can address those in a variety of ways - again, do a Google search on binary logistic regression with rare events.