# Approach to Analyzing Semi-Rare Events

I am often faced with analyzing data that follow a pattern as shown in a mock example in the image below. Key data characteristics:

1. for any value of the predictor (e.g. temperature), the most frequently observed outcome is always Zero. If I were to calculate the median for example at any predictor value, I would get zero. Thus the outcome (e.g. crack area) is highly skewed toward zero or exponentially distributed.

2. often I have many more observations available in a certain predictor range.

My interest here is not so much Prediction as finding associations between effects and outcomes.

I have read on this forum (for example here) about the dangers of turning continuous variables into discrete variables.

My question: in a case such as this, would it be acceptable to classify the predictor and outcome and then analyze the observed frequencies in a Contingency Table using the Chi-Square statistic. For example, in my view there is a clear conceptual difference between "No Cracks" and "Some Cracks" or "High Cracks".

If this approach (Contingency tables) is not appropriate, can you please suggest other approaches to analyze this type of data, bearing in mind I am primarily interested to find associations between variables, and secondly only in predictive modelling. My analysis outcomes need to be explained to non-statisticians. • I presume that you can (?) express the crack area as a percentage of the total area of whatever object it is that you are measuring? If yes, do the different objects represented in your plots have the same total area? – Isabella Ghement Jul 11 '19 at 23:49
• @IsabellaGhement - you can assume each dot is an observation taken under the same conditions. I just used cracks vs temperature as an example. Another example could be annual repair cost of air conditioners vs daily usage in hours for the same brand of air conditioner. Most observations would not have any repair costs, but you may see those with higher daily use having (a) a greater probability of needing repairs; and (b) greater repair cost if they do need repair. – Fritz45 Jul 12 '19 at 2:55
• Thanks - see my answer below. – Isabella Ghement Jul 12 '19 at 3:50

Your second example (the air conditioners one) makes a bit more sense to me, though ideally the time scale used for cost of repair would be the same as that for the usage. To make it simpler, I will assume that you have n air conditioners you monitor over the duration of a year. For each air conditioner, you would record the following information:

1. Usage amount in hours (for the entire year);
2. Whether or not the air conditioner needed any repairs during that year;
3. The total repair cost for the year.

The total repair cost would be 0 dollars for air conditioners not needing any repairs for the year.

The resulting data could be modelled via a two-step model.

In the first step, you would fit a binary logistic regression model relating Y to X for all n air conditioners, where:

• Y = 1 if the air conditioner did need repairs and 0 if it didn't;
• X = usage amount.

This way you can model the probability of needing repairs to the usage amount.

In the second step, you could use something like log-linear regression to relate Y to X only for those air conditioners that need repairs, where this time:

• Y = total repair cost for the year;
• X = usage amount.

The reason you would need to log-transform cost is because cost variables tend to be right-skewed. (Another option here would be gamma regression.)

This kind of two-step model tends to be used in the medical literature to model health care utilization costs - you can do a Google search to find some references.

If you have issues with rare events in the binary logistic regression model (where event would be akin to an air conditioner needing repairs), then you can address those in a variety of ways - again, do a Google search on binary logistic regression with rare events.

• many thanks for your extensive answer. I have some experience with your proposal and I will look into it a bit more. As indicated in my question, I am actually more interested in opinions about the use of Contingency tables to explore the strength between effects and outcomes. A prediction model in itself may not be that useful to me, although I can see how it can be used. Many thanks again for your answer. Please allow a few more days - if I do not receive more specific answers I will accept yours. – Fritz45 Jul 12 '19 at 4:38
• You seem to believe that the models I suggested don’t allow you to assess the strength of effects? In my example, the exponentiated coefficient of usage amount in each model will help you assess the strength and direction of the effect of usage amount on the probability of needing (at least one) repair and on the repair cost, respectively. No need to accept my answer - I can delete it if you think it takes away from what you are after. – Isabella Ghement Jul 12 '19 at 14:04
• I think your answer will be very useful to many people like myself who are not statisticians. Also I take your point about the strength of effects being explained by the exponentiated coefficients. I am just specifically interested in the potential use of Contingency Tables in this context and want to see if other views on this come up. Thanks again! – Fritz45 Jul 12 '19 at 17:32