Regression on imbalanced and zero-inflated data. How to deal with less frequent values?

Question

I am implementing a regression, however my regressor has not been able to predict the least frequent counts. I've tried adjusting the hyperparameters (as you can see below), but I haven't had much success.

As you can see below, my dataset is zero-inflated and the count frequency drops dramatically as the count value increases.

df_count = df.groupby(['count']).size().to_frame(name = 'size').reset_index()
df_count

count   size
0   2291939
1   23796
2   3513
3   595
4   209
5   52
6   24
7   7
8   2
10  1
15  1

As the frequency of some values ​​is very small, I have had difficulty in predicting them, as you can see in my attempts below. Any suggestions on how to handle this situation?

from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor

zir_rf = ZeroInflatedRegressor(
    classifier=RandomForestClassifier(random_state=0),
    regressor=RandomForestRegressor(random_state=0)
)

zir_rf.fit(scaled_train, y_train.values.ravel())

predictions_zir_rf  = zir_rf.predict(scaled_test)

mean_squared_error(y_test['count'].values,predictions_zir_rf)
0.004358037315598189

plt.clf()
plt.hist([y_test['count'].values, predictions_zir_rf], log=True)
plt.legend(('real','predction'))
plt.show()

After some RandomizedSearchCV I tried the following model and still got the same problem of not being able to predict the less frequent counts.

zir_rf_tunned = ZeroInflatedRegressor(
    classifier=RandomForestClassifier(random_state=0),
    regressor=RandomForestRegressor(n_estimators=200, min_samples_split = 10, min_samples_leaf = 2,  max_features = 'sqrt', max_depth =50, bootstrap = True)
)

predictions_zir_rf_tunned  = zir_rf_tunned.predict(scaled_test)
mean_squared_error(y_test['count'].values,predictions_zir_rf_tunned)
0.004107122405590996

plt.clf()
plt.hist([y_test['count'].values, predictions_zir_rf_tunned], log=True)
plt.legend(('real','predction'))
plt.show()

Answer 1

You tell the model that certain outcomes are uncommon. The model shys away from predicting them. I find this to be fairly expected behavior.

If you want to be able to predict these unusual events, there must be something that distinguishes them from the common events. Let's look at a simulation.

library(ggplot2)
set.seed(2023)
N <- 1000
x1 <- runif(N, 0, 10)
x2 <- rt(N, 1.5)
y <- x1 + x2 
d <- data.frame(x1 = x1, x2 = x2, y = y)
ggplot(d, aes(x = x1, y = y)) +
  geom_point()

In this simulation, y is totally determined by x1 and x2. However, if we never think to model with x2, there will be some rather large deviations from the main part of the data.

If you just use x1 to chase after these unusual y values like -77.215 and +36.134, your modeling is going to present some issues. The data simply do not allow for reliable predictions like that. There is nothing in the feature space to differentiate y=-77.215 from the mainstream of the data, so your model will (and should) predict something less extreme.

If, however, you consider the other feature, x2, then there is something unique about these extreme y values.

ggplot(d, aes(x = x1, y = x2, col = y)) +
  geom_point() +
  scale_color_distiller(palette = 'Spectral')

This is what is meant about regression explaining the variance in y. You observe many different values of some outcome of interest. Perhaps you can start to account for some of those differences by considering additional information. In this case, x1 does not fully explain y, but bringing x2 into the mix does the trick.

There's probably a feature missing from your analysis. For instance, if your count is how many chocolate boxes are sold in various stores across various days, perhaps you are not accounting for some of those days being Valentine's Day.