2
$\begingroup$

I am using the random forest for classifing if it will rain (1) or not (0) in my daily rain dataset with a small quantity of data (8103 tuples). Currently running a walking foward evaluation looking at the recall metric and getting a mean of 83.

I'm using a multivariate approach with the following columns: enter image description here

I know the RF doesn't use autoregression, so I'm looking for ways to help it understand the time property better. Also if I there is any argument I can pass to RFClassifier that would be good, I'm already using bootstrap=False.

I already tried other techniques such as ETS and ARIMA, but they could not beat mean (only STLF has beaten it actually) so I'm looking into the Machine Learning approach.

$\endgroup$

2 Answers 2

1
$\begingroup$

You can always include lagged values of the target variable to account for autocorrelation. However, for a Boolean target, that will likely not add a lot of value. Also, much of the autoregressive behavior is probably already captured by the fact that your weather variables are already highly autoregressive (incidentally, you will need to forecast them, too).

You may want to consider Fourier transforms of day_of_year and wind_direction so your RF understands the circular nature of these predictors.

Are you training your RF on recall? If not, be aware it will likely optimize on a different loss function. Optimizing and evaluating on two different loss functions does not make a lot of sense (Kolassa, 2020), and recall is not a very good evaluation measure in any case.

$\endgroup$
6
  • $\begingroup$ Hey, thanks for the response, I will try the transforms. As for the training, the RF is not being trained on recall, I'm using recall to evaluate how the model performs on the walk foward validation. How does one change the loss function on RF (I'm using sklearn)? The question you linked seems to suggest the accuracy is bad, and it really is even more on an unbalaced target such as rain, but from multiple sources I've come to the conclusion that recall on class 1 makes sense to evaluate rain. $\endgroup$
    – Guilherme
    Jul 20, 2022 at 18:08
  • $\begingroup$ Your paper seems to be focused on regression problems, I don't know if its possible to apply this line of thought for RF classification, the "loss functions" would be the "criterion" parameter if I'm not wrong, and the values it assumes are “gini”, “entropy”, “log_loss” $\endgroup$
    – Guilherme
    Jul 20, 2022 at 18:29
  • 1
    $\begingroup$ Unfortunately, I don't know how to change the loss function in the sklearn RF, sorry. The IMO best approach would be to aim for well-calibrated probabilistic predictions, then compare these to a threshold that is optimized for recall. (If you have found convincing sources that argue for recall, good for you. My experience is that most "data science" sources that argue for recall don't even understand the underlying statistical problem, much less address it.) ... $\endgroup$ Jul 20, 2022 at 18:46
  • $\begingroup$ ... Yes, my paper looks at numerical predictions, but I suspect the issue is precisely the same. Assume you have well-calibrated probabilistic predictions, and you look for one threshold that optimizes (say) expected recall, and another threshold that optimizes (say) expected precision, are the two thresholds the same? I haven't thought through this, but I would be surprised if they were. Thus, different loss functions imply different classifications if I am right about my hunch. Which is the classification analogue of what my paper looks at. $\endgroup$ Jul 20, 2022 at 18:48
  • 1
    $\begingroup$ Good luck! Incidentally, if you are only interested in recall, an easy way of getting a recall of 1 is to classify everything as the target class (essentially: comparing implicit probabilistic predictions to a threshold of 0). That is presumably not what you want, so you probably also want some precision or accuracy and use a larger threshold. Which is exactly my point: changing the accuracy measure changes the optimal output, even for precisely well-calibrated probabilistic predictions. You may be interested in this thread. $\endgroup$ Jul 21, 2022 at 6:36
1
$\begingroup$

You could give a dev package I wrote up a shot. It just wraps LightGBM (boosted trees) with some time series functionality such as fourier basis functions, 'custom' linear basis functions, and ar components.

Here is a quick example:

from sklearn.datasets import fetch_openml
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('darkgrid')
bike_sharing = fetch_openml("Bike_Sharing_Demand", version=2, as_frame=True)
y = bike_sharing.frame['count']
y = y[-800:].values
y_class = y > 70
y_class = y_class * 1

from LazyProphet import LazyProphet as lp
lp_model = lp.LazyProphet(seasonal_period=[24],
                      n_basis=10,
                      objective='classification',
                      fourier_order=5,
                      decay=.99,
                      ar=3,
                      return_proba=True)
fitted = lp_model.fit(y_class)
predicted_class = lp_model.predict(100)
plt.plot(y_class)
plt.plot(np.append(fitted, predicted_class), alpha=.5)
plt.axvline(800)
plt.show()

enter image description here

You can pass your own params such as the objective and all that through the 'boosting_params' argument.

And here is a multivariate example. It may be wonky so just let me know if there are issues.

$\endgroup$
9
  • $\begingroup$ Thanks for the reponse, I'm trying to run the quick example you showed on collab but I'm getting ValueError: cannot convert float NaN to integer on fitted = lp_model.fit(y_class), but there is no NaN in y or y_class. Any ideas? $\endgroup$
    – Guilherme
    Jul 20, 2022 at 19:43
  • 1
    $\begingroup$ Awesome, I'll take a look. In the mean time it should work if you don't use ar components at all. $\endgroup$
    – Tylerr
    Jul 20, 2022 at 21:25
  • 1
    $\begingroup$ I just debugged the issue, if you pass y as all ints with AR and classification it errors. The short fix should be to convert your y to floats before fitting. Fingers crossed it fixes the issue. $\endgroup$
    – Tylerr
    Jul 21, 2022 at 2:12
  • 1
    $\begingroup$ Good to hear! In terms of documentation, LightGBM itself has a ton of resources for the model itself. Most important things to tune are things like num_leaves and num_iterations. Most of the other stuff is just fancy feature engineering I got from GAMs: sin and cosine pairs for seasonality, and piecewise linear basis functions for trend. The weighting scheme for the plbf I think is novel so you are stuck with reading my tin foil conspiracies towardsdatascience.com/…. Feel free to reach out with anymore questions. $\endgroup$
    – Tylerr
    Jul 21, 2022 at 2:14
  • 1
    $\begingroup$ return_proba=True will return the probabilities, False with just do 1/0. $\endgroup$
    – Tylerr
    Jul 24, 2022 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.