4
$\begingroup$

I'm currently working on a data set where the goal is to predict the number of rented bikes in Seoul, given information about the weather at the time.

The data set can be downloaded here: https://archive.ics.uci.edu/dataset/560/seoul+bike+sharing+demand

One of the possible predictors is the variable rainfall, indicating the rainfall measured in millimetres at a given hour.

The distribution of that variable however is extremely skewed:

enter image description here

As you can see most of the observations had no rainfall at all, making the observations with rain almost invisible in the plot.

Applying a log-transformation didn't really improve the skewness too much: enter image description here

Also using other transformation techniques like the Box-Cox-Tranformation didn't yield any desirable results.

What would be an appropriate way to transform or use a variable like this as a regressor for a linear regression model?

Thank you very much for your help!

EDIT:

Here is a histogram of only the positive values of rainfall: enter image description here

And the distribution of the log of the positive values:

enter image description here

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ I suggest to add a dummy variable for rain > 0 ( since that's a critical discontinuity in the factors influencing bike rental) , and then use the log1p transform for rest. Can you show density/ecdfplot for rain > 0 only $\endgroup$
    – Georg M. Goerg
    Commented Jul 14, 2023 at 12:16
  • $\begingroup$ Thank you very much already. How could I include a dummy in linear regression that considers different values, when rain > 0? $\endgroup$
    – Jan Philip Richter
    Commented Jul 14, 2023 at 12:28
  • 1
    $\begingroup$ I agree with the suggestion of @Georg M. Georg. But in light of your reply on multiple dummy variables: Do you really have to use linear regression? A GAM with a smoothing spline for rain (e.g., fit using R package mgcv) will do the scaling in an appropriate manner (driven by observed data), more or less automatically. Still, it allows you to do inference on the effect of rain. A model where you create dummy indicators for different values of rain is probably more cumbersome to fit and more difficult to interpret. $\endgroup$
    – Marjolein Fokkema
    Commented Jul 14, 2023 at 12:46
  • 1
    $\begingroup$ @Georg The suggestion to use a zero-inflated approach is good, but the log1p method is (highly) problematic, as discussed at stats.stackexchange.com/questions/30728. In my experience, rainfall is a challenging variable to analyze and requires a careful study of its distribution and of the data measurement process. $\endgroup$
    – whuber ♦
    Commented Jul 14, 2023 at 13:09
  • 1
    $\begingroup$ @GeorgM.Goerg My understanding of log1p is that it uses a start value of $1.$ The thread I linked to explains why that is arbitrary and shows how bad it can be. $\endgroup$
    – whuber ♦
    Commented Jul 16, 2023 at 15:00

1 Answer 1

Reset to default
10
$\begingroup$

First, note that OLS regression does not require normal variables, it requires normal errors and we look at residuals (since we don't know the error). However, it seems very likely that your residuals will not be normal.

Second, I agree with Georg Goerg that adding a dummy for "rain" and then looking at rainfall is a good idea. Surely some people don't rent bikes if there is any rain at all, while others will do so if it's raining a little, but not in a huge storm. But see below.

Third, rather than use OLS regression, which assumes normal residuals, why not use a method that doesn't? In your case, I think quantile regression might suit your purposes. Not only does it not make assumptions about the residuals, but it lets you look at quantiles of the DV. I'm guessing you might be very interested in what relates to very high usage (perhaps you want to move more bikes when a huge usage is predicted).

Finally, I think you should look at a spline of rainfall. If you do this, then you might not have to separate "no rain" vs. "rain" as one of the knots in the spline will surely be there. I think the biggest drawback of using splines of independent variables is that they can be hard to interpret. But you say your goal is prediction. Another possibility is multivariate adaptive regression splines (MARS).

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ I must object to the notion that OLS requires normal errors yet minimizing MAE for a quantile regression does not. If OLS requires normal errors, it is to coincide with maximum likelihood estimation of the parameters, and then minimizing absolute loss corresponds to maximum likelihood estimation for Laplace-distributed errors. $\endgroup$
    – Dave
    Commented Jul 14, 2023 at 23:58
  • 1
    $\begingroup$ I believe, since the stated goal is to predict, the point about the residuals' distribution would not apply here. $\endgroup$
    – Firebug
    Commented Jul 15, 2023 at 8:39
  • 1
    $\begingroup$ @Firebug: If the residuals are very far5 from normal (especially if fat tails) then some other method than least squares could be much better $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 22, 2023 at 19:12
  • $\begingroup$ @kjetilbhalvorsen I have to think about this. Would it be better in terms of mean squared error or in terms of some other log likelihood? $\endgroup$
    – Firebug
    Commented Jul 23, 2023 at 20:54
  • 1
    $\begingroup$ @Firebug by definition, nothing will beat an MSE-minimizing estimator in terms of in-sample MSE. However, the story can change for our-of-sample MSE. I have some examples here. $\endgroup$
    – Dave
    Commented Jul 23, 2023 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.