0
$\begingroup$

Disclaimer: I know this is a long-ish post but I don't need code solutions just high level general direction approaches that are usually used in situations like these.

So let's say I want to predict number of people on the street or city square at any given moment.

I have hourly data of average number of people seen on the street. Let's say this data was collected automatically with camera and image recognition software which counted the people.

I also have weather data: temperature, rain and wind speed.

From some basic data analysis we can see some general patterns:
- more people on the street when it's warmer, no rain and no wind
- fewer people when it's colder or when it rains or when there is very strong wind

If I take some standard model like RandomForestRegressor from scikit-learn and ignore the weather variables it will capture daily and weekly trends quite well. It will not take into account weather conditions but overall it works and predicts fairly good (especially on larger timescale).

Now my question begins:

How do I handle sparse events like weather conditions to improve Random Forest's predictions (like multiplying it with some coef or some better approach)?

E.g. let's say I have historical data on raining on Wednesday (20% less people) and raining on Thursday (15% less people). But I don't have data on raining on Friday. And my weather forecast says it's going to rain next Friday so I have opportunity to use this information and improve my prediction but how?

If I simply put everything in Random Forest it will "say" there is no bin or data for this case (Friday and raining). No data means prediction is zero. (which is not true - in reality it will be lower than usual but not zero)

Yes I could calculate how much is an average number of people lower on rainy days compared to regular days (Wednesday, Thursday) and then apply this number to Friday but I wonder if this is the right approach?

Also what happens if I want to include many different factors (or just experiment with many different factors)? This seems like a very slow and tedious process.

What would be better approach? (different models or neural nets or...?)

$\endgroup$
1
$\begingroup$

You can have a categorical (nominal) feature called "Weather" with the cardinalities: "Rainy", "Sunny", etc. For the days where you have no data, you can represent missingness explicitly by adding a "No Weather Data" category.

You can feed your data into a random forest, or any linear / non-linear model in this way. Obviously you will need to one-hot encode your nominal feature(s) before feeding it in.

If you use R, this is done automatically with most packages so you don't have to worry about it. In Python, you have to do this manually.

$\endgroup$
  • $\begingroup$ That will not work as I mentioned in my question. I already tried dummy variables with linear regression before switching to random forest which does basically the same thing out of the box without one-hot encode. The problem is that training data only has "No Weather Data" and Friday and prediction data will have "Rainy" and Friday. (at which point predictions will break) $\endgroup$ – veich Dec 21 '17 at 18:28
  • $\begingroup$ A good algorithm will, when faced with this situation, in effect "back up" the tree(s) until it gets to a node which does have data and therefore can make a prediction, so predictions should not break (it's not done this way in the code, but conceptually you can think of it this way.) $\endgroup$ – jbowman Dec 21 '17 at 18:52
  • 1
    $\begingroup$ Actually you're right this will solve the problem (although it has some assumptions contained in it but it can be good enough). I was thinking in totally different direction at the time I posted this question. I'll accept this as correct answer. $\endgroup$ – veich Jan 21 '18 at 11:32
1
$\begingroup$

Your problem "So let's say I want to predict number of people on the street or city square at any given moment." is fundamentally no different than Simple method of forecasting number of guests given current and historical data and http://autobox.com/cms/index.php/afs-university/intro-to-forecasting/doc_download/53-capabilities-presentation slide 50 . Your problem involves identifying all that is discussed there with the addition that hourly patterns may be different on weekends as compared to weekdays.

In terms of rainfall/weather , I have found that cooling degree days and heating degree days are very useful. Additionally there could be lead and lag effects around degree days i.e. both anticipatory and responsive activity.

The way auxiliary predictors come into play is fully discussed here
http://www.math.cts.nthu.edu.tw/download.php?filename=569_fe0ff1a2.pdf&dir=publish&title=Ruey+S.+Tsay-Lec1 and here http://autobox.com/dave/TFFLOW.png

Identifying day-weather interactions is very tricky but could be done by evaluating alternative structure for the design matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.