0
$\begingroup$

I'm currently working on a prediction problem which deals with prediction of rainfall across US. The data I'm dealing with a time horizon which ranges from 1970 through 2019, at monthly intervals. I have historic rainfall recorded at a zip level and I have had these zip codes mapped to a metropolitan statistical area (MSA) and combined statistical area (CBSA) across each state in the US. For example, there are about 30 MSAs in California for which I need to predict the rainfall for future time periods.

I have my doubts on the approach - generally I am used to passing an array into any ML algorithm (Persistence, ARIMA, ETS, CNN/RNN) which gives me the results. In this case I feel it is not feasible to pass 30 MSAs*50 states on an average for the same time frame, which is more than passing 2000 arrays into the model for the same time period.

Is there a better way of approaching this problem in terms of modeling and recording the output?

$\endgroup$
0
$\begingroup$

I think the model should learn two things: time series behaviour and some general rules from the data for each state, i.e. if you have the following data:

day | state | rainfall
  1 |    CA |       10
  2 |    CA |       25
  3 |    CA |        0
  1 |    WA |      100
  2 |    WA |       90
  3 |    WA |       80

then an example for those two rules would be

  1. It will rain roughly the same amount as the day before
  2. In southern states it rains much less

In order to make it learn rules of the first time you need to transform the data into the following format

day | state | rainfall | rainfallBefore
  1 |    CA |       10 |             NA
  2 |    CA |       25 |             10
  3 |    CA |        0 |             25
  1 |    WA |      100 |             NA
  2 |    WA |       90 |            100
  3 |    WA |       80 |             90

This essentially covers the time series part because all the models that people usually use when they say 'time series' is ARIMA or so which is basically one very special instance (not necessarily the best one) of a very normal regression with the transformed table as above (containing lagged variables). However, one can use many different models: https://www.r-bloggers.com/timeseries-forecasting-using-extreme-gradient-boosting/.

In order to make it learn about the second part (i.e. "surfing the wave of right now" features) you need to join in as much information as possible about the states that could be related to rainfall. One possibility that would it make easy for a model to learn a rule as given above would be to insert the lat/lon coordinates of the middle point of each state. Then the model yould learn the rule:

The bigger the latitude the more south the state is, hence, the less rain I should expect.

So:

  1. transform the data table and introduce lagged features
  2. introduce many state rain relation features
  3. take any regression model and let it run on this training set with rainfall as a target variable
$\endgroup$
  • $\begingroup$ Noted. However, if this entire data is plugged into a model as a multivariate approach, how will this only return the predicted output (rainfall) for each state for observations 4,5,6 in the example you mentioned, since 1,2,3 will repeat for each state and there can also be a feature selection step based on that column alone, like periodic indicators $\endgroup$ – Ayan Jun 7 at 10:54
  • $\begingroup$ @Ayan: I do not understand... given one row of this table, the model predicts rainfall (for that particular state and time period). What exactly do you mean when you say that 1,2,3 will repeat? $\endgroup$ – Fabian Werner Jun 7 at 21:43
  • $\begingroup$ Apologies if I have been vague. Given your example, I would want the prediction of rainfall for each state for the next h periods. I understand the formulation suggested by you. However, given I prepare the data table as you suggest, will the prediction be at the state level (since state is just a feature here)? $\endgroup$ – Ayan Jun 9 at 18:38
  • $\begingroup$ I think you have to do the usual thing in time series analysis then: predict the rainfall on day x using the model and the rainfall of x-1 as input feature. Then you use the prediction of the model on day x as an input feature and predict the rainfall on x+1 and so forth... $\endgroup$ – Fabian Werner Jun 10 at 8:02
  • $\begingroup$ But this is only one possibility and there are many more... for example you could also train one model for rainfall at day x, another one for day x+1, yet another one for day x+2 and so on. Also you could try to fit a curve to the past days of rainfall and let the model help you to select the right parameters for that fitting. I am not an expert in TSA but I am sure that there are still more. You need to play around with some of them and figure out which ones go into the right direction by testing the models... $\endgroup$ – Fabian Werner Jun 10 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.