1
$\begingroup$

My goal is to predict taxi demand depending on location and hour in NYC. I constructed a large dataset with ~19 million observations. However, it is computationally very expensive to perform modelling on such a big dataset. I am contemplating to reduce the dataset into sets according to tracts in NYC.

E.g.: There are 2166 tracts in NYC. A dataset for one specific tract would result in 8760 observations, which is (365*24) one observation for each hour of the day in a year.

What would be my disadvantages from such an approach?

Improve this question
$\endgroup$
3
  • $\begingroup$ Could you say a little bit about the nature of the observations you are making? Apparently they are some kind of summary by tract and hour, but summaries of what quantities? $\endgroup$
    – whuber ♦
    Commented Sep 4, 2018 at 14:18
  • 1
    $\begingroup$ I am measuring taxi demand according to pickups in a certain tract and hour. So pickups is my target variable. Furthermore, I added data about population, weather, and so on according to the tracts. $\endgroup$
    – vranjes
    Commented Sep 4, 2018 at 14:21
  • $\begingroup$ Apparently, I would have to do the modelling for each tract specifically (2166 times). Which I could perform through parallelisation. But what I am worried about is that important information gets lost. $\endgroup$
    – vranjes
    Commented Sep 4, 2018 at 14:25

1 Answer 1

Reset to default
1
$\begingroup$

If you select the tracts by hand you run the risk of accidentally selecting non-representative samples. I think this is likely - even living in NYC, I would not be able to select "normal" areas. There may not even be such a thing as "normal".

Most people I know who work with this specific data set start by taking a random subsample of the data and storing it for some preliminary exploration, creation of readable figures in their paper, etc. (On that note, I'd like to hear more about what you're doing with demand and the taxi data, but that is up to you. My email is linked in my profile.)

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for your input RegressForward. The thing is, I would divide the complete dataset by tracts and then create a model on the observed data of each tract. In the end I would have a model for each tract predicting the taxi demand for the given tract. $\endgroup$
    – vranjes
    Commented Sep 5, 2018 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.