I am trying to use this dataset to build a predictive model.

The hubway.db file contains 3 tables. One of which is is bike_trips which has ~1 million entries. Some of my variables have high number of missing values:

zip_code: 340,438
birth_date: 697,931
gender: 340,372

Moreover these variables are fundamentally different. Gender can take 1 of 2 values but birthdate cannot, same with zip codes.

The 2 other tables contain bike_station and weather data. The end goal is to find underlying patterns and ultimately build a model to predict trip duration and final destination (from a given starting station). However I first need to clean the data, hence my issue.

How am I supposed to either impute or generally deal with this?

  • 1
    $\begingroup$ It depends on what you're trying to do with the dataset. "Working" is too vague to justify any answer at all. It also depends on why those variables are missing and what their pattern of missingness is. Could you edit your post to supply this additional information? $\endgroup$ – whuber Apr 8 '19 at 22:59
  • $\begingroup$ to echo @whuber, depends on your project. A first step would be a check if the data is missing at random (MAR), if this is the case then you can use any of the popular data imputation algorithms (AMELIA and missforest are my personal favorites) $\endgroup$ – V. Aslanyan Apr 8 '19 at 23:35

It depends on the data set.

Imputation using K-NN, Multivariate imputation by Chained equation, Stochastic regression, hot deck imputation, Extrapolation and Interpolation, Mean mode imputation, Probability likelyhood, many other techniques and based on our problem we need to choose.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.