I am having a hard time making a decision with how to handle missing data under a specific set of circumstances and what it means to the model.

Consider that I have the following fictitious dataset.

import pandas as pd
year_built = [1920, 1962, 1999, 1972, 1987]
garage_type = ['DetSingle', None, 'DetDbl', 'AttTpl', None]
yr_garage_bld = [1920, None, 1999, None, None]
footage = [1876, 2002, 2436, 1583, 1409]
sale_price = [170000, 189000, 224999, 207432, 184000]
df = pd.DataFrame({'year_built':year_built,
                   'footage':footage, 'sale_price':sale_price})
      footage garage_type  sale_price  year_built  yr_garage_built
0     1876   DetSingle      170000        1920           1920.0
1     2002        None      189000        1962              NaN
2     2436      DetDbl      224999        1999           1999.0
3     1583      AttTpl      207432        1972              NaN
4     1409        None      184000        1987              NaN

Things I need help with are:

  1. For the garage_type feature I want to encode as categorical but, missing values have meaning and I want to maintain a sense of missingness (ie, None means there is not a garage). Should I just fill the None's with 'missing' then encode as categorical?
  2. For the yr_garage_built feature it makes sense to have a missing value (NaN) for the records that do not have a garage_type (ie, garage does not exist / has never been built). However, For the 4th entry a garage exists (AttTpl) and I have a missing yr_garage_built but, I will fill that with the year_built value (problem solved). My issue is, how do represent / maintain a sense of missing in yr_garage_built in the cases where garage_type indicates there is no garage knowing that maintaining NaN's are likely to cause a Linear Regression algorithm to error due to the presence of Nan?

The specific technologies I'm working with are Sci-Kit Learn's LinearRegressor and TensorFlow's estimator.DNNRegressor but, I'm more concerned about the decision process for dealing with missing values rather than implementation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.