1
$\begingroup$

I am having a hard time making a decision with how to handle missing data under a specific set of circumstances and what it means to the model.

Consider that I have the following fictitious dataset.

import pandas as pd
year_built = [1920, 1962, 1999, 1972, 1987]
garage_type = ['DetSingle', None, 'DetDbl', 'AttTpl', None]
yr_garage_bld = [1920, None, 1999, None, None]
footage = [1876, 2002, 2436, 1583, 1409]
sale_price = [170000, 189000, 224999, 207432, 184000]
df = pd.DataFrame({'year_built':year_built,
                   'garage_type':garage_type, 
                   'yr_garage_built':yr_garage_bld,
                   'footage':footage, 'sale_price':sale_price})
df
      footage garage_type  sale_price  year_built  yr_garage_built
0     1876   DetSingle      170000        1920           1920.0
1     2002        None      189000        1962              NaN
2     2436      DetDbl      224999        1999           1999.0
3     1583      AttTpl      207432        1972              NaN
4     1409        None      184000        1987              NaN

Things I need help with are:

  1. For the garage_type feature I want to encode as categorical but, missing values have meaning and I want to maintain a sense of missingness (ie, None means there is not a garage). Should I just fill the None's with 'missing' then encode as categorical?
  2. For the yr_garage_built feature it makes sense to have a missing value (NaN) for the records that do not have a garage_type (ie, garage does not exist / has never been built). However, For the 4th entry a garage exists (AttTpl) and I have a missing yr_garage_built but, I will fill that with the year_built value (problem solved). My issue is, how do represent / maintain a sense of missing in yr_garage_built in the cases where garage_type indicates there is no garage knowing that maintaining NaN's are likely to cause a Linear Regression algorithm to error due to the presence of Nan?

The specific technologies I'm working with are Sci-Kit Learn's LinearRegressor and TensorFlow's estimator.DNNRegressor but, I'm more concerned about the decision process for dealing with missing values rather than implementation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.