Methods to work around the problem of missing data in machine learning

Question

Virtually any database we want to make predictions using machine learning algorithms will find missing values ​​for some of the characteristics.

There are several approaches to address this problem, to exclude lines that have missing values ​​until they fill with the mean values ​​of the characteristics.

I would like to use for a somewhat more robust approach, which would basically run a regression (or another method) where the dependent variable (Y) would be each of the columns that have missing values ​​but only with the rows of the table that contain all the data , and predict the missing values ​​with this method, complete the table by the table and move to the next 'column' with missing values ​​and repeat the method until everything is filled.

But that gives me some doubts.

Why any column start? I believe that the one with the smallest missing values ​​until the one with the most

Is there any threshold of missing values ​​that is not worth trying to complete it? (for example, if this characteristic only has 10% of the values ​​filled would not it be more interesting to exclude it)

Is there any kind of implementation in traditional packages or other methods that are robust to missings?

Answer 1

The technique you describe is called imputation by sequential regressions or multiple imputation by chained equations. The technique was pioneered by Raghunathan (2001) and implemented in a well working R package called mice (van Buuren, 2012).

A paper by Schafer and Graham (2002) explains well why mean imputation and listwise deletion (what you call line exclusion) usually are no good alternatives to the above mentioned techniques. Principally mean imputation is not conditional and thus can bias the imputed distributions towards the observed mean. It will also shrink the variance, among other undesirable impacts on the imputed distribution. Furthermore, listwise deletion indeed will only work if the data are missing completely at random, like by the flip of a coin. Also it will increase the sampling error, as the sample size is reduced.

The authors quoted above usually recommend starting with the variable featuring the least missing values. Also, the technique is usually applied in a Bayesian way (i.e. an extension of your suggestion). Variables are visited more often in the imputation procedure, not only once. In particular, each variable is completed by draws from its conditional posterior predictive distribution, starting with the variable featuring least missing values. Once all variables in a data set have been completed, the algorithm again starts at the first variable and then re-iterates until convergence. The authors have shown that this algorithm is Gibbs, thus it usually converges to the correct multivariate distribution of the variables.

Usually, because there are some untestable assumptions involved, in particular missing at random data (i.e. whether data are observed or not depends on the observed data only, and not on the unobaserved values). Also the procedures can be partially incompatible, which is why they have been called PIGS (partially incompatible Gibbs sampler).

In practice Bayesian multiple imputation is still a good way to deal with multivariate non-monotone missing data problems. Also, non-parametric extensions such as predictive mean matching help to relax regression modeling assumptions.

---

Raghunathan, T. E., Lepkowski, J., van Hoewyk, J., & Solenberger, P. (2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27(1), 85–95.

Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147

van Buuren, S. (2012). Flexible Imputation of Missing Data. Boca Raton: CRC Press.

Answer 2

I did not find anything that solved my problem so I wrote a function that mixes some solutions to a Pandas dataframe with missing numerical values (with fancyimpute) and categorical (with a random forest).

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
import fancyimpute as fi

def separe_numeric_categoric(df):
    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
    df_n = df.select_dtypes(include=numerics)
    df_c = df.select_dtypes(exclude=numerics)
    print(f'The DF have {len(list(df_n))} numerical features and {len(list(df_c))} categorical fets')
    return df_n, df_c


def find_missing(df):
    total = df.isnull().sum().sort_values(ascending=False)
    percent = (df.isnull().sum()/df.isnull().count()).sort_values(ascending=False)
    filter(lambda x: x>=minimum, percent)
    return percent


def count_missing(df):
    missing = find_missing(df)
    total_columns_with_missing = 0
    for i in (missing):
        if i>0:
            total_columns_with_missing += 1
    return total_columns_with_missing


def remove_missing_data(df,minimum=.1):
    percent = find_missing(df)
    number = len(list(filter(lambda x: x>=(1.0-minimum), percent)))
    names = list(percent.keys()[:number])
    df = df.drop(names, 1, errors='ignore')
    print(f'{number} columns exclude because haven`t minimium data.')
    return df


def one_hot(df, cols):
    for each in cols:
        dummies = pd.get_dummies(df[each], prefix=each, drop_first=False)
        df = pd.concat([df, dummies], axis=1)
    df = df.drop(cols, axis=1)
    return df



def impute_missing_data(df,minimium_data=.1):
    columns_missing = count_missing(df)
    print(f'Total columns with missing values: {count_missing(df)} of a {len(list(df))} columns in df')

    # remove features without minimium size of information
    df = remove_missing_data(df,minimium_data)

    numerical_df, categorical_df = separe_numeric_categoric(df)

    # Autocomplete using MICE for numerical features.
    try:
        df_numerical_complete = fi.MICE(verbose=False).complete(numerical_df.values)
        n_missing = count_missing(df)
        print(f'{columns_missing-n_missing} numerical features imputated')

        # Complete the columns name.
        temp = pd.DataFrame(columns=numerical_df.columns, data=df_numerical_complete)

        # df temp com os dados numericos completados e os categóricos.
        df = pd.concat([temp, categorical_df], axis=1)

    except Exception as e:
        print(e)
        print('Without Missing data in numerical features')

    missing = find_missing(df)
    names = missing.keys()
    n = 0
    for i, c in enumerate(missing):
        if c > 0:
            col = names[i]
            print(f'Start the prediction of {col}')
            clf = RandomForestClassifier()
            le = LabelEncoder()
            ## inverter a ordem da predição das categóricas pode melhorar a precisao.
            categorical_train = list(categorical_df.loc[:,categorical_df.columns != col])

            temp = one_hot(df,categorical_train)
            df1 = temp[temp[col].notnull()]
            df2 = temp[temp[col].isnull()]
            df1_x = df1.loc[:, df1.columns != col]
            df2_x = df2.loc[:, df1.columns != col]

            df1_y = df1[col]
            le.fit(df1_y)
            df1_y = le.transform(df1_y)
            clf.fit(df1_x, df1_y)
            df2_yHat = clf.predict(df2_x)
            df2_yHat = le.inverse_transform(df2_yHat)
            df2_yHat = pd.DataFrame(data=df2_yHat, columns=[col])
            df1_y = le.inverse_transform(df1_y)
            df1_y = pd.DataFrame(data=df1_y,columns=[col])

            df2_x.reset_index(inplace=True)   
            result2 = pd.concat([df2_yHat, df2_x], axis=1)
            try:
                del result2['index']
            except:
                pass

            df1_x.reset_index(inplace=True)
            result1 = pd.concat([df1_y, df1_x], axis=1)
            try:
                del result1['index']
            except:
                pass

            result = pd.concat([result1, result2])
            result = result.set_index(['Id'])
            df.reset_index()            
            try:
                df.set_index(['Id'],inplace=True)
            except:
                pass
            df[col] = result[col]

            n += 1

    print(f'Number of columns categorical with missing data solved: {n}')

    return df


df = impute_missing_data(df)

Answer 3

Though typically more involved, you can try and created a Maximum Entropy Distribution based on what data you have.

http://proceedings.mlr.press/v5/huang09a/huang09a.pdf