Virtually any database we want to make predictions using machine learning algorithms will find missing values ​​for some of the characteristics.

There are several approaches to address this problem, to exclude lines that have missing values ​​until they fill with the mean values ​​of the characteristics.

I would like to use for a somewhat more robust approach, which would basically run a regression (or another method) where the dependent variable (Y) would be each of the columns that have missing values ​​but only with the rows of the table that contain all the data , and predict the missing values ​​with this method, complete the table by the table and move to the next 'column' with missing values ​​and repeat the method until everything is filled.

But that gives me some doubts.

Why any column start? I believe that the one with the smallest missing values ​​until the one with the most

Is there any threshold of missing values ​​that is not worth trying to complete it? (for example, if this characteristic only has 10% of the values ​​filled would not it be more interesting to exclude it)

Is there any kind of implementation in traditional packages or other methods that are robust to missings?

  • 6
    $\begingroup$ The term of art you're looking for is "imputation," of which multiple imputation is a popular, modern choice. Note that excluding observations with missing observations or replacing missing observations with the mean can badly bias the data. One place to start is Gelman et al, Bayesian Data Analysis 3rd Edition, "Chapter 18: Models for Missing Data." $\endgroup$
    – Sycorax
    Sep 18, 2017 at 15:51
  • $\begingroup$ Thanks for the tip, I'll search with that term and look at the cap18. Deleting lines can bias the model a lot (if the missings are not random, which is very likely) and placing the average can put a strong 'inertial load' around the mean, also depending on the exogeneity of the data missings. My big question is the best approach to handle this and my suggestion would be to run pre-regressions to complete the data before the main regression (is there any packages that do this or should I create one?) $\endgroup$
    – sn3fru
    Sep 18, 2017 at 16:10
  • 1
    $\begingroup$ Modern multiple imputation estimates a model for missing and non-missing data side-by-side. The Bayesian take on missing data is to estimate a distribution over the missing data, conditional on the observed data and the model for missingness. Statistical software in python leaves much to be desired. For TSCS data, Amelia II in R is a solid choice. Or you could roll your own using stan. $\endgroup$
    – Sycorax
    Sep 18, 2017 at 16:14

3 Answers 3


The technique you describe is called imputation by sequential regressions or multiple imputation by chained equations. The technique was pioneered by Raghunathan (2001) and implemented in a well working R package called mice (van Buuren, 2012).

A paper by Schafer and Graham (2002) explains well why mean imputation and listwise deletion (what you call line exclusion) usually are no good alternatives to the above mentioned techniques. Principally mean imputation is not conditional and thus can bias the imputed distributions towards the observed mean. It will also shrink the variance, among other undesirable impacts on the imputed distribution. Furthermore, listwise deletion indeed will only work if the data are missing completely at random, like by the flip of a coin. Also it will increase the sampling error, as the sample size is reduced.

The authors quoted above usually recommend starting with the variable featuring the least missing values. Also, the technique is usually applied in a Bayesian way (i.e. an extension of your suggestion). Variables are visited more often in the imputation procedure, not only once. In particular, each variable is completed by draws from its conditional posterior predictive distribution, starting with the variable featuring least missing values. Once all variables in a data set have been completed, the algorithm again starts at the first variable and then re-iterates until convergence. The authors have shown that this algorithm is Gibbs, thus it usually converges to the correct multivariate distribution of the variables.

Usually, because there are some untestable assumptions involved, in particular missing at random data (i.e. whether data are observed or not depends on the observed data only, and not on the unobaserved values). Also the procedures can be partially incompatible, which is why they have been called PIGS (partially incompatible Gibbs sampler).

In practice Bayesian multiple imputation is still a good way to deal with multivariate non-monotone missing data problems. Also, non-parametric extensions such as predictive mean matching help to relax regression modeling assumptions.

Raghunathan, T. E., Lepkowski, J., van Hoewyk, J., & Solenberger, P. (2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27(1), 85–95.

Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147

van Buuren, S. (2012). Flexible Imputation of Missing Data. Boca Raton: CRC Press.

  • 1
    $\begingroup$ excellent response, on the one hand I am glad to have advanced at least the direction that I must follow, on the other hand I am sad not to have a genial approach that I did not think. On the interactive prediction of missing data by the bayes method, how could I reproduce something like this in python? Is it a regression too? and after predicting all the possible missing data, should I go over the predictor so that the new data also participates in that prediction? Many thanks for the help, I believe it will benefit many others. $\endgroup$
    – sn3fru
    Sep 18, 2017 at 19:56
  • 1
    $\begingroup$ @sn3fru Well, these questions are answered in the references, among other places. I am not aware if a Python implementation exists, but replicating it should not be too difficult. I suppose it would require studying the details of the algorithm a bit. In general any Bayesian model can be used to create multiple imputes, but the mice algorithm either uses regression or predictive mean matching. You initially complete the missing data by draws from the observed distribution and then impute sequentially. Once finished you repeat, but using the newly imputed values. The new data participates,yes $\endgroup$
    – tomka
    Sep 18, 2017 at 20:31

I did not find anything that solved my problem so I wrote a function that mixes some solutions to a Pandas dataframe with missing numerical values (with fancyimpute) and categorical (with a random forest).

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
import fancyimpute as fi

def separe_numeric_categoric(df):
    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
    df_n = df.select_dtypes(include=numerics)
    df_c = df.select_dtypes(exclude=numerics)
    print(f'The DF have {len(list(df_n))} numerical features and {len(list(df_c))} categorical fets')
    return df_n, df_c

def find_missing(df):
    total = df.isnull().sum().sort_values(ascending=False)
    percent = (df.isnull().sum()/df.isnull().count()).sort_values(ascending=False)
    filter(lambda x: x>=minimum, percent)
    return percent

def count_missing(df):
    missing = find_missing(df)
    total_columns_with_missing = 0
    for i in (missing):
        if i>0:
            total_columns_with_missing += 1
    return total_columns_with_missing

def remove_missing_data(df,minimum=.1):
    percent = find_missing(df)
    number = len(list(filter(lambda x: x>=(1.0-minimum), percent)))
    names = list(percent.keys()[:number])
    df = df.drop(names, 1, errors='ignore')
    print(f'{number} columns exclude because haven`t minimium data.')
    return df

def one_hot(df, cols):
    for each in cols:
        dummies = pd.get_dummies(df[each], prefix=each, drop_first=False)
        df = pd.concat([df, dummies], axis=1)
    df = df.drop(cols, axis=1)
    return df

def impute_missing_data(df,minimium_data=.1):
    columns_missing = count_missing(df)
    print(f'Total columns with missing values: {count_missing(df)} of a {len(list(df))} columns in df')

    # remove features without minimium size of information
    df = remove_missing_data(df,minimium_data)

    numerical_df, categorical_df = separe_numeric_categoric(df)

    # Autocomplete using MICE for numerical features.
        df_numerical_complete = fi.MICE(verbose=False).complete(numerical_df.values)
        n_missing = count_missing(df)
        print(f'{columns_missing-n_missing} numerical features imputated')

        # Complete the columns name.
        temp = pd.DataFrame(columns=numerical_df.columns, data=df_numerical_complete)

        # df temp com os dados numericos completados e os categóricos.
        df = pd.concat([temp, categorical_df], axis=1)

    except Exception as e:
        print('Without Missing data in numerical features')

    missing = find_missing(df)
    names = missing.keys()
    n = 0
    for i, c in enumerate(missing):
        if c > 0:
            col = names[i]
            print(f'Start the prediction of {col}')
            clf = RandomForestClassifier()
            le = LabelEncoder()
            ## inverter a ordem da predição das categóricas pode melhorar a precisao.
            categorical_train = list(categorical_df.loc[:,categorical_df.columns != col])

            temp = one_hot(df,categorical_train)
            df1 = temp[temp[col].notnull()]
            df2 = temp[temp[col].isnull()]
            df1_x = df1.loc[:, df1.columns != col]
            df2_x = df2.loc[:, df1.columns != col]

            df1_y = df1[col]
            df1_y = le.transform(df1_y)
            clf.fit(df1_x, df1_y)
            df2_yHat = clf.predict(df2_x)
            df2_yHat = le.inverse_transform(df2_yHat)
            df2_yHat = pd.DataFrame(data=df2_yHat, columns=[col])
            df1_y = le.inverse_transform(df1_y)
            df1_y = pd.DataFrame(data=df1_y,columns=[col])

            result2 = pd.concat([df2_yHat, df2_x], axis=1)
                del result2['index']

            result1 = pd.concat([df1_y, df1_x], axis=1)
                del result1['index']

            result = pd.concat([result1, result2])
            result = result.set_index(['Id'])
            df[col] = result[col]

            n += 1

    print(f'Number of columns categorical with missing data solved: {n}')

    return df

df = impute_missing_data(df)
  • $\begingroup$ Nice, this may help others (I did not check it) - it may also be interesting for you to contact the creator of the R function mice, Stef van Buuren. He may be interested in your Python code and/or point you to other people's work in this regard. stefvanbuuren.nl $\endgroup$
    – tomka
    Sep 28, 2017 at 12:45
  • $\begingroup$ I do not know if they would be interested in something so simple, I'm just sharing here as it can help other people needing solving missing in a Pandas dataframe. $\endgroup$
    – sn3fru
    Sep 28, 2017 at 12:49
  • $\begingroup$ Well they may be interested in implementing it in Python in general and they may know whether somebody has already done it. I have contacted Stef before and he is very responsive and helpful. If there is a Python implementation it may also be useful to share it here under this thread. See e.g. pypi.python.org/pypi/fancyimpute/0.0.4 $\endgroup$
    – tomka
    Sep 28, 2017 at 12:50

Though typically more involved, you can try and created a Maximum Entropy Distribution based on what data you have.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.