handling many missing values within a regression

Question

I am struggling with answering a question on how i should handle the vast number of NAs in my data. It is a behavioural study of the impact of traffic on certain mammals and i have approximately 500 rows (one for each type of vehicle) across 3 years. However, there is variable stating if there was a 'behavioural response' or not (Y/N). This I had planned to use in a 'binary logistic regression' as the response variable. However, many of the rows indicate no response and therefore the continuous variables in the dataset (distance to vehicle etc..) have not been filled in, leaving many blanks!

Should I be performing the regression based on a small subset of the data e.g. fewer rows? and if so does this have to only be the complete rows. Any help on how to proceed with this problem would be appreciated

Answer 1

I believe what you want to do is perform data imputation. Here is a good quick (16 pages) pdf on imputation from Columbia.

Generally if you have a large enough set of data and your NAs/NANs account for ~10% of your data, you can simply remove the affected rows. If removing data will not work for you then you should look into imputation. Simple approaches include taking the average of the column and use that value, or if there is a heavy skew the median might be better. A better approach, you can perform regression or nearest neighbor imputation on the column to predict the missing values. Then continue on with your analysis/model.

Another approach would be to build a RandomForest classifier. RandomForest models can neutrally deal with missing data by ignoring them when deciding splits. Berkeley has a good write up on RandomForests. If you choose to go down this road there is also a good paper discussing NAs in tree based models: An Investigation of Missing Data Methods for Classification Trees Applied to Binary Response Data by Ding and Simonoff.

If you are using python, the Scipy library has an interpolation function which produces data points from within a range of known discrete data points. This is another way to fill in missing data.

Hope this helps!

Answer 2

We have to divide between two cases

1.) The NA values of the response do not occur systematically:

Without a value of your target/response variable $Y$ any given row becomes useless for binary logistic regression or most other supervised classification algorithm. Supervised means that your examples must be labeled while unsupervised algorithms do not use the label information.

In your situation you can use those rows without the behavioural response in unsupervised preprocessing / feature generation techniques.

As an example you could take all given samples of your independent variables $X_1, \ldots, X_k$ to calculate principal component analysis (PCA) components $PC_1,\ldots,PC_k$. Then later you could train your bbinary logistic regression on a subset of those components $PC_1,\ldots, PC_l$ with $l \le k$.

Why is this a good idea? According to wikipedia https://en.wikipedia.org/wiki/Principal_component_analysis

This transformation is defined in such a way that the first principal component has the largest possible variance (that is, accounts for as much of the variability in the data as possible), and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to the preceding components.

So by taking the first principal components you maintain most of the variance in your samples but you reduce the number of variables and with it the chance of overfitting. Further the information contained in the response less samples is also included.

2.) The NA values of the response do not occur systematically

Here further analysis is necessary. You could try to train a classifier that predicts if the response is NA or not which results in the two classes $C_2=\{Y\neq NA \}$ versus $C_2=\{Y==NA \}$. If you can predict the NA response values than this is not a two classes problem $Y/N$ but a three classes classification taks $Y/N/NA$.