4
$\begingroup$

I have two data sets

  1. Train data
  2. Test data (with no dependent variable values but I have data on independent variable or you can say I need to forecast).

Using the training data (which has some NAs in the cell) I performed ordinary least square regression (OLS) using lm() in R and fitted the model & I got the $\beta $ coefficients of the regression model. (All is good so far!)

Now, in the process of prediction for the fitted values, I have some missing values for some cells in the test dataset. I used function predict() as follows:

 predict(ols, test_data.df, interval= "prediction", na.action=na.pass)

for the cell (or cells) with NA value the entire row is discarded in generating the output (yhat). Is there any function that could generate the yhat values (other than NAs) for the test data without discarding any rows with missing value in the cell.

$\endgroup$
8
$\begingroup$

First, let me preface this by stating that missing data is its own specialty in statistics, so there's lots and lots of different answers to this question.

As you've discovered, by default, R uses case-wise deletion of missing values. This means that whenever a missing value is encountered in your data (on either side of your regression formula), it simply ignores that row. This isn't great, since if you have 100 observations, but half of your rows has at least one variable value missing, you effectively have 50 observations. In some disciplines, the prevalence of missing data can rapidly diminish the size of your data. When I was an undergraduate, I analyzed a 3,000-person survey which shrank to just 316 people when using case-wise deletion!

But this gets even worse than shrinking your sample size: there may be hidden problems, such as an association between the pattern of missingness and the value of the missing element. For example, people with higher income are more likely to not disclose their salary. This will make it difficult to conduct meaningful, statistically sound judgments related to income.

One common method for dealing with missing values is imputation. There are many packages for imputation in R available. In my specialty area, political science, a widely-used one is AMELIA II, by Gary King. This treats your variables as multivariate normal and iteratively improves its "guesses" of what the missing values must be based on some convergence criteria: when convergence is declared when the "guess" seems to fit well with the rest of the data. (I'm sorry that this is nonspecific. I haven't used AMELIA II in several years. The documentation is thorough and lucidly written, so I would start there.)

But this is just one option. I'm sure that more knowledgeable people will speak up with their contributions.

$\endgroup$
  • $\begingroup$ Thanks for the clarification and nice edit of the post. I am going to check AMELIA II package for dealing with missing data. $\endgroup$ – Chandra May 1 '14 at 17:14
  • $\begingroup$ Glad that I could help. And good luck! $\endgroup$ – Sycorax May 1 '14 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.