Dealing with new Factor Levels in a Regression in R

Question

I originally posted this in stackoverflow (as given here) but was told to try here since it might be more relevant here. I am very new to statistics and R in general so my question might be a bit dumb, but since I cannot find my solutions online I thought I should try ask it here.

I have a data frame dataset of a whole lot of different variables very similar to as follows:

 Item | Size   | Value    | Town
----------------------------------
A     |  10    |   800   | 1
B     |  11    |   100   | 2
A     |  17    |   900   | 2
D     |  13    |   200   | 3
B     |  15    |   500   | 1
C     |  12    |   250   | 3
E     |  14    |    NA   | 2
A     |        |   800   | 1
C     |        |   800   | 2

Basically, I have to try and 'guess' the Size based on the type of Item, it's Value, and the Town it was sold in, so I think a regression method would be a good idea.

I now use a regression as follows:

lm(Size ~ factor(Item) + factor(Town) + Value,...)

The problem however, occurs when I try and predict the Size using this model. I have the following lines of code:

pmodel <- lm(Size ~ factor(Item) + factor(Town) + Value,...)
prediction <- predict(pmodel, dataset2)

(where dataset2 is a subset of dataset which has all the empty "Size" values which I want to predict)

But this now comes up with the error:

Error in model.frame.default(Terms, newdata, na.action=na.action, xlev = object$xlevels): factor factor(Town) has new levels

Is there any way around this, or is there any way I can get the model to make prediction based on the other values if it encounters a 'level' which is not in the original dataset?

Answer 1

You have a factor variable Town. Assuming that it represents different town's, you have an estimated model and now wants predictions for towns not in the original sample.

That is a natural want, but then you need a model which can be used that way. A usual linear model lm that you have used, with Town as a nominal factor, estimates a separate parameter for each town. To make predictions for your new town, it needs a parameter that was not estimated.

There are multiple solutions, but the principled way is to model the effects of towns. Maybe you can get covariates describing towns, like population, average income, % unemployment, ... and include them in the model. If that works well, maybe there is no need for the Town factor in the model.

But it could still be used, but now as a random effect, which would result in unseen towns giving predictions with higher standard error.

Answer 2

Since you have not provided a reproducible example, all we can really offer is to make an educated guess of the cause of the problem based on the error message. The error message here is telling you that factor(Town) has new levels. This probably means that dataset2 includes values of the Town variable that were not present in dataset.

Remember that factor variables are treated in regression as categorical variables ---i.e., they go into the design matrix for the regression as a set of indicator variables. Hence, if you fit your model with an initial set of towns (treated as a factor variable, as it should be) then the model will estimate coefficients only for the towns that are in the data for the model. If you then try to use that model to predict the response variable for an new town, there is no term in the model to allow that new value of that factor variable.

Answer 3

I'm not exactly sure what you are trying to accomplish.

If you want to "fill up" the missing values under "Size" so that you can use the data you have in the other columns of those rows for further calculations, you can impute the missing values as Roland has suggested in his comment. What method of imputation would be most appropriate for your data cannot be answered based on the incomplete information you have given.

But if you want (no more than) to predict the missing values under "Size" from the information you have (and do not want to perform any further calculations with the thus completed dataset), I would:

1. Subset the rows where "Size" has a value.
2. Calculate the predictors, as you have done, but from that subset.
3. Apply the prediction to the subset of rows with a missing size value, as you have done.

But you cannot then go on and use these filled up rows to predict the missing value in the "Value" column! If you did that, your predictions would become ever more unlikely.

For example, you can predict the body height of a child from their age, but if you then use the predicted body height to predict that persons's shoe size, the probability that your prediction is correct is pretty low, despite the fact that body height and shoe size are correlated. But since children of the same age can be of differing sizes, and since people with the same body height can have different shoe sizes, using age to predict shoe size from a predicted age increases the possible error in your estimation.