4
$\begingroup$

I wrote my own cross validation function for model output in R (lm, glm and so on; named MOD_k_fold_r2 in my R package). It works for nominal predictors as well. In a few rare cases, a level of the nominal predictors does not appear in the training data, but does appear in the test data. When this happens, the predict function throws an error because it doesn't know which beta to assign to this new level and therefore it can't predict the Y values for the cases.

Is there any standard way to deal with this? Perhaps assign the beta a 0 coefficient manually? This would require scoring the new cases manually instead of using the predict function I think. Currently, I am instead just disregarding these runs that produce errors and rely on the remaining runs. This is a rare problem for most datasets.

Update

Proposals so far:

  • Skip runs that use test data that have levels that do not appear in the training data.
  • Use stratified sampling to make sure that all training datasets include at least one case with each level.
  • If levels are missing, then insert pseudocases with arbitrary datapoints, e.g. mean of all values.
$\endgroup$
  • 2
    $\begingroup$ One way to avoid this is to deliberately form the training set such that all levels of each nominal predictor are represented at least once in the training set. I have no idea what the "standard" way of dealing with this is though. $\endgroup$ – Jake Westfall May 22 '16 at 20:10
  • $\begingroup$ @JakeWestfall: "stratification" should be a start point to look this up. $\endgroup$ – cbeleites May 31 '16 at 7:05
1
$\begingroup$

If you are doing one hot encoding without a reference level, then set all encoded features of that test instance to zero. I don't know if that's common practice, but it sounds logical.

If it's an ordered factor, then you will probably transform it to a numeric feature, and the transformation would probably still applies for new levels.

$\endgroup$
1
$\begingroup$

You could look into stratified sampling, i.e. constraining your train/test splits so that they have (approximately) the same relative frequencies for your predictor levels.

However, I think it worth considering whether the current behavior is actually wanted: So random splitting with non-negligible frequency results in sets that do not cover all predictor levels. Can you consider such a set representative for whatever the application is?
I've been working with such small sample sizes and went for stratified splitting. But I insist that thinking hard about the data and the consequences of working with such small samples is at least as necessary as fixing the pure computational error.

$\endgroup$
0
$\begingroup$

You definitely want to warn the user when this happens. But if robustness and ease of use are important, you might consider a preprocessing step that changes the missing nominal values to an arbitrary value that was in the training data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.