0
$\begingroup$

I’m wondering how to deal (and if I have to) with values on train that are not present on test and vice versa.

Let’s say that category1 on my train set can have one of these possible values: A,B,C,D and E; On my test set, I can have: C,D,E,F and G

Clearly you can see that “A and B” occur on train but do not occur on test and “F and G” occur on test but do not occur on train.

I’m wondering if my model (any model) would benefit from updating A and B to something like “not in test” so the algorithm won’t bother to find “logic” on train categories that can’t be applied on test (or at least will bother less because there is only one cat now instead of many). Not sure if updating F and G to “not in train” makes any sense as well.

Two extra points: If mentioned only two categories missing for simplicity, There are actually more than two but the row count of missing values is very small, like 100 out of 150.000 both ways.

$\endgroup$
1
$\begingroup$

First, just to make things clear,you should divide your samples from all categories between train and test, but I guess you can't do that or else you wouldn't ask.

As for your question, I think you should define a "neither" class for train and test as it will help classify samples which don't look like C,D or E. Then on test, samples F and G will be classified to this category and you can label them randomly there as F or G. That might be the best you can get out of it by means of accuracy.

$\endgroup$
3
  • $\begingroup$ are you sure it should be the same "neither" class for train and test? If we do so, the model will consider "F" and "G" on test the same as "A" and "B" on train (because we converted everything to "neither". I presume it should be two different categories. $\endgroup$
    – Diego
    Oct 18 '16 at 15:30
  • $\begingroup$ It will but why does it matter? Do you plan on seeing A and B later in the future? They just help you in the training phase by telling you how C,D or E doesn't look like so you can classify F and G better. $\endgroup$
    – Gabizon
    Oct 18 '16 at 16:14
  • $\begingroup$ I think it matter because the model may conclude things about F and G based on rows A and B on training data...and that doesn't sound right. it looks like two different labels make more sense or even just creating new labels on train as on test, F, G and the supposed new label would be the same thing as all of them do not exist on train. Thanks $\endgroup$
    – Diego
    Oct 19 '16 at 9:00
1
$\begingroup$

The goal of having a training set is not trying to see all the data, but capture the "trend / pattern" of the data.

For continuous case:

I can easily make up one example, that the training set and test set have no intersections, but the model is still successful. (black dots are training data and red triangle are testing data. the black line is the model).

enter image description here

par(cex=2)
set.seed(0)
x_train=runif(10)
y_train=3*x_train+0.5*runif(10)
x_test=runif(10)
y_test=3*x_test+0.5*runif(10)
plot(x_train,y_train)
points(x_test,y_test,col=2,pch=2)
abline(lm(y_train~x_train))
grid()

For discrete case (you example):

In general, the training data should be big and representative enough to see all the possible values. However, we may still have some exceptions. In such a case, we can use "Other" category to do the coding. And we should always model the "common values".

For example, in training set, the possible value is A:40%, B:20%, C:38%, D:1%, E:1%. It is better to code D, and E as "Others". The intuition is if we do not have sufficient data for certain observations, we may not able to model that correctly.

$\endgroup$
2
  • $\begingroup$ Dont you mean something like: A:1%, B:1%, C:38%, D:40%, E:20%? Because per my example, A and B are the values that are missing from the test set? $\endgroup$
    – Diego
    Oct 18 '16 at 15:29
  • 1
    $\begingroup$ @Diego Yes, He was just giving an example. $\endgroup$
    – Gabizon
    Oct 18 '16 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.