What happens if a dummy variable is 0 or 1 in the training set, but always 1 in the test set I have regression models (linear, lasso and Random forest), and there is a feature that is 0 or 1 in the training set (same number of occurrences approximately) and has a small effect on the target outcome variable (when is 1, the value of the target variable is slightly less).
However, in the test set, this feature always has a value of 1.
Since I've added that feature, my models perform way less good. I am wondering what is happening ?
Thank you
 A: What happens if your target variable contains only ones in the test sample, is that you do not learn anything about performance of your model in cases where your target is zero. Simple example: "always return 1" model will have 100% accuracy and very good results on other error metrics, but many of those performance metrics would be quite useless if you know that your test sample didn't contain any zeros. You wouldn't learn about performance of your model in terms of false positives. If choosing your model based on test set metrics, you would tend to choose models that predict more ones in total.
If your features take only ones in the test sample, then it is not fully representative for your data. If your dataset is small, then it is possible that it does not represent all the possible combinations of your features, but it should be  reasonably representative for the real-life data. The good thing is that it's a known-unknown, so you know what you don't know about the model's performance. The bad thing is that you still don't know how does your model behaves in some scenarios. What you can do is to


*

*get more data for your test set (best option if available),

*move (preferably random) part of your training sample that contains zeros to the test sample,

*use $k$-fold cross-validation (especially if your sample is relatively small), i.e. split your data randomly for $k$ equal parts and for $i=1,...,k$ iterations, in each case take the $i$-th part as your test sample and use the rest of the parts as your test sample. By using randomized procedure you make it more likely to have balanced test sets.


If you can't obtain more data, or resample your data, then you should at last acknowledge in your report that performance of your model is unknown in such-and-such scenario.
