Sampling Technique: Categorical Data, Many Levels

I have a data set that has a categorical variable with almost half the number of observations as categories.

Certain categories have only one observation.

A minimal reproducable example in R would look like:

set.seed(1)

# number of observations
n <- 100

# create factors, for simplicity just numbers
f <- factor(seq(1, n))

# the data frame contains the factors in the first column
# and the number of occurrences of that factor in the second column
df.test <- data.frame(fact = factor(seq(1, n)), n = numeric(n))

# sample from 1 to 4, for the number of occurrences of  each factor
for(i in 1:n){
df.test\$n[i] <- sample(1:4, 1, p = c())
}

df.test


For modeling purpose I want (have) to divide the dateset into a training and a test set. Now comes my problem:

To train the model it would make sense to sample the data such that an observation from each category is represented in the training set.

For example I found this article here with an implementation for that particular problem. However, having used my single occurring categories for the training, they are not available for testing.

Now, my suggestion would be to use the single occurring observations for training, just to have that information within my model.

But I do not see an optimal solution here. Either I "overfit" the training model by using up all my single occurring observations, or I use them strictly for testing and loose the information in the model.

EDIT As discussed in the comments the categories represent spacial data, however they are just numbered: area 1, area 2, ..., area n. Hence there is no information regarding the distance between the areas.

• Maybe take a step back, tell us why you have so many rare levels. Is the level set closed, or could novel levels arise with future data? What does your levels represent, in the real world? Is there some structure to the level set, so it can be modeled, somehow? Maybe you can use a random-effects model? See win-vector.com/2012/07/23/… which might have some ideas. – kjetil b halvorsen Jul 19 '20 at 23:58
• Hi, thx for the response. So the variable in question refers to a (numbered) name of a sample area and it's one of the primary question whether or not the variable is significant in explaining the response variable. Hence I have to incorporate it in the model. Anyway thx for the link. I had no idea that there was research done on this rather particular problem. However the problem here is two fold: once there is the high dimensionality of the data and second there is the low dimensionality of the occurrences. Got to read the literature though. Maybe it's answered somewhere in there. – user2550228 Jul 20 '20 at 5:18
• Are this sample areas contiguous? Then maybe resonses (or errors) in close areas would be correlated? Modeling such a spatial correlation and using random effects could maybe help. Consider add the tag spatial, and please, add this new information as an edit to the original post, not everybody reads comments! – kjetil b halvorsen Jul 20 '20 at 16:08