How to handle large number of categories in a regression model? I have a dataset with 200,000 entries with four columns (time_of_day, order_size, time_taken, shop_number). I need to build a model and predict time_taken using the other three variables. There are more than 30,000 shop ids.
My approach has been to:
1. Try to use one_hot_encoding to encode each shop no. However, this leads to very large number of columns in my dataset.
2. Build a separate model for each shop assuming all shops have different efficiencies independent of one another.
Any other approach will be appreciated? Also, what type of model should I use in my data? I have tried simple linear regression and bayesian regression till now.
 A: This should be manageable, if a little slow, with a linear mixed model, e.g. in R with lme4:
lmer(time_taken ~ time_of_day*order_size + (time_of_day*order_size|shop_number),
     data=your_data)

A couple of advantages of LMMs:


*

*in most LMM computational frameworks the random-effects model matrix $Z$ is automatically constructed using a sparse model matrix, so 30K shop IDs is not a big problem

*the model is fitted with an empirical Bayes shrinkage estimator, so shops with less information are automatically adjusted closer to the population mean


If something like this works (you could start with a simpler model that uses (1|shop_number), i.e. only the intercept and not the effects of time of day and order size vary among shops) you might want to consider a more complex, additive model (e.g. the gamm4 package) that allows for more complicated patterns of variation with time of day and order size.
If you want to do this and R's lme4 is too slow you can try MixedModels.jl in Julia (or perhaps the commercial AS-REML, which is said to be very fast) (I'm not sure how SAS's speed compares).
