2
$\begingroup$

This question already has an answer here:

I am running multinomial logistic regression Model for rating Hotels.

I have a variable called CHAIN ID which is a bunch of numbers but there are too many such IDs, maybe around 1000. I want to include this variable in my model.

I have tried that but R is not responding.

Is there any way to deal with these many levels of that variable?

$\endgroup$

marked as duplicate by kjetil b halvorsen, Michael Chernick, Momo, John, Peter Flom regression May 18 '17 at 11:22

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ What is it that you have tried in R? $\endgroup$ – Winks Apr 1 '16 at 10:39
  • $\begingroup$ mllogit package to run regression $\endgroup$ – Milan Amrut Joshi Apr 1 '16 at 10:45
  • $\begingroup$ Is there any way to deal with huge levels in variables $\endgroup$ – Milan Amrut Joshi Apr 1 '16 at 10:46
2
$\begingroup$

Massively categorical variables like this require special handling simply due to the memory (RAM) required for inversion of a large cross-products matrix (assuming that's what is needed here). Depending on the installation and module, R software can be quite limited in how many levels it can invert. There are workarounds, depending on the hardware platform and software you have available.

Here's a link to the best paper I'm aware of about a Bayesian multilevel model for crunching massively categorical variables. Regrettably, I'm not aware of any R modules that do this but they probably exist somewhere. It would be readily programmable in R, Stan or Winbugs -- depending on which software you're most comfortable with:

http://www.people.hbs.edu/tsteenburgh/articles/Steenburgh_Ainslie_and_Engebretson_(winter_2003).pdf

Bayes may not be your preferred approach. Here are a couple of other things to try:

1) 1,000 levels really isn't that many, not when you think of a truly massive variable like zip codes which can contain 40,000 levels or more (if business zips are included). Other software is able to crunch this many levels (and more) as standard practice on today's laptops. Given that, there must be an R module which can do classic matrix inversion for a few thousand levels. I don't know what that module is.

2) Gain access to a massively parallel platform and use a "divide and conquer" approach to slicing your data up into many smaller datasets, distributing them across the core. The goal is to size them appropriately for the memory capacities of each CPU. Then, on the backend, roll them up to summarize the test statistics that are needed. Here's a link to a paper about this "D&C" methodology:

http://dimacs.rutgers.edu/TechnicalReports/TechReports/2012/2012-01.pdf

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.