# In regression, what is the limit of independent variables?

After having taken the Coursera Data Science specialization, I am faced with my first "practical" problem which I plan on solving with some sort of regression. This is my first real world, business-problem application of what I've learned.

The plan is to predict customer attrition for our hotel/hospitality client by analyzing reservations (and the rate codes, room types, promotions, applied to the stays) and Recency, Frequency, Monetary-type values.

The way I understand it, I would set up my training data set for logistic regression (to predict the probability of attrition), and each independent variable needs to be continuous. So in the case that my independent variables are nominal in nature I would pivot those nominal values to be a series of columns with a value of either "0" or "1".

For example if I had three customers who made reservations with promo codes "FUN" "FAM" and "WKND" respectively, my training data set would look like 3 records with three additional columns, something like: "FUN_USED" , "FAM_USED" and "WKND_USED"

Is this a correct way of thinking? My concern is that if I need to pivot these independent variables to columns I could easily end up with hundreds of variables from which I build my prediction function.

Thanks for the help - I'm sure I violated some codes of conduct here, so please correct me where I've gone astray.

• I am confused by two things. First, I don't understand precisely what you mean by "pivoting ... a variable": could you explain what procedure that is? Second, where you assert that "each independent variable needs to be continuous," what kind of model are you talking about? Very few models indeed enforce this restriction--and logistic regression certainly is not among them. I am wondering whether you're really trying to ask about how to dummy-code nominal variables.
– whuber
Jan 22 '16 at 21:43
• I would also suggest, that the OP -- as a beginner -- try creating dummy variables using full rank design matrices to avoid confusion as this is his first logistic regression project. Instead of creating 3 dummy variable columns, resulting in a less than full rank design matrix, I'd suggest creating p-1 columns, where p = number of promo codes. While it's possible to obtain unique estimates of linear combinations from matrices that are less than full rank, starting out, it's usually easier to work with a full rank design matrix to get unique estimates of each promo codes effect. Jan 22 '16 at 22:27
• @whuber - My background is data warehousing, so I tend to view things in that context. Basically I would pivot the data from a row-based format (where each customer-stay record has a promo code) to a column format where each distinct promo code is represented by a column. That data is nominal, so upon pivoting it I would represent it with either a "0" (promo code not used) or a "1" (promo code used). The dummy-code recommendation was precisely what I was going after. Jan 24 '16 at 15:24

Correct with a small tag of revision. Dealing with categorical variables using many binary variables is indeed the way to model it, but the names you used in the question can be confusing to people who are familiar with statistics. And I'm referring to this sentence:

I would set up my training data set for logistic regression (to predict the probability of attrition), and each independent variable needs to be continuous.

Linear regression (logistic regression included) is suitable for independent variables that are continuous (weight in kg, rain fall in cm, etc.,) and categorical (color of eyes, highest education qualification attained, being a male versus female) as well. By saying "regression can only be used when the independent variables are continuous" will make a lot of statisticians scratch their heads because they have already jumped to "using binary indicators for categorical variables" as the approach. Deep down, yes, a binary indicator is just a special case of a continuous variable, but this bit of details are often accepted as is, rather than spelled out as a requirement.

The way you model your three types of code is correct. And if there are 100s of promo codes, you surely can end up with 100s of columns. But notice that codes that no one ever claimed (e.g. whole column is 0) and codes that everyone claimed (e.g. whole column is 1) are not usable as an independent variable because independent variables cannot be a constant. So, if you have 1000s of codes, but people only used 30 of them, the rest do not need to be included.

The technical limit of independent variables is (sample size - 2) but we are talking about a very bad case here. The "2" are spared for the regression intercept and the "unexplained" (also called error or residual) as their "degrees of freedom." In linear regression we do want the degrees of freedom for the residuals to be reasonably big. (Yet, how big is big involves the art of sample calculation and power calculation, which is a whole other barrel of worms.) So, basically don't go to town. I would recommend passing your suggested model to your instructor or teaching assistant for a quick check before fitting it if your online module provides such service.

A small caveat is that please make sure there are some people who had claimed no code. If everyone used a code and each can only use one code, the sum across all code columns will be 1. In that case, a phenomenon called "perfect collinearity" will happen and one of your binary indicators will need to be removed, and you'd let the intercept to pick up the job of capturing that probability. The one that you removed is commonly known as the "reference group." However, if there are at least one case who didn't use a code, then you can fit all of the code columns into the model without causing errors that are bad enough to stop the process.

• Thank you for your input. It helps me formulate a plan for this analysis, certainly. I might have misrepresented my situation, but now that I'm done with the course I am just about on my own as far as developing this solution! Maybe once I get a data created I'll use CV again to get some feedback. Jan 24 '16 at 15:15