Removing attributes with few observations in R

Question

I have roughly 15 variables / attributes characterizing 6k customers in my data set. As they are categorical I have transformed them into 1 attribute for each possible value (1-out-of-K coding). An example could be Region with values "A", "B" and "C", which is transformed into 3 variables: Region_A, Region_B and Region_C. The same goes for other variables such as the Sales Channel. After this transformation I now have around 70 attributes.

I would like to examine if there are any significant 2-way interactions between the different variables with regards to a response variable (concerning customer quality) using logistic regression. For instance, it is interesting to see if there is an interaction between Region_A and Sales Channel 1. However, there are very many possible interactions and therefore I would like to start by removing some variables, which have very few observations connected to them. An example could be that only 3 customers come from Region_A.

More specifically, I would start by removing all attributes that have 5 observations or less connected to them (out of 6k observations). However, I cannot find out how to do that. Thus I have the following questions:

1. Does my thinking make sense? Or should I approach the issue in another way?

2. How do I remove all attributes in a dataset which has fewer than 5 observations connected to them? The values of the variables are always 0 or 1 as the customer is either from Region A (=1) or not from Region A (=0).

3. After removing these variables there should be fewer interactions. However, it would still be quite a large amount. I would therefore also like to only examine interactions with 5 observations or more. I am thinking this could be done using a formula in the logistic regression, but can you help me how I would find the right variables for the formula?

Answer 1

I interpret your situation as follows: You have 15 categorical variables, each with varying numbers of levels, but with about 70 levels between them. You have represented these variables / levels with dummy codes using level means coding (which you call "1-out-of-k"). That is, instead of forming k-1 dummy codes for each factor and letting the reference level be indicted by having all 0's in those dummy codes, and having it's mean value represented by the intercept, you have k dummy codes, one for each level. Note that, unless you suppress the intercept, this coding scheme will yield perfect multicollinearity (cf. my answer here: Qualitative variable coding in regression leads to "singularities"). In addition, by "observations connected to them", I gather you mean that there are only, say, 3 customers who are from, say, Region_A. Thus, you think you could do away with Region_A somehow. Correct me if this is wrong.

I'm not sure your strategy will work. Here are some thoughts:

1. With 15 categorical variables, you will have $(15\times 14)/2 = 105$ 2-way interactions to explore. The number of levels within each of those categorical variables will affect the number of degrees of freedom consumed to test the interaction, but will not affect the number of interactions you need to test. This could still be of value, though, as 6k observations may be a lot, but could quickly be consumed by so many tests.
2. Another consideration regarding your predicament is that your power will be maximized when you have equal numbers of observations / customers in each level. (For more on that, it may help to read my answer here: How should one interpret the comparison of means from different sample sizes?)
3. What do you intend to do about the dropped levels? You could drop them from your dataset altogether or you could combine those levels with other existing levels. There are potential problems with both of these strategies. Dropping them from the analysis throws out information about the other variables and their inter-relationships, you could end up biasing the results with something analogous to the omitted variable bias. If you wanted to merge several levels together, which levels should be merged? It isn't as obvious as it might sound.

Answer 2

With 15 or so variables, it wouldn't take too long to try this: first obtain frequencies; then prepare the data so as to remove values that occur fewer than 5 times; and then run an ordinary glm test for interactions allowing the program to default to its usual use of k-1 levels for each variable (in which it would preserve one level per variable as the reference category).

I'd obtain frequencies using the "table" command. Then I'd trim down the data using something like

data2 = data1 [data$Alabama != "0" & data$Samoan != 1, ] # 2nd dataset = 1st dataset without values of "0" for the "Alabama" variable and without values of "1" for the "Samoan" variable

If each variable has the same N, a more efficient solution would involve ascertaining the mean of each variable when treated as numeric. Then you'd include in the 2nd dataset any variable with a mean at least as high as the desired mean.

With different Ns per variable, you might make use of the snippet length(data1$variable[data1$variable==1]) <5 | length(data1$variable[data1$variable==0]) <5 # Either the # of ones or the # of zeroes < 5

Answer 3

There are many similar questions here! With 6k observations you should not need to do removal of variables, but go for some more principled way of model building. So:

1. You should use some other approach, look into regularization, lasso, ridge, elasticnet, followed by cross-validation. For instance LASSO with interaction terms - is it okay if main effects are shrunk to zero?. You might want to choose different penalty parameters for main effects and for interactions, maybe. And, since you used the r tag, you should use the modeling (r formula) language to describe interactions, that is, as x1 * x2 (or sometimes just x1:x2) to represent interactions, not introduce them as variables yourself.

2. Don't do it, unnecessary (maybe even harming) when using regularization.

3. For ideas about modeling interactions see this list of posts.