I have a regression problem and I am thinking of using ridge regression. One of the predictors is subject's gender, which is a categorical variable. How to take care of this variable for ridge regression modeling? Can it be encoded as $0$ and $1$? What to do with a categorical variable with more than $2$ categories?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
12
You are correct to assume that a categorical variable is encoded as indicator function/vector in your design matrix $X$; this is standard. To that respect, usually one of the levels is omitted and subsequently treated as baseline (if not you would have surely a rank-deficient design matrix when incorporating an intercept).
If you have a categorical variable with multiple categories you will once more treat is as an indicator function in your design matrix $X$. Just now you will not have a vector but a smaller submatrix. Lets see and example with R:
set.seed(123)
N = 50; # Sample size
x = rep(1:5, each = 10) # Make a discrete variable with five levels
b = 2
a = 3 # Intercept
epsilon = rnorm(N, sd = 0.1)
y = a + x*b + epsilon; # Your dependant variable
xCat = as.factor(x) # Define a new categorical variable based on 'a'
lm0 = lm(y ~ xCat)
MM = model.matrix(lm0) # The model matrix you use
image(x = 1:ncol(MM), y = 1:N, z=t(MM)) # The matrix image used. / Red is zero
As you see the levels 2,3,4 and 5 are encoded as separate indicator variables along the columns 2 to 5. The columns of ones in the column 1 is your intercept, level 1 is automatically omitted as an individual column and is assumed active alongside the intercept.
OK, so what about the ridge parameter $\lambda$? Remember that ridge regression is essentially using a Tikhonov regularized version of the covariance matrix of $X$. ie. $\hat{\beta} = (X^TX + \lambda I)^{-1} (X^T y)$, to generate the estimates $\hat{\beta}$. That is not problem for you if you have discrete (categorical) or continuous variables in your $X$ matrix. The regularization takes part outside the actual variable definition and essentially "amps" the variance across the diagonal of the matrix $X^TX$ (the matrix $X^TX$ can be thought of as a scaled version of covariance matrix when that the elements of $X$ are centred).
Please note that, as seanv507 and amoeba correctly comment, when using ridge regression it might make sense to standardise all the variables beforehand. If you fail to do that, the effect of regularisation can vary substantially. This is because increasing the observed variance of a particular variable $x$ by $\lambda$ can massively alter your intuition about $x$ depending on the original variance of $x$. This recent thread here shows such a case where regularization made a very observable difference.
answered Dec 10, 2015 at 11:59
-
1$\begingroup$ Ridge regression depends on scale of input variables, so have to decide how you rescale your input variables.. Eg zero mean variance 1 for categorical variables too? It's not so clear (same as with ordinary variables... Depends on whether scale is meaningful relative to other variables) $\endgroup$– seanv507Dec 10, 2015 at 12:06
-
$\begingroup$ Sorry, I cannot understand your comment fully. All regression algorithms depend on the scale of your variables. You can $z$-transform them but your need to be careful what you want to ultimately do. The thread here elaborates on standardising your variables more. Clearly picking the parameter $\lambda$ is an issue in itself. $\endgroup$ Dec 10, 2015 at 12:12
-
2$\begingroup$ I mean that linear regression is invariant [equivariant?]to a rescaling of the input variables ( the size of the coefficients changes, but the output doesn't change). But if you rescale the inputs for ridge regression you will get a different solution - because it is penalising the size of the coefficients (see eg bottom of page 63 Elements of statistical learning) $\endgroup$– seanv507Dec 10, 2015 at 12:33
-
1$\begingroup$ that's not the point: consider eg a 'polynomial' linear regression ( ie you add $x^2,...x^n$), if your variables are not normalised with variance >1, then ridge regression will penalise the linear terms vs highest order terms ( which will have very small coefficients) - if you normalise, ridge regression does what you would like..ie prefer lower order polynomial. On other hand, lets say x and y are coordinates of something only moving in x direction but measurement noise adds a bit of variance to the y direction then ridge regression with unnormalised variables does what you want. $\endgroup$– seanv507Dec 10, 2015 at 13:17
-
3$\begingroup$ +1, but @seanv507 raises a good point (+1), which I would formulate as follows: if some of the predictors are continuous and some are binary, then the binary ones are likely to have very different variance from the continuous ones, and the ridge penalty will have very different effects on them. Hence, when using ridge regression it might make sense to standardize all variables, including the binary ones. $\endgroup$– amoebaDec 10, 2015 at 14:35
$\begingroup$
$\endgroup$
6
Yes you can, your beta_ridge will be a number that will pop up when gender is 1 and won't have an effect when gender is 0. If you have more than one category, in my experience, make them all binary. e.g. if you have apple, orange, pear, instead of saying 1,2,3 say is_apple = [0,1] is_orange=[0,1] is_pear=[0,1].
-
$\begingroup$ Thanks. So you mean to say that instead of using a single variable, make it to three variables which are orange (0/1), apple (0/1) and pear (0/1) $\endgroup$ Dec 10, 2015 at 11:59
-
1$\begingroup$ @user3761534 If you want you use all the levels of your
fruitvariable you will need to make sure you do not use an intercept. If you do have an intercept you will need to omit one of them. Otherwise your design matrix will be rank-deficient. This is whysexis usually a single indicator and "pops up when gender is 1'. $\endgroup$ Dec 10, 2015 at 12:08 -
-
$\begingroup$ @usεr11852 Im not sure if identifiability is an issue when we have 1 equation and n>p, as opposed to a ,say, multinomial logit setting. If you could please provide a proof of that you would save me a lot of time :) . Another issue with using all the levels is that our betas will scale up and down based on the category value, in that pear might have a lower beta than apple even if their effects were equal. $\endgroup$ Dec 10, 2015 at 19:23
-
$\begingroup$ I do not comment on identifiability; I am talking abut rank-deficiency of your design matrix in the case you write out the full
fruitvariable and use an intercept. See the example in my answer. If you have "one equation" it means you already did what I describe, ie. you did not use all the available levels. $\endgroup$ Dec 10, 2015 at 19:47