Maximum Likelihood with Categorical Variables - Does this Change Anything?

Question

Recently, I was thinking about the following question - suppose we have some continuous variables and some categorical variables and we want to fit a regression model to this data: How will this change the overall estimation and optimization process?

The way I see it, optimizing the Maximum Likelihood Equation for this regression model might become a "Mixed Integer Optimization Problem" - whereas when there were only continuous variables, optimizing the Maximum Likelihood Equation was a standard optimization problem.

Many times, standard Maximum Likelihood Equations have closed form solutions and can be "directly" solved - other times when these equations do not have closed form solutions (e.g. estimating the parameters in a Multinomial Logistic Regression), we can use standard optimization algorithms such as Newton-Raphson or Gradient Descent and iteratively find a solution.

However, now that some of the variables are categorical, (in theory) this would mean that the Likelihood is no longer "differentiable" in the classical sense, seeing as integer constraints have now been imposed on this optimization problem. I have heard that for such problems, optimization algorithms such as "Branch and Bound" are used instead of Gradient Based Optimization Algorithms.

This leads me to my question:

• When we have a Likelihood Equation that involves both continuous and categorical variables, how do usually optimize the Likelihood - can this equation still have closed form solutions or be solved using some Gradient Method?

• In classical Maximum Likelihood Estimation, the parameter estimates have certain "desirable" properties such as "Unbiasedness" and "Consistency" - when there are categorical variables in the Likelihood, do the parameter estimates still "maintain these attractive properties"?

Thanks!

Note: I am interested in applying a Cox Proportional Hazards Model to some data with continuous and categorical variables. Although R seems to be able to estimate the regression parameters, I was wondering which optimization algorithm was being used in the background and if the resulting parameter estimates still maintain their "attractive statistical properties".

Answer 1

The algorithm used is exactly the same: although the features are not continuous in your example, the coefficients in a regression model still are. We are optimizing the coefficients in the model, the features do not change.

Answer 2

On the one hand, there are features, which are attributes of your data, stuff that you measure, and on the other hand, there are the parameters to your model. Both can be continuous or discrete.

If the features are discrete, this doesn't mean that your parameters have to be discrete as well. E.g. if you enter a factor $f$ with $\ell$ levels into the formula of the R function lm(), the belonging design matrix (displayed with model.matrix()) will contain at least $\ell-1$ new columns only filled with zeros and ones (watch collinearity). However, the parameters belonging to those categorical columns are doubles.

But, of course, you could also have situations where your parameters are discrete, e.g. you might want to know which of two different models is the better one, which would lead to a binary parameter that you have to fit. Then you have the situation that you have described: optimization methods used for continuous parameters are not applicable and you have to use discrete optimization techniques, possibly in combination with those for continuous parameters.

Notions like bias and consistency could still be defined for discrete parameters (as long as you have a metric space) but it usually doesn't make much sense. E.g. convergence in a finite discrete space is usually not very interesting.