0
$\begingroup$

In multinomial regression, is it theoretically possible to have issues with having fewer predictors than responses? I'm assuming all predictors are continuous for purposes of simplicity.

I am thinking in analogy to linear regression, wherein having too few predictors may cause the resulting linear system to be underdetermined. I mean this in the sense of the effective dimensions (degrees of freedom?) of the predictor being less than that of the response. If we view multinomial regression just as classification, this does not make sense, but if we view it as nonlinear regression I think it is less clear.

Note: this is substantially edited from the original posted question.

Improve this question
$\endgroup$
13
  • 4
    $\begingroup$ There is no problem with having less predictors than response classes. E.g. Suppose you have variables $Y = \text{letters A, B, ...}$, and the predictor $X = Y$. Then, you have perfect prediction capabilities running the regression $Y \sim X$. $\endgroup$
    – Alex
    Commented Apr 27, 2017 at 0:11
  • $\begingroup$ @Alex Interesting point. I guess I don't really quite know how to make my idea precise, because I don't see that as a counterexample. $\endgroup$
    – Jacob Maibach
    Commented Apr 27, 2017 at 0:20
  • 1
    $\begingroup$ glm in R coerces the first class/factor level to 0 and the rest to 1. It may be that whatever method you are using to do 'logistic' regression does the same. Please clarify whether you meant multinomial logistic regression. $\endgroup$
    – Alex
    Commented Apr 27, 2017 at 0:40
  • 2
    $\begingroup$ As @Alex notes, there is no problem with having more response categories than predictors (what would be problematic would be having more predictors than observations). It looks like your main problem is strange responses. I'm afraid we won't be able to help you unless you provide much more information. Have you looked at the parameter estimates of your multinomial logistic regression? $\endgroup$
    – Stephan Kolassa
    Commented Apr 27, 2017 at 7:15
  • 1
    $\begingroup$ "too few predictors" could cause the system to be over-determined, not underdetermined (which would be "too many predictors", relative to the number of observations) $\endgroup$
    – GeoMatt22
    Commented May 4, 2017 at 19:54

1 Answer 1

Reset to default
1
$\begingroup$

This is mostly answered in comments, some summary here.

First, this last version contains a formulation "In multinomial regression, is it theoretically possible to have issues with having fewer predictors than responses?" should probably be more as that is more consistent with the first versions. (fewer should definitely not be a problem) but the comparison in the quote is to "responses", not to "(number of) observations", so it is still unclear what is meant.

I will try to answer to the possible versions of Q:

  1. More predictors than number of response categories: This is not in itself a problem, as you mostly have many parallel regression equations, linked by the requirement that sum of estimated probs should be one.
  2. More predictors than number of observations: This is more of a problem, since the estimating equations are underdetermined, with infinitely many solutions. If the goal is inference on model parameters, this is definitely a problem, you should get more data. If the goal is prediction of future observations, you could try regularization, such as lasso, ridge or elasticnet (implemented in R in CRAN package glmnet).
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.