How to assess if a model is good in multinomial logistic regression?

Question

I have some ordinal response $y$ that I modeled using both ordinal logistic regression and multinomial logistic regression (to avoid the proportional odds assumption), using two continuous variables as predictors $x_1$ and $x_2$.

I tested different models by including the predictor variables one at a time and their interactions.

$$y \sim x_1 \\ y \sim x_2 \\ y \sim x_1 + x_2 \\ y \sim x_1 + x_2 + x_1x_2 $$

In this way I obtained 8 different models (4 models using ordinal, and 4 models using multinomial logistic regression) and therefore 8 AIC values. It turn out that the best model (the difference in AIC is like 200) is the multinomial logistic with the following predictors:

$$ y \sim x_1 + x_2 + x_1x_2 $$

As far as I know, this model provides the best fit amongst all the various options. Now, how do I quantify if this model is good for the data in an absolute sense (in order for it to be published)?

I can do the regression both using frequentist glm and bayesian glm (so far I did frequentist way because it was more computationally cheap). Ideally I'd like to have the methodology that is most honest and convincing.

EDIT:

I'm more interested to assess model fit in terms of inference rather than prediction. In my specific problem, one of the classes is much more probable than the others on a wide range of the parameters, so prediction is unfeasible. But I'm still interested in discovering how appropriate are the estimates of the underlying probabilities.

Answer 1

Let's take apart your modeling approach to see if we can figure out why a certain model is going to "fit" better.

• Multinomial vs ordinal: Multinomial I would bet is almost always going to fit better than an ordinal because it gives you coefficients for every level. It is the most flexible here and has the least restrictive assumptions, namely, ordinal logit assumes parallel lines /proportional odds between each level. In an absolute sense, I think an ordinal logit would only fit better if you certainly have proportional odds or really close to it so that the fewer estimated parameters saves you in an information criteria like AIC or BIC.
• More vs less predictors: More predictors means you have more explanatory power in your model, so naturally it will fit better. Only if the increase in fit is not sufficient to make up for the penalty of added parameters will the simpler model fit better based on AIC/BIC.

I would recommend using more comprehensive fit statistics - while the AIC is really good, it is not the only one out there (have you used the BIC as well?). Use Wald tests or LR tests to compare the interaction effects, looking into cross-validation for overall predictive ability, test the parallel lines assumptions. Also think about it theoretically - do you think your measures are ordinal? Does an interaction term make sense? Depending on the field, many journals (particularly in the social sciences) look down on purely data driven modeling approaches if you don't have strong theory to support your decisions.