# How can I derive goodness-of-fit measures (R-squared and chi-square values) from multinom (multinomial logistic regression) in R?

I am running multinomial logistic regression using multinom command in R. However, I could not figure out how to derive R-squared and chi-squared values from it. I somewhat approached my own way to calcualte the R-squared value but have no idea on the chi-squared value calcuation given the limited types of outputs I have. My model has three dependent variable categories (1,2,3) and have the results are as below.

    > summary(reg1)
Call:
multinom(formula = y1 ~ x1 * x2 + x1 * x3 +
x1 * x5 + x1 * x4 + x4 * x2 +
x4 * x3 + x4 * x5 + x2 * x3 +
x2 * x5 + x3 * x5 + x6 * x7 + x8 *
x7 + x9 + x10 + x11 + x12 + x13 + x14 + factor(year) +
factor(country), data = data)

Coefficients:
(Intercept) x1  x2  x3  x5  x4  x6  x7  x8  x9  x10 x11 x12 x13 x14 factor(year)2010    factor(year)2011    factor(year)2012    factor(year)2013
1   1604.4106   557.059 250.25276   1129.9204   26.64224    -1692.5912  71.58297    -246.4225   -399.3082   -231.52773  148.241722  7.622912    -0.316299   -159.8697   -133.5494   -505.5238   0   0   -424.58794
2   -1241.8927  589.1741    55.73597    342.5489    296.72497   467.3187    -49.9078    193.1122    1171.3274   230.19175   67.84545    -18.71366   -20.663976  -415.0831   -154.0205   -839.9936   0   0   38.17058
3   784.3395    1183.9142   1574.71006  -789.4121   497.71202   -150.5316   97.21926    -258.1482   306.2686    16.28096    -4.102168   -29.353747  415.633861  -1049.1961  -152.2329   1333.7646   0   0   1411.90537
factor(year)2014    factor(year)2015    factor(year)2016    factor(year)2017    factor(year)2018    factor(year)2019    factor(country)Austria  factor(country)Belgium  factor(country)Brazil   factor(country)China    factor(country)Colombia
1   1872.228    -85.28593   -288.8298   1476.744    -167.1055   -273.2286   100.4447    1235.8743   0   -1264.8407  0
2   2005.117    -2103.90701 -1431.0125  1668.921    -231.2164   -347.9716   -1169.6037  794.8244    0   -1540.5872  0
3   3031.994    -3418.86814 -2275.21    1630.276    -687.9121   -241.61 1879.3645   156.1329    0   177.4229    0
factor(country)Croatia  factor(country)Cyprus   factor(country)Czechia  factor(country)Denmark  factor(country)Estonia  factor(country)Finland  factor(country)France   factor(country)Georgia  factor(country)Germany  factor(country)Hungary
1   0   0   0   -906.7667   0   -1005.6461  2713.3697   0   -447.0311   0
2   0   0   0   1119.9752   0   -559.6754   873.7002    0   -1070.0891  0
3   0   0   0   1265.3938   0   1215.8249   3197.9996   0   1966.3596   0
factor(country)India    factor(country)Indonesia    factor(country)Ireland  factor(country)Israel   factor(country)Italy    factor(country)Japan    factor(country)Luxembourg   factor(country)Malaysia factor(country)Mexico   factor(country)Netherlands
1   0   0   -969.7374   0   605.9866    0   0   0   -1368.938   -1602.809
2   0   0   -1095.938   0   -1242.9418  0   0   0   -4203.134   1525.387
3   0   0   -286.3756   0   -1364.9962  0   0   0   -2647.337   3371.85
factor(country)New  Zealand factor(country)Norway   factor(country)Portugal factor(country)Russia   factor(country)Slovakia factor(country)South Africa factor(country)Spain    factor(country)Sweden   factor(country)Switzerland
1       0   -254.8904   -132.4134   0   0   1605.67266  524.807 -1929.42198 21.94178
2       0   1007.4591   2997.7089   0   0   55.15372    -940.2266   34.02166    -1017.61987
3       0   -298.5265   -220.0908   0   0   -644.40301  318.4926    2498.67407  -1565.71413
factor(country)Turkey   factor(country)United Kingdom   x1:x2   x1:x3   x1:x5   x1:x4   x2:x4   x3:x4   x5:x4   x2:x3   x2:x5   x3:x5   x6:x7   x7:x8
1   0   -284.6684   268.6287    237.449956  141.08515   -877.33 1661.7453   691.06058   -597.3844   -245.548    362.4153    -52.76805   -13.22399   643.6151
2   0   -362.9036   542.9315    5.787197    260.41842   1660.5545   -313.9221   -115.86746  -328.22 -407.5323   625.9459    -676.70772  47.07726    -409.2386
3   0   -2753.4001  800.5233    717.304288  35.99007    398.3228    414.6911    -59.42649   -564.2693   937.7502    555.7716    241.46169   38.46552    325.2159

Std. Errors:
(Intercept) x1  x2  x3  x5  x4  x6  x7  x8  x9  x10 x11 x12 x13 x14 factor(year)2010    factor(year)2011    factor(year)2012    factor(year)2013
1   0.6367411   0.4608092   1.007373    0.9188201   1.851734    0.1061235   10.00047    1.670603    0.6213382   9.870034    8.702194    10.17336    1.426588    2.409377    20.02163    0.00E+00    0   0   0.00E+00
2   3.8250776   5.8659486   19.038343   2.4869914   14.599087   0.6375129   46.5159 13.069139   3.9075164   45.259471   35.717226   12.46371    38.248475   7.623069    36.13595    8.70E-10    0   0   3.68E-82
3   3.9013219   5.8189039   19.216861   2.4092377   14.623989   0.6502203   46.77807    13.140232   3.8745704   46.220404   35.483221   24.64454    38.068055   7.592431    36.29654    8.70E-10    0   0   3.68E-82
factor(year)2014    factor(year)2015    factor(year)2016    factor(year)2017    factor(year)2018    factor(year)2019    factor(country)Austria  factor(country)Belgium  factor(country)Brazil   factor(country)China    factor(country)Colombia
1   0.000895394 1.0662  0.002877624 5.69E-08    5.45E-01    3.65E-08    3.65E-150   7.60E-52    0   0.00E+00    0
2   0.005722742 10.85441    0.021171664 1.12E+01    5.45E-01    4.66E+00    0.00E+00    7.60E-52    0   3.49E-13    0
3   0.004838609 10.85441    0.024037041 1.12E+01    5.29E-40    4.66E+00    0.00E+00    0.00E+00    0   0.00E+00    0
factor(country)Croatia  factor(country)Cyprus   factor(country)Czechia  factor(country)Denmark  factor(country)Estonia  factor(country)Finland  factor(country)France   factor(country)Georgia  factor(country)Germany  factor(country)Hungary
1   0   0   0   0.00E+00    0   0   2.71E-08    0   0   0
2   0   0   0   6.35E-05    0   0   4.69E+00    0   0.000140461 0
3   0   0   0   6.35E-05    0   0   4.69E+00    0   0.000140461 0
factor(country)India    factor(country)Indonesia    factor(country)Ireland  factor(country)Israel   factor(country)Italy    factor(country)Japan    factor(country)Luxembourg   factor(country)Malaysia factor(country)Mexico   factor(country)Netherlands
1   0   0   0   0   3.11E-09    0   0   0   2.72E-22    0
2   0   0   0   0   3.64E-09    0   0   0   5.67E-23    10.85441
3   0   0   0   0   8.70E-10    0   0   0   0.00E+00    10.85441
factor(country)New  Zealand factor(country)Norway   factor(country)Portugal factor(country)Russia   factor(country)Slovakia factor(country)South    Africa  factor(country)Spain    factor(country)Sweden   factor(country)Switzerland
1       0   0   0   0   0       0   5.49E-08    0   5.45E-01
2       0   0   0   0   0       0   5.49E-08    11.24449    5.45E-01
3       0   0   0   0   0       0   4.76E-127   11.24449    2.11E-09
factor(country)Turkey   factor(country)United Kingdom   x1:x2   x1:x3   x1:x5   x1:x4   x2:x4   x3:x4   x5:x4   x2:x3   x2:x5   x3:x5   x6:x7   x7:x8
1   0   1.07E+00    0.729201    0.157802    0.3040747   0.07680153  0.1678956   0.1531367   0.3086223   1.454584    2.931503    1.138881    22.83679    1.638788
2   0   9.25E-04    8.569019    2.962367    12.6959086  0.97765809  3.1730571   0.4144986   2.4331812   15.542936   4.590969    13.140722   20.14723    3.879428
3   0   2.95E-05    8.395867    2.961991    12.6778769  0.96981732  3.2028101   0.4015396   2.4373315   15.495472   4.51289 13.130588   21.69382    3.911349


While I cannot derive the chi-squared goodness of fit measure as I do not have the degrees of freedom for null and residual deviances, I somehow approached a method to calculate R-squared value as below:

    nnet.mod.loglik <- nnet:::logLik.multinom(reg1)
nnet.mod0 <- multinom(y1 ~ 1, data=data)
nnet.mod0.loglik <- nnet:::logLik.multinom(nnet.mod0)
(nnet.mod.mfr2 <- as.numeric(1 - nnet.mod.loglik/nnet.mod0.loglik))


But even here, I have a problem. If I run this using fixed-effects I added for country and year, I get 0.9949825 of R-squared value using the code above. But if I drop both of these fixed effects, I somehow get a reasonable R-squared figure of 0.7517982.

Thus, my questions are three:

1. Is the R-squared measure above is correct? If so, is it normal to get such high R-squared value as 0.9949825? Do I need to exclude the fixed effects and report the lower R-squared value 0.7517982 instead although I still have to report the fixed effect model results? Is the method for R-squared value correct?

2. How can I get the chi-squared values given the limited information I have with no degrees of freedom values for null and residual deviances?

3. Is it possible to report the R-squared and chi-squared values for each of the three different dependent variables? Otherwise, is it normal to report only one R-squared and chi-squared values for the entire three different dependent variable regression?

I'll try to answer in-line below, but one general point is that the thresholds for defining normal/abnormal results, as well as the conventions for which goodness-of-fit measures should be reported, can vary a great deal across research problems, industries, and academic disciplines.

Is the R-squared measure above is correct? If so, is it normal to get such high R-squared value as 0.9949825? Do I need to exclude the fixed effects and report the lower R-squared value 0.7517982 instead although I still have to report the fixed effect model results? Is the method for R-squared value correct?

If the independent variables in your model explain the vast majority of variability in your dependent variable, then yes it is plausible to observe an R-squared value that high in a descriptive model.

In this case, the difference between the R-squared value from the model including your fixed effects (~0.99) and the model excluding your fixed effects (~0.75) suggests that those fixed effects might explain a significant amount of variability in your class labels - is it possible that some of the categories only occur in certain countries or in certain years? That might explain why including these variables increases R-squared to near ceiling.

Ultimately, whether you need to include in your model depends on the question you're trying to answer or the problem you're trying to solve. If you need to be able to explain the impact of x1 on your outcome, conditioned on the year, then it would be important to leave year in the model.

How can I get the chi-squared values given the limited information I have with no degrees of freedom values for null and residual deviances?

One possibility is the multinomial.test() from the EMT package (see documentation here), which only requires class probabilities and true population proportions. Whether you need to report chi-squared along with R-squared depends on the audience.

Is it possible to report the R-squared and chi-squared values for each of the three different dependent variables? Otherwise, is it normal to report only one R-squared and chi-squared values for the entire three different dependent variable regression?

One common way to accomplish this is to estimate separate binomial logistic regression models, treating each category as a binary outcome. Here, you'd have a "1 vs. not-1" model, a "2 vs. not-2" model, and a "3 vs. not-3" model.

• Thank you. Although it does not sound perfect, it makes a lot of sense to me.
– Eric
Commented Mar 4, 2020 at 18:43