1
$\begingroup$

I am running a multinomial logistic regression with SPSS and I have encountered a problem (?) with my data. I have a dependent variable: foreign language enjoyment (FLE) (DV) with five categories extracted by conducting a latent profile analysis of the three dimensions of FLE measured by a 5-point Likert scale with 11 items in total; two independent variables: L2 proficiency, measured by a standardized English test totaled 150 points; gender as covariates. There are 1370 respondents in the data. The distribution of my DV is: First category (N=358,28.4 %), second category (N=116, 9.2 %), third category (N=429,34 %), Fourth category (N=18, 1.4%), fifth category (N=339, 26.9%) . As I run the regression (last category as the reference), the classification table shows that 0 % percent are classified in the second and fourth category. See below: enter image description here

The results of the logistic model otherwise seem "reliable", though.

enter image description here

I read previous posts on this issue but I am not clear about how to run a decision tree and what are the suggested other methods to uncover the non-linear relation between FLE and L2? Is there any consulting service that I can resort? Thanks a lot! enter image description here

Best regards

$\endgroup$
7
  • $\begingroup$ How did you validate the results of the latent profile analysis? I notice that category #4 has only 5 + 4 + 9 = 19 or 1.3% of the respondents. I'd guess that it would be difficult for a classifier to have high recall on this one. Category #2 has 49 + 45 + 22 = 116 or 8.5% but is still smaller than the others. So before working on the classification I would review the first step of the analysis. $\endgroup$
    – dipetkov
    Apr 6 at 9:29
  • $\begingroup$ Thank you for your kind reply. I understand your concerns. I validated the results through MANOVA where the means of the three dimensions differ across the five profiles. Besides, I double checked Mplus by running again with at least twice the random starts to avoid a local maxima solution. Furthermore, I drew an elbow plot mapping information criteria estimates for successive models. The five solution is concluded as the marginal gains become negligible from then on. $\endgroup$
    – Vivien Gao
    Apr 6 at 12:36
  • $\begingroup$ Good, thank you for providing extra details. I think the MANOVA analysis has an inherent limitation in that the actual responses are ordinal (on a 5-point Lickert scale) but MANOVA treats them continuous. (On the other hand there is a considerable step up in complexity from continuous to ordinal outcome.) $\endgroup$
    – dipetkov
    Apr 6 at 13:10
  • $\begingroup$ Do you interpret FLE as ordinal as suggested by the two answers: 1 < 2 < 3 < 4 < 5 in terms of foreign language enjoyment? Then note that the problematic levels are the intermediate levels 2 and 4. So one approach you could consider is to use a coarser 3-level scale for FLE. Yes, there'll be less resolution but it may be worth it if you have more confidence in the final analysis. $\endgroup$
    – dipetkov
    Apr 6 at 13:10
  • $\begingroup$ Thank you a lot. The central aim of my study is to uncover the non-linear relation between foreign language enjoyment (FLE) and L2 (second language achievement). The five FLE categories extracted from latent profile analysis through Mplus are nominal instead of ordinal, where category 1 means second highest FLE with a lower FLEP and FLEA,category 2 means second lowest FLE, category 3 means average FLE, category4 means lowest FLE, category 5 means highest FLE. $\endgroup$
    – Vivien Gao
    Apr 7 at 1:19

2 Answers 2

1
$\begingroup$

It is not a big problem that the classification table, usually referred to as a confusion matrix outside of SPSS, shows that 0 % percent are classified in the second and fourth category unless your objective is to accurately predict the category of FLE. The way a confusion matrix is constructed is that for a specific observation, the category with the highest predicted probability is take as the predicted class. For example, for observation 36, the predicted probabilities of the five categories are 40%, 30%, 20%, 1%, 9%, then category 1 is the predicted category. If the observed response was category 2, it is not a bad fitting since the model was only off by less than 10% in probability to make the correct prediction for this particular observation. The reason for the column of predicted classes 2 and 4 to be empty is that they are never the dominant one with the highest probabilities compared to other categories in any observation.

A more important issue is whether multinomial logit model is be best choice. If FLE is not nominal but ordinal, where category 1 means a lower enjoyment level than category 5, then you should look for cumulative logit models. Adjacent-category, continuing-ratio, stopping-ratio logit models are also for ordinal responses and worth considering.

Your concerns about nonlinear effects are reasonable, but you don't have to run a decision tree to uncover nonlinear effects. To include nonlinear effects when you have only two predictors, one binary and one continuous, you should create functional (e.g., log, polynomials) terms of L2, interaction terms between gender and L2 terms, or additive flexible terms of L2. See my general answer in such situations Is it possible to calculate x-intercept from a mixed model?.

$\endgroup$
3
  • $\begingroup$ Thank you so much for your explanation.The central aim of my study is to uncover the non-linear relation between foreign language enjoyment (FLE) and L2 (second language achievement). I am interested in the threshold value where L2 (second language achievement) cannot predict FLE. How can I further get in touch with you on this topic? $\endgroup$
    – Vivien Gao
    Apr 7 at 1:23
  • $\begingroup$ The five FLE categories extracted from latent profile analysis through Mplus are nominal instead of ordinal, where category 1 means second highest FLE with a lower FLEP and FLEA,category 2 means second lowest FLE, category 3 means average FLE, category4 means lowest FLE, category 5 means highest FLE. I am satisfied with this five-profile solution as they represent the qualitative differences that I am exploring instead of a parallel three-profile solution where only quantitative differences are discovered.My concern is will the small sample size in group 4(N=18) hamper the predictive power? $\endgroup$
    – Vivien Gao
    Apr 7 at 1:24
  • $\begingroup$ You can find me on LinkedIn through my profile and use my consulting service. If your study objective is to uncover a relationship, then you don't have to worry about accurate predictions of particular cases. However, whether the true relationship is nonlinear depends on the model specification. It could be linear in cumulative logit model but nonlinear in a multinomial logit model. $\endgroup$
    – DrJerryTAO
    Apr 8 at 5:57
0
$\begingroup$

First, it's not really an intrinsic problem that no one is predicted to be in the second or fourth categories. Those are very rare categories. However, if you need to have some people in those categories for some reason, you could change the cost function. I don't know how you would do this in SPSS, and questions about coding are off topic here, but it may be possible.

Second, you are using multinomial logistic. But it seems that the DV is ordinal. Did you try ordinal logistic? There can be reasons for using multinomial logistic even with an ordinal DV, but I'd start with ordinal.

Third, I question whether the other results are "reliable" (in quotes, I suppose, because this is not the meaning of "reliable" in statistics; maybe you mean "good" or "reasonable"?). All three of the pseudo $R^2$ measures are very low. The overall percentage correct is pretty low and quite a lot are way off. About a quarter of the 1s are predicted to be 5s and about a third of the 5s are predicted to be 1s.

$\endgroup$
3
  • $\begingroup$ Thank you so much. The five FLE categories extracted from latent profile analysis through Mplus are nominal instead of ordinal, where category 1 means second highest FLE with a lower FLEP and FLEA,category 2 means second lowest FLE, category 3 means average FLE, category4 means lowest FLE, category 5 means highest FLE. I am satisfied with this five-profile solution as they represent the qualitative differences that I am exploring instead of a parallel three-profile solution where only quantitative differences are discovered. $\endgroup$
    – Vivien Gao
    Apr 7 at 1:26
  • $\begingroup$ My concern is the small sample size in group 4(N=18), will it hamper the predictive power? Besides, given the pseudo R2 measures, overall percentage correct, are there any other methods to uncover the non-linear relation between FLE and L2? Thanks a lot! I tried the factor mixture model in Mplus but I kept getting the local maxima solution that cannot be trusted although I tried to resolve the local maxim by increasing the random starts. Best regards. $\endgroup$
    – Vivien Gao
    Apr 7 at 1:30
  • 1
    $\begingroup$ You've described an ordinal scale, but misnumbered. You should recode them so they are in order (1 = lowest to 5 = highest). Ordinal logistic has far fewer parameters to estimate. $\endgroup$
    – Peter Flom
    Apr 7 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.