For a university project I am supposed to predict one survey question (with 5 different answers) using 20 other survey questions (each with 4 or 5 different answers i.e. all predictor variables are categorical). The outcome of interest is the predicteed distribution, i.e. rather than individual classifications, the overall population probabilities are of interest. There are 800 observations in my data set.

Initially I used random forest / probability forest (using ranger) to fit this model. The outcome is that for any individual observation in the test set, the predicted probability is very much equal to the prior distribution of the response variables in the train set, i.e. there is no separation between the answers whatsoever.

My professor indicated that I should use multinomial regression for more explainability regarding this outcome. I do obtain the same results. The standard errors obtained using multinomial regression are so inflated (all $\geq$8000) that I do not know how to utilize this model. I could use LASSO, but then explainability would be gone again (it performs the same, also).

I thought of perhaps running a $\chi^2$-test comparing the original distribution of the response to all factor levels of each individual question, to show that they are all equal. But that way interactions are still unaccounted for.

So, generally, I am wondering how I can "prove" that the classes of the response variable really aren't separable, apart from just reporting my model results.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.