Modeling an Ordinal dependent variable

Question

I'm trying to model an ordinal customer satisfaction score with a mix of categorical and continuous predictors, and am just looking for some advice/critiques of my plan of attack by some of those more knowledgeable than myself. The DV was originally on a 1-10 scale, but has been binned (essentially just due to tradition within the company), the balance of classes is roughly 48/30/22 (%), there are 4000 observations with 11 predictors. The goal of this is to identify areas (predictors) which are most relevant to the customer satisfaction score and then to spend efforts focusing on improving those areas within the company.

My idea is to present some EDA in the form of graphs and the like of variables of interest, run an ordinal logistic regression to attempt to get a rough estimate of effect sizes/significances (possibly using adjacent categories), and finally to use something like ordinalForest cforest or perhaps both to get an estimate for the relative ordering variable importances for the best predictive model.

Specifically my questions/concerns are:

1. Setting seeds: I've read I should be varying the seed when running RF-based models and checking the variable importance in each case. Does it make sense to average the relative importances of these different cases? i.e. I do this twice and predictor x1 ends up ranked in the top place out of 3, but then ends up in the last place on the next run, so gets averaged to rank 2.

2. The party package has a conditional option within it's varimp function which, as I understand it in part, attempts to quell the influence of spurious correlations on the variable importance rankings. This seems to be useful, though time intensive, but can anyone suggest perhaps why I would not want to use it?

Apologies for long winded-ness, but this is my first project for the company (which doesn't have anyone more knowledgeable than myself of these types of analyses) in my first role outside of school, so am trying to handle it as thoroughly as possible.

Thank you much.

Answer 1

A few thoughts:

1) the binning of the DV may influence how you think about the problem. 0-10 scale is very different than selective binning which both explicitly changes the scale as well as implicitly making assumptions. Have these assumptions been validated? I would suggest running on the original DV as well as the new binned series.

2) By definition of seeds, using one should replicate the same results across trials. The goal is randomness but the seed makes sure we can replicate the randomness again. It is thus pseudo-random. Here's a quick read http://www0.cs.ucl.ac.uk/staff/d.jones/GoodPracticeRNG.pdf to get a better feel for the importance of, or lack of (depending on the problem at hand), seeds.

3) You may not want to use the function if there is seasonality or other patterns in your time series data. IE: if the sum is really greater than its parts than when the sum is split it can only be its component parts, not the sum.