0
$\begingroup$

I am trying to fit a cumulative probability model (ordinal logistic regression with 17 categories and 827 observations) with elastic net penalty using the ordinalNet function from the ordinalNet package in R.

I have 33 covariates and observe the following:

  1. When I include all 33 covariates, all coefficients are set to zero.
  2. When I include only 31 of the 33 covariates, many coefficients are different from zero.

The datasets in 1 and 2 are the same.

I did not expect that removing covariates would lead to less coefficients equal to zero. Rather the opposite. What could be the cause of this?

The code I use looks as follows:

ordinalNet(x = as.matrix(dataSet[, ..tested.factors]),
           y = dataSet[, category_number],
           family = "cumulative",
           link = "logit",
           standardize = FALSE)

dataSet is a data.table.

$\endgroup$

1 Answer 1

2
$\begingroup$

The first step, if you are not using unsupervised learning (data reduction), is to show evidence that there is a predictive signal. Fit the full model without penalty and get the likelihood ratio $\chi^2$ test with say 33 degrees of freedom. This corrects for having 33 chances to find something. If this $\chi^2$ is not large you do not have a basis for finding predictors.

Note that lasso and elastic net have a low probability of finding the right predictors. That is why data reduction is so valuable. Reduce the dimensionality of the problem, without using Y in any way, using things like variable clustering followed by regular principal components, or sparse principal components analysis. After you score groups of predictors into a single metric you can use these to develop the outcome model and also to run the overall likelihood ratio test to check for predictive signal with fewer than 33 d.f.

$\endgroup$
4
  • $\begingroup$ Thank you for your answer and suggestions, @Frank Harell. The likelihood ratio suggests there is a basis for finding predictors. I am bound by some constraints that do not allow me to apply dimension reduction techniques, but for my own development I should try to apply these. $\endgroup$
    – koteletje
    Oct 22, 2020 at 8:40
  • $\begingroup$ With data reduction being so promising for your particular situation what are your constraints? $\endgroup$ Oct 22, 2020 at 11:29
  • $\begingroup$ Most importantly restrictions imposed by model users: the model needs to be understandable/intuitive to model users without a quantitative background. $\endgroup$
    – koteletje
    Oct 22, 2020 at 12:39
  • $\begingroup$ This does not impose a restriction for most models. It does put a burden on the modeler to translate the model results to various formats that are interpretable by the researcher. Semiparametric odels can be restated in a variety of useful ways and so can other models and so can data reduction results. The desire for parsimony turns out to be a desire for bad predictions and arbitrary feature selection. $\endgroup$ Oct 22, 2020 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.