2
$\begingroup$

I'm analyzing biological data where I'd like to see the impact of scaled gene expression on the classification of the sample. I binarized the response variable as 0 and 1 and used lasso with cross-validation. My goal is inference rather than prediction at this point.

I compared two different type.measure parameters (auc and class) in cv.glmnet function and plotted coefficient estimates in a scatter plot:

coefficient-comparison

Correct me if I'm wrong, but the documentation suggests both methods can be suitable for binary classification problems. I'm trying to understand the differences I see here. When using misclassification error (class) loss function, more variables shink to zero compared to the area under the ROC curve (auc) What is the reason for this different behavior?

$\endgroup$
2
  • $\begingroup$ Does this answer your question? Measuring accuracy of a logistic regression-based model $\endgroup$
    – EdM
    Commented Jul 22, 2020 at 19:14
  • $\begingroup$ I got some great insights from that link, but I don't think it necessarily answers my question. I'm trying to get a sense of why misclassification loss function is shrinking more variables, where in essence, the ROC curve is drawn using the frequency of true and false predictions. Is the situation I'm looking at here that the accuracy of the model measured at a certain point as opposed to the entire continuum of the possible values (ie a fixed (average?) point on ROC curve versus the whole area under the curve?) $\endgroup$
    – Atakan
    Commented Jul 23, 2020 at 6:10

1 Answer 1

2
$\begingroup$

The main point is that accuracy is not really "suitable for binary classification problems" despite its frequent use as a criterion in model evaluation.

In an important sense there is no single "accuracy" measure as it depends on selection of a particular probability cutoff for assigning class membership. For binary classification this selection is often hidden from view (as it seems to be in cv.glmnet() when class is selected as the criterion) and set at a value of p = 0.5; that is, class membership is assigned to whichever class has the highest probability. That's only appropriate if you assign the same cost to false-positive and false-negative errors. Other relative costs would lead to different choices of the probability cutoff. See this recent page for an introduction and links to further discussion about selecting cutoffs.

So your sense expressed in a comment is correct: the difference is that AUC examines the whole range of potential false-positive versus false-negative tradeoffs versus the single choice imposed by the p = 0.5 class-assignment threshold. As this page discusses, auc is thus preferable to class as a criterion for comparing models as you are effectively doing with cross validation.

This answer describes how the best way to evaluate such models is with a proper scoring rule, which is optimized when you have identified the correct probability model. The deviance criterion in cv.glmnet() (the default for logistic regression) is equivalent to a strictly proper log-loss scoring rule. That may be bit more sensitive than auc for distinguishing among models; see this page.

I can't say with certainty why the class criterion maintains fewer genes in the final model than does auc. I suspect that's because the class criterion is less sensitive to distinguishing among models, which is what you're doing when you try to minimize over a range of penalty values, so it ends up with larger weights on fewer predictors. But that's an intuitive heuristic argument with no formal basis.

A final note: inference following LASSO is not straightforward. See this page for some discussion. With gene-expression data you typically have a large number of correlated potential predictors, among which LASSO will make choices that can be very data dependent. So even if you calculate p-values and CI properly that doesn't mean you have identified "the most important" genes for the classification, just a particular set that is justifiable. Try repeating the modeling on multiple bootstrapped samples of the data to gauge how stable the gene-selection process is.

$\endgroup$
6
  • $\begingroup$ would you then suggest one should use type = "deviance" over auc even with binary outcomes as it equals a proper log-loss scoring? This of course also relates to my question, you refer to in this answer? $\endgroup$
    – Thomas
    Commented Jul 23, 2020 at 13:06
  • 1
    $\begingroup$ @Thomas yes I think that's the safest choice from what's available built-in to cv.glmnet(); it's the default. Log-loss is far from the only proper scoring rule, but it comes directly from the maximum-likelihood calculations used to solve the logistic regression. It puts a lot of weight on extreme cases. In principle you could use any proper scoring rule for choosing the logistic-regression penalty by cross validation, although you would have to adapt the cross-validation code yourself. $\endgroup$
    – EdM
    Commented Jul 23, 2020 at 13:38
  • $\begingroup$ Interesting. I will definitely give it a shot to compare models. Can you readily obtain the log-loss once the cv.glmnet has run then? Thank you. $\endgroup$
    – Thomas
    Commented Jul 23, 2020 at 13:45
  • $\begingroup$ @Thomas yes, it's equivalent to the deviance error criterion. The cross-validated error (by whatever criterion was chosen) as a function of $\lambda$ is in the cvm part of the object returned by cv.glmnet(). Be carful when comparing models with different numbers of parameters, however; see the Akaike Information Criterion and its relationship with deviance. $\endgroup$
    – EdM
    Commented Jul 23, 2020 at 15:38
  • 1
    $\begingroup$ @Atakan this is an inherent problem with LASSO. How you will be using the results matters. Signaling pathways are most important; multiple genes can be proxies for the same altered pathway. For a test to distinguish 2 classes, just using the original LASSO model would be fine as which particular proxy is chosen doesn't matter much. As a guide to further experiments you want to examine a wider set of genes. You might consider instead the limma Bioconductor function (or its more recent extensions) to gauge differentially expressed genes, together with pathway analysis. $\endgroup$
    – EdM
    Commented Jul 23, 2020 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.