2
$\begingroup$

I would like to use recursive feature elimination (implemented via caret in R) to perform feature selection for about 40 test results with 2 possible outcomes. Consequently, RFE either models by Accuracy or by Kappa. Now, I would like to pre-define a specificity threshold since I explicitly care more about specificity than about sensitivity. Is there a way to define this in the training?

Thank you!

Update To be more clear, I have 527 different cases. Each case has 42 results (of a multiplex antigen panel, on a continuous scale) and is classified in 2 possible outcomes by a different test (126 positives and 401 negatives in the gold standard). Now I would like to select important features out of the 42 results to achieve a good prediction of the outcome (positive vs negative). High specificity is especially important.

$\endgroup$
4
  • $\begingroup$ Please say more about the type of model and your purpose in modeling (e.g., inference versus prediction). I believe that some types of models supported by caret allow for other criteria. Also, the criteria you mention are generally poor choices. With only 40 cases you don't have more than 20 members of the minority outcome class, so you would be restricted to about 2 predictors in a logistic regression without overfitting, or a heavily restricted or penalized combination of predictors in other model types, in any event. $\endgroup$ – EdM Jul 21 '20 at 17:40
  • $\begingroup$ @EdM thank you for your comments! I tried to make it more clear. Especially, please note that I have >500 cases and ~40 measurements. These measurements should now be used to predict a binomial outcome that is verified by a different test (the gold standard). The goal is to develop that achieves a sensitivity as high as possible without coming below a certain specificity. $\endgroup$ – Felix Jul 21 '20 at 18:05
  • $\begingroup$ Much clearer, thank you. But are you using logistic regression, boosted trees, or some other type of model? The caret package handles many types of models, and the answer might depend on the specific type of model. Also, you could get very high specificity just by calling all cases "negative" regardless of the antigen results. So what is your trade-off between false positives and false negatives, the relative costs of the two types of errors? Finally, why do you want to cut down on the number of antigens used? Usually, the more information you provide to a prediction system, the better. $\endgroup$ – EdM Jul 21 '20 at 20:36
  • $\begingroup$ Yes, that is right. I am currently classifying using LVQ with repeated cross-validation (but open for suggestions). The antigen results are continuously scaled but could be preprocessed to be pos/neg if necessary. I want to cut down / perform feature selection because some antigens clearly add no information (because they are not significantly different between positive and negative cases). The threshold I am thinking of is something like specificity of >=97% while getting the highest possible sensitivity. I hope that is helpful to give me a hint - thank you! $\endgroup$ – Felix Jul 21 '20 at 21:22
1
$\begingroup$

I'm not sure that learning vector quantization (LVQ) is the best choice for this project. It requires some measure of similarity between cases, to match cases to prototype cases representing each of the classes. You don't say what similarity measure you use; it's often a Euclidean distance calculated over the multi-dimensional predictor space. Unless the distance measure is carefully chosen you might be throwing away information. LVQ can have some advantage for multiple-class problems and for interpreting models, but it has one serious drawback for a binary outcome: all it reports is a yes/no predicted class membership, not a probability of class membership.

As this post explains, even if your ultimate goal is classification it's best to use a criterion that is a proper scoring rule. That's a measure that is optimized when you have the correct probability model, so it requires a probability estimate for the class membership of each case. Logistic regression effectively uses a log-loss scoring rule, but there is a large variety of rules. For example, the equivalent of mean-square error when you have a probability estimate for the class membership of each case and the true membership is the Brier score, another proper scoring rule.

With 126 cases in the smallest class, you probably can get away with about 8 unpenalized predictors out of the 42 in your final model without overfitting, or with a larger number of predictors in a type of model that penalizes individual predictor contributions to avoid overfitting. There are many methods other than LVQ to choose from.

As a preliminary step you might just want to see if any of your 42 predictors has a small range of values relative to its measurement error over all the cases, ignoring their apparent associations with outcome. Since your data aren't too badly imbalanced, that might be an efficient way to cut down on the number of candidate predictors, however you proceed, without biasing your results by "peeking" at the outcomes. Then consider some other possibilities.

Logistic regression with variable selection by LASSO is one good possibility for this type of data, as it can give you a selection of specific predictors that together provide good probability estimates. So if for reasons like cost you want to cut way down from your 42 antigens, that could be a good choice. If there's no problem with analyzing a large number of antigens then you could consider logistic ridge regression instead, which keeps all of the predictors but differentially weights them according to their contributions to outcome while minimizing overfitting.

LASSO and ridge can be unwieldy if you need to consider interactions among the predictors rather than just their individual contributions to the probability estimates. Gradient-boosted trees are another possibility, in which you can include a large number of predictors and specify how many levels of interaction to consider, in a slow-learning process that can minimize overfitting. It's possible to get estimates of predictor importance from such models, which you could in principle use to help design an ultimate testing protocol with further experimental validation.

Those are only a few possibilities; just make sure that the type of model returns probability estimates for the cases.

Once you have good probability estimates you can adjust the probability cutoff for the ultimate classification in a way that matches the relative costs of false-negative and false-positive decisions in your application. There's no need to use the cutoff of p = 0.5 that is so often an explicit or implicit default. If false negatives are very costly to you, as your emphasis on specificity suggests, choose a higher probability cutoff to capture more of the true negatives. But make that choice at the end, after you have a reliable probability model.

$\endgroup$
6
  • $\begingroup$ Thanks for the great clarifications, they were really helpful! I have removed features with very low variance as an unbiased way of feature selection and then compared a number of different linear and nonlinear models (logistic regression/lasso, knn, naive Bayes, svm, classification and regression trees; the nonlinear seemed to perform better. One additional question: Is it appropriate to do LASSO, extract the selected features and use these for the nonlinear models? That seems to further increase the specificity. Thank you very much for your extremely helpful input on that! $\endgroup$ – Felix Jul 22 '20 at 1:38
  • $\begingroup$ @Felix it depends on how you use the results. If you are using them to guide development of a new test that will be validated independently, then you have some flexibility. But not if you're trying to convince others that the model you built on this data set is generally valid. Once you use the results of multiple models to decide which model to select, you necessarily introduce bias. Similar considerations apply to selecting predictors by LASSO and then developing models based on them. $\endgroup$ – EdM Jul 22 '20 at 15:09
  • $\begingroup$ @Felix there are principled ways to combine predictions from multiple models on the same data set. See for example the R SuperLearner package. Your emphasis on high selectivity might benefit from approaches that focus on a region of high estimated probability of positive cases. See the R tmle package as an example. $\endgroup$ – EdM Jul 22 '20 at 15:16
  • $\begingroup$ Yes, that is definitely an important addition. I would like the model to be convincing without further external validation (for now). $\endgroup$ – Felix Jul 22 '20 at 18:50
  • 1
    $\begingroup$ @Felix you have too few cases to separate out training and test sets; see this discussion. Do cross-validation on the entire data set to choose the LASSO penalty factor. To estimate reliability, repeat the entire modeling process on multiple bootstrap samples of the data, evaluating each model's performance against the original data set. The LASSO models might differ in the features maintained, but that typically doesn't affect predictive performance. That does mean you can't claim to have found uniquely "most important" predictors. $\endgroup$ – EdM Jul 24 '20 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.