2
$\begingroup$

Having built a regression model with an ordinal response variable and predictors comprised of categorical and continuous nature, I have some questions that pertain to one of the final goals, i.e. suggest another class for certain observations (which ones is unknown) in the training set. I approach this problem with a sort of outlier analysis methodology:

  1. Is there any way to know if the labels of certain observations in the training set were misplaced to begin with?

  2. Even if I do find some observations that are likely "outliers", based on model's studentized residuals OR Mahalanobis Distance etc, how do I know that it is not due to model bias? Since the model is trained on mislabeled data.

  3. How do I go about choosing another class label for these records?

Thank you for any thoughts !

$\endgroup$
3
$\begingroup$

Is there any way to know if the labels of certain observations in the training set were misplaced to begin with?

Not from the values in the data set; you may be able to trace back through the data source to identify potential mislabelling, but normally you can't know that values were mislabelled just from looking at them.

There are some exceptions - if your data are on different types of operations, a Caesarian section on a male, or a person 183 inches tall may be a pretty clear indication of 'mislabelled' data, but outside of that kind of logical identification, strange values cannot normally be known to be mislabelled.

Even if I do find some observations that are likely "outliers", based on model's studentized residuals OR Mahalanobis Distance etc, how do I know that it is not due to model bias? Since the model may not capture every detail.

You usually don't -- outliers are outliers with respect to some model for the data. A different model may regard values as perfectly reasonable; you might come to regard some observations as unusual under a variety of reasonable models, or perhaps even conclude that no reasonable understanding of the data would make those points in accord with the rest but - especially with categorical responses - such situations aren't the most common.

How do I go about choosing another class label for these records?

If the data are implausible, you might treat the implausible value as missing and use missing data methods. If the label were missing, how would you fill it in?

One approach is to use model-based imputation: If you were making a prediction at that set of predictor-values, what would your predicted value be?

You may want to consider that there may be several quite plausible categories; it's worth considering multiple imputation methods.

There are many other approaches to imputing missing values, and numerous books with sections on the topic (there are also whole books, such as the one by Little and Rubin).

For example, there's a short discussion of a variety of approaches in Gelman and Hill's "Data Analysis Using Regression and Multilevel/Hierarchical Models". See here. While that's not explicitly about your particular model, many of the approaches discussed do carry across.

$\endgroup$
  • $\begingroup$ +1 @Glen_b, this is useful! Well, the data aren't implausible, so I believe the mislabeled identification may have something to do with observations closer to the decision boundaries (well separated from its class means). Here is my understanding now, for (1): any model based outlier measures should do the trick. (2) can be worked upon by having multiple models and taking an intersection. Those obs found in (2) can be considered unlabeled and imputed (or predicted). $\endgroup$ – tool.ish Jun 24 '14 at 6:46
1
$\begingroup$

Regarding 1. Only by looking at the data itself and seeing if any were mislabeled

Regarding 2. I am not sure I understand the question.

Regarding 3. The output of an ordinal logistic regression will allow you to calculate an estimated probability of each class for each subject. Depending on your software, it is probably possible to have this done by the program itself.

$\endgroup$
  • $\begingroup$ Thanks @Peter, For (1) I'm not sure by just looking at the data, how can I find mislabeled instances (by mislabeled I mean that some labels in training set were wrong to begin with). In (2), I meant that since the model was trained on some noisy mislabeled data, the residuals would themselves be incorrect to reflect any true outliers. (3) Any references on that? $\endgroup$ – tool.ish Jun 23 '14 at 13:54
  • $\begingroup$ For 1) I meant looking at the data itself and seeing if there were typos, etc. For 2) Then there is nothing to do about that. That's how regression works For 3) Any book on ordinal logistic or the documentation for e.g SAS PROC LOGISTIC or R or whatever. $\endgroup$ – Peter Flom - Reinstate Monica Jun 23 '14 at 20:45
  • 1
    $\begingroup$ Hmm. This is the way to go, as all the answers are pointing out. Appreciate your advise @Peter. $\endgroup$ – tool.ish Jun 24 '14 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.