Does anyone know an outlier detection method for a univariate categorical (nominal, unordered) statistical variable? Without any assumptions about the categorical variable distribution (non-parametric method)?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Welcome to our site. Have you considered searching it for existing answers to your question? What would an "outlier" be in such a case? $\endgroup$– whuber ♦Commented Nov 14, 2015 at 19:27
-
$\begingroup$ It is a quite typical situation, and practical need to detect outliers in nominal data. Take for example communication data in closed group. There is a typical pattern in communication pairs, and sometimes communication occurs between nodes which never communicated before. This is outlier and it can not depend on anything numerical ( the amount of information exchanged, bandwidth etc) $\endgroup$– kakazCommented Oct 27, 2021 at 12:40
-
$\begingroup$ Communication between two nodes sounds bivariate. How do you see it as univariate? $\endgroup$– DaveCommented May 17, 2023 at 19:55
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
1
Think about your question once more because you ask for an algorithm to detect which of these is an outlier:
- London
- Munich
- Paris
- Barcelona
Nominal scale means that you have just labels of items like city names or car brands. You can't tell which is an outlier without additional info.
-
1$\begingroup$ What about London, Munich, Paris, Barcelona, Kuala Lumpur? $\endgroup$– DaveCommented Mar 3, 2021 at 20:51
$\begingroup$
$\endgroup$
5
Outliers are extreme values that we come across, where they may be influential to the model or not. When it comes to categorical data (say Gender: as in male and female). There's no way of any outlier detection in that. If you mean something like this: You take a sample of 10 with 9 males and 1 female. So you mean that "1 female" is an outlier? NO! It's just the composition of the sample which you have selected.
-
5$\begingroup$ The spirit is right, but the example isn't. If the data are supposedly for pregnant females, a single male is a puzzle, and may be a coding error or something more complicated with a transgender patient. A set of categories doesn't imply that seeing observations for one or more categories in a dataset might not be a real puzzle. Much depends on the context as well as the data. $\endgroup$– Nick CoxCommented Feb 1, 2019 at 9:30
-
$\begingroup$ Thanks for the comment. I read it several times, but hard to capture what is referred to from the word "puzzle". Could you please clarify it further? $\endgroup$ Commented Feb 1, 2019 at 9:38
-
$\begingroup$ Puzzle = problem or difficulty. I don't know how to clarify further. Puzzle is just an everyday English word used informally as something that should puzzle a researcher: Why do I have this value in my data? Sorry, but I don't know any other language used in Sri Lanka to offer a translation. $\endgroup$– Nick CoxCommented Feb 1, 2019 at 9:42
-
$\begingroup$ Anyhow, thank you so much for your comments/ editing and so on I consider it as a help to get used to this site. Now anyway, I got the meaning of your comment. Of course I do agree, that it depends on the data set which we consider. If the population is full of females, then a male data would be erroneous. That I suppose, the analyst should have an idea about the population where the sample is drawn from. Then a single male representation from a female population can be assumed to be a data entry error. $\endgroup$ Commented Feb 1, 2019 at 9:49
-
1$\begingroup$ Not necessarily. As I said, transgender patients can be hard to fit in. People who regard themselves as male can still get pregnant, depending .... $\endgroup$– Nick CoxCommented Feb 1, 2019 at 10:41
$\begingroup$
$\endgroup$
1
I disagree with the statement that there are no ways to identify outliers for categorical variables. In the same way you do it for numerical ones -- out of 1000 data points - in 99.8% cases fuel price is under 3usd/l and in 0.02% cases fuel price is $40/L. Statistically you want to remove the 0.02% from your data so your model accuracy is higher. For numerical vars the rule is around the interquartile range where outliers lie outside (Q1-1.5IQR ; Q3+1.5IQR). In the same way we can consider outliers for categorical data - only we look at the lower end of frequency percenatge. You have 99.8% cases that says Fuel price="low" and 0.02% cases where Fuel price="high". You would remove the 0.02% from your training data. One way of doing it is to just set a threshold of up to 5% which would apply on non-missing data. The other would be to sort your var by ascending freq, number code your categories from small to large, leaving -- say 10 points between each --, and then use the same method as for the numerical outliers, only range would be <Q1-1.5IQR since anything above Q3 is high freq and you don't want eliminated.
-
3$\begingroup$ Welcome to CV. You lost me at "you would remove" due to its implicit but incorrect assumption that outliers are to be removed automatically. The other material is questionable, too, because (1) "the" rule you give is one of many to be used judiciously and appropriately rather than blindly followed and (2) it will almost never be the case that extremely low relative frequencies would be identified using your rule. $\endgroup$– whuber ♦Commented May 17, 2023 at 18:47
$\begingroup$
$\endgroup$
7
As per my understanding, there is no concept of outliers detection in categorical variables(nominal), as each value is count as labels. Based on frequency(Mode), we can't do outliers treatment for categorical variables. Plz prove me wrong :)..
-
2$\begingroup$ No proof is needed or possible. This is not a topic for formal logic, just statistical judgment. But you need at least informally a definition of outlier. Here's one of mine for this context: An outlier is a unusual data value that causes surprise or needs some attention. If asked for gender, suppose 50 people say male, 49 people say female and 1 person says "none of your business". Then the last value could be regarded as an outlier in this sense only: you need to decide what to do about it. An outlier is not necessarily wrong or self-evidently to be ignored. $\endgroup$– Nick CoxCommented Sep 3, 2020 at 10:07
-
$\begingroup$ Yes, same way like if I had a below table having colors, based on frequency I can't say "Green' color is an outliers.It's applicable if the record counts in Millions as well. Red Red Red Black Black Black Green $\endgroup$– KapilCommented Sep 3, 2020 at 10:13
-
$\begingroup$ I don't understand what you are saying but I think you're agreeing with my comment. The mode is usually irrelevant in assessing outliers. $\endgroup$– Nick CoxCommented Sep 3, 2020 at 10:42
-
$\begingroup$ Yes, for categorical variables there is no concept of outliers. $\endgroup$– KapilCommented Sep 8, 2020 at 10:42
-
$\begingroup$ @NickCox: Also, can you clarify one more doubt regarding K-Mode Clustering. Like - 1) how we do feature engg. or dimensional reduction for categorical variables for ML model. 2) And how you evaluate your K-Model Clustering ML Model is giving good result (like- accuracy, etc). $\endgroup$– KapilCommented Sep 8, 2020 at 10:45