# How to predict the second most likely categorical feature?

Suppose I have a dataset $X$ that contains both numerical and categorical features. For concreteness let's assume that one of the categorical features is a sample's color, and that it has been properly preprocessed via one-hot encoding. A property of $X$ is that about half the samples are described by one color, and the other half by two.

For instance the data could look like

Color A     Color B      Height (m)     Weight (kg)     other features

Red         Blue         0.5            1               ...
Green       NaN          0.2            1.2             ...
Purple      Red          0.3            0.5             ...
Blue        NaN          0.45           0.75            ...


I was wondering whether it would be possible to predict the most likely second color for monochromatic samples given the information contained in $X$, and if so what is the best way to frame this question as a statistical/machine learning model?

This problem seems related to clustering, although I have a few issues with taking this point of view. If I were to naively cluster the samples in $X$ that have two colors, then nothing guarantees that the clusters would be based on the color feature only. In fact clustering would most likely group samples with different colors together, invalidating my goal from the start.

Another point of view would be to treat the monochromatic samples as having missing data. I have heard that Expectation Maximization can be used to replace missing data, but it still does so by clustering data using a mixture model, and I go back to my argument in the previous paragraph.

Any guidance on how to approach this problem, if possible, would be greatly appreciated.

• I'm confused by your description. If color is a single feature, such that you can dummy-code it, how can a single case have two colors? (Don't use the word "samples" to mean "cases" because in statistics, a "sample" means a set of cases, not a single case.) Jul 1 '17 at 1:11
• @Kodiologist You can think of it as being two separate features, one of them being primary color and the other secondary color, with the latter being NaN for monochromatic observations. It's then straightforward to dummy-code the combination of the two so that some observations effectively have two colors. I'm intrigued in knowing what the most likely secondary color would be based on the information contained in $X$. Jul 1 '17 at 1:22
• Is there anything to distinguish a case with primary color A and secondary color B from a case with primary color B and secondary color A? Or is the distinction arbitrary? Jul 1 '17 at 1:29
• I am not clear why this is not simply a classification problem to predict the second color.
– G5W
Jul 4 '17 at 1:12
• Since you say certain values of color A imply no color B, there does appear to be a relationship? And by any means, I cannot see any relationship to clustering here. Jul 5 '17 at 6:12

## 1 Answer

You could try multinomial logistic regression, see for example Multinomial logistic regression vs one-vs-rest binary logistic regression and search this site. Use the Color variable as a (nominal) outcome variable, and the other variables as predictors.

From the fitted model you will get fitted/predicted probabilities for each color, for each case. Then look at the second largest fitted probability.

As for the problems with missing data, look into https://stats.stackexchange.com/questions/tagged/data-imputation and https://stats.stackexchange.com/questions/tagged/multiple-imputation.