0
$\begingroup$

I'm working with a dataset in which each row is an event with a number of categorical properties, most of which are also nominal. Each of these properties has around 5-10 possible values, but some columns can have more then 200. There is a column which denotes if an event had a positive, negative or neutral outcome and a column with a certain result code. I already made an analysis on a purely statistical basis of what values have larger or smaller chance to be positive or negative or have a certain result code. This works great when looking at a single property, but when combining them, there are too many categories and they only have a handful of data points in them, making finding statistically significant differences difficult to find.

There are two things I want to do with this data.

First, I want to see what combinations of parameters have the highest or lowest chance to have a positive result. For this, I'm not sure what to do as there are no numerical columns to use to calculate correlations etc. Summing over certain sets where combinations of parameters are the same (so all events where column A has the same value, column B has the same value etc), but I think that the sets would become too small as there is also a lot of missing data.

I also want to use machine learning so the outcome of certain combinations of parameters can be estimated, but again, I'm not sure what to use because of the categorical data. I think one-hot encoding would result in too many features (300+ that are mostly 0) and since the features are nominal, a single column with different numbers would create a bias in the data.

Any insights?

Improve this question
$\endgroup$
2
  • $\begingroup$ Welcome to Cross Validated! Could you please explain “a single column with different numbers would create a bias in the data”? $\endgroup$
    – Dave
    May 25 at 4:25
  • 1
    $\begingroup$ Thanks! If a column would have for instance 10 different possible values (again, non-numerical), you could make them numbers 1 through 10. This created a problem however, the data is ordinal and the value that got assigned to 10 is 'greater' than the one that got assigned to 1, while 5 is somewhere in between. There is no relationship in the original table, while there is one in the single column with 10 different numerical values. $\endgroup$
    – Wannes
    May 25 at 7:30

1 Answer 1

Reset to default
0
$\begingroup$

one hot coding is fine, but you need to use sparse matrices ( you only store position of non zero entries to avoid memory overload). there are many ml libraries that handle this. bag of words language models is the typical example of this. ( eg library scikit-learn, glmnet)

for combinations you might look into market basket analysis which has computationally efficient methods of finding combinations, and various metrics

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! I'll try it and I'll come back if there should be a problem. $\endgroup$
    – Wannes
    May 25 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.