3
$\begingroup$

I have a large data set consisting of ca. 40 categorical data items and a few interval data items (real numbers, less than 5 such items). Most categories should have a lot of values that repeat themselves over and over and very few that appear very rarely. Some categories are also overcategories of others (like country and city). The outcome of each data is either 1 if the event occured or 0 if it did not occur.

The idea is to calibrate a machine learning model or a staistical model that can predict for every given data row the probability that the outcome is 1. The data set I will use will have at least 1 million rows.

What machine learning approaches and statistical models will perform well on such a task? My initial thoughts are logarithmic regression and support vector machines (with extensions like random forest)

How do I deal with the interval data items? The easiest approach is obiously to convert different ranges into categories, which I think will not be a problem.

What libraries and tools can I use, when my data set has a size of 10 GB? I am interested in tools/libraries that include machine learning algorithms but also statistical tools to help me find attributes with significant influence on the outcome. I can code in Java and C++ at the moment. I looked into Root, a data analysis tool from CERN for lare data sets and its machine learning module TMVA but it can only handle real numbers and integers as input as far as I know.

$\endgroup$
1
$\begingroup$

These are classical challenges in any Big Data Machine Learning problems.

Most categories should have a lot of values that repeat themselves over and over and very few that appear very rarely.

For categorical features where you have this occurs, you can use one-hot encoding to create additional features.

few interval data items (real numbers, less than 5 such items)

For interval data items you can encode it using label encoders. If any business insights can be drawn like mean, median, mode, frequency etc. then the same should be reflected in your approach.

Some categories are also overcategories of others (like country and city). The outcome of each data is either 1 if the event occured or 0 if it did not occur.

Since these dependent variables already cleanly reflect the independent variable there is little to be done here. I don't see a way to separate the features into hierarchy either.

What machine learning approaches and statistical models will perform well on such a task? My initial thoughts are logarithmic regression and support vector machines (with extensions like random forest)

You can start with logistic regression. However since your data is big you might be able to find some distribution to the input feature vectors. In that case using Gaussian Discriminant Analysis might help. You can model p(y) as a Bernoulli random variable and predict p(x|y=0) and p(x|y=1) as Normal distribution.

SVM algorithm can also be used but the training time will be high. Also, we don't know the best choice of the kernel before looking at the performance on learning curve.

What is your evaluation measure through? Is it accuracy or precision/recall? If it is latter then using Random Forest will not work, instead you can go by simple decision trees.

$\endgroup$
1
$\begingroup$

A very specific suggestion to serve as an example: Check SGD classifier with class_weight='balanced' from scikit_learn. Make sure the label you want to predict comes out as 1 in your encoding and you use e.g. precision as scoring metric if you were to search for model hyperparameters with GridsearchCV or RandomsearchCV.

$\endgroup$
  • $\begingroup$ Make sure the definition if precision is a proper accuracy scoring rule, i.e., is not optimized by a bogus model. $\endgroup$ – Frank Harrell Jul 17 '16 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.