Sklearn - Choosing the right model for supervised learning/classification task

Question

I am beginning to learn how to use scikit-learn and I have a hard time choosing the right model.

Here is my dataset: I have 100 persons. Each person was measured three times: baseline, first event and second event. Each measurement had 100 different markers per person that range from 0.1 to 1000. Additionally I have outcome measurements of each event: outcome can be 0, 1 or 2. My task is to find just a few markers (let’s say 10) that can predict outcome with a good accuracy. If I am right it should be: Supervised learning/Classification problem. What model would be the best?

Thanks for your help!

Answer 1

I'd try visualizing data in 3d, use generalized scatterplot matrix or use PCA (if you know what it is) to project data to 2d and then try to see the structure.

In such low-dimensional dataset it should be easy to see visually which classifier will be the best, using scikit-learn's comparison of different classifiers.

Answer 2

You could use decision trees, they have some desirable properties for your application:

• they will automatically search among your many variables and select the ones that best predict the outcome

• they are very easy to interpret. Basically they tell you that for each record you look first at a specific variable and compare the value to a cutoff. Based on the result of that comparison, you go on the corresponding branch of the tree to the next branching based on some other variable etc. When you arrive in a leaf, the majority class among the training records in that leaf will tell you how to classify your new record.

• they can natively deal with classification into more than two classes. Some algorithms like support vector machines would need workarounds to be able to predict your three possible outcomes.

Random forests would usually have better classification performance than a single tree, but they are more difficult to interpret.

Answer 3

Your data set has the following characteristics and consequences:

• Not to many data points (100 * 3), so the method should work well on small datasets
• Only a subset of the features may be used to predict the outcome, so you search for a sparse model
• You want to interpret your model after learning it.
• Finite, discrete outcome variable so model should be able to handle classification task
• You do not mention it, but often it is best to start simple so the model should be simple and fast.

Starting simple is key to figure out if and how you want to improve. Based on these I would suggest regularized logistic regression. Examples are:

• Logistic regression with L2-norm. This minimizes the sum of the squared weights (besides the prediction errors), i.e. close to zero is good enough.
• Logistic regression with L1-norm. This minimizes the sum of the weights (besides the prediction error), i.e. most weights are exactly zero.

Both models can implemented with LogisticRegression. Note the penalty parameter that specifies the regularization (i.e. norm). Also check LogisticRegressionCV that can optimize the regularization strength for you.