I am beginning to learn how to use scikit-learn and I have a hard time choosing the right model.

Here is my dataset: I have 100 persons. Each person was measured three times: baseline, first event and second event. Each measurement had 100 different markers per person that range from 0.1 to 1000. Additionally I have outcome measurements of each event: outcome can be 0, 1 or 2. My task is to find just a few markers (let’s say 10) that can predict outcome with a good accuracy. If I am right it should be: Supervised learning/Classification problem. What model would be the best?

Thanks for your help!

up vote 1 down vote accepted

I'd try visualizing data in 3d, use generalized scatterplot matrix or use PCA (if you know what it is) to project data to 2d and then try to see the structure.

In such low-dimensional dataset it should be easy to see visually which classifier will be the best, using scikit-learn's comparison of different classifiers.

You could use decision trees, they have some desirable properties for your application:

  • they will automatically search among your many variables and select the ones that best predict the outcome

  • they are very easy to interpret. Basically they tell you that for each record you look first at a specific variable and compare the value to a cutoff. Based on the result of that comparison, you go on the corresponding branch of the tree to the next branching based on some other variable etc. When you arrive in a leaf, the majority class among the training records in that leaf will tell you how to classify your new record.

  • they can natively deal with classification into more than two classes. Some algorithms like support vector machines would need workarounds to be able to predict your three possible outcomes.

Random forests would usually have better classification performance than a single tree, but they are more difficult to interpret.

Your data set has the following characteristics and consequences:

  • Not to many data points (100 * 3), so the method should work well on small datasets
  • Only a subset of the features may be used to predict the outcome, so you search for a sparse model
  • You want to interpret your model after learning it.
  • Finite, discrete outcome variable so model should be able to handle classification task
  • You do not mention it, but often it is best to start simple so the model should be simple and fast.

Starting simple is key to figure out if and how you want to improve. Based on these I would suggest regularized logistic regression. Examples are:

  • Logistic regression with L2-norm. This minimizes the sum of the squared weights (besides the prediction errors), i.e. close to zero is good enough.
  • Logistic regression with L1-norm. This minimizes the sum of the weights (besides the prediction error), i.e. most weights are exactly zero.

Both models can implemented with LogisticRegression. Note the penalty parameter that specifies the regularization (i.e. norm). Also check LogisticRegressionCV that can optimize the regularization strength for you.

  • Thanks for the rapid answer. I will look into these three models. – ZMK Aug 29 '17 at 7:24
  • 1
    Question asked for classification model. Linear regression is not used for classification. – Jakub Bartczuk Aug 29 '17 at 9:35
  • That's true, I will check in sklearn for the logistic regressions variant – Pieter Aug 29 '17 at 11:02
  • Check, now advising LogisticRegression – Pieter Aug 29 '17 at 15:42

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.