7
$\begingroup$

Background

A laboratory wants to evaluate whether a certain form of gel electrophoresis is suited as a classification method for the quality of a certain substance. Several gels were loaded, each with a clean sample of the substance and with a sample that contains impurities. In addition, a molecular marker was also loaded which serves as a reference. The following picture illustrates the setup (the picture doesn't show the actual experiment, I have taken it from Wikipedia for illustration):

Example of a gel electrophoresis

Two parameters were measured for each gel and each lane:

  1. The molecular weight (that is how "high up" a compound wandered during the electrophoresis)
  2. The relative quantity. The total quantity of each lane is normalized to 1 and the density of each band is measured which results in the relative quantity of each band.

A scatterplot of the relative quantity vs. molecular weight is then produced which could look something like this (it's artificial data):

Example scatterplot

This graphic can be read as follows: Both the "good" (blue points) and "impure" (red points) substance exhibit two bands, one at around a molecular weight of 120 and one at around 165. The bands of the "impure" substance at a molecular weight around 120 are considerably less dense than the "good" substance and can be well distinguished.


Goal

The goal is to determine two boxed (see graphic below) which determine a "good" substance. These boxes will then be used for classification of the substance in the future into "good" and "impure". If a substance exhibits lanes that fall within the boxes it is classified as "good" and else as "impure".

These decision-rules should be simple to apply for someone in the laboratory. That's why it should be boxes instead of curved decision boundaries.

False-negatives (i.e. classify a sample as "impure" when it's really "good") are considered worse than false-positives. That is, an emphasis should be placed on the sensitivity, rather than on the specificity.

Example decision boxed


Question

I'm am no expert in machine learning. I know, however, that there are quite a few machine learning algorithms/techniques that could be helpful: $k$-nearest neighbors (e.g. knn in R), classification trees (e.g. rpart or ctree), support vector machines (ksvm), logistic regression, boosting and bagging methods and many more.

One problem of many of those algorithms is that they don't provide a simple ruleset or linear boundaries. In addition, the sample size is around 70.

My questions are:

  • Has anyone an idea of how to proceed here?
  • Does it make sense to split the dataset into training- and test-set?
  • What proportion of the data should the training set be (I thought around a 60/40-split).
  • What, in general, is the workflow for such an analysis? Something like: Splitting dataset -> fit algorithm on the training set -> predict outcome for the test set?
  • How to avoid overfitting (i.e. boxes that are too small)?
  • What is a good statistic to assess the predictive performance in this case? AUC? Accurary? Positive predictive value? Matthews correlation coefficient?

Assume that I'm familiar with R and the caret package. Thank you very much for you time and help.


Example data

Here is an example dataset.

structure(list(mol.wt = c(125.145401455869, 118.210252208676, 
165.048583787746, 126.003687476776, 170.149347112565, 127.761533014759, 
155.523172614798, 120.094514977175, 161.234986765321, 168.471542655269, 
156.522990530521, 154.377948321209, 165.365756398877, 167.965538771316, 
116.132241687833, 115.143539160903, 156.696830822196, 162.578494491556, 
136.830624758899, 123.886594633942, 124.247484227948, 126.257226352824, 
160.684010454816, 166.618872115047, 126.599387146887, 165.690375912529, 
159.786861142652, 114.520735974329, 125.753594471656, 157.551537154148, 
157.320636890647, 171.5759136115, 158.580005438661, 125.647463565197, 
130.404710783509, 127.128218318572, 162.144126888907, 161.804616951055, 
167.917268243627, 168.582197247178), rel.qtd = c(57.68339235957, 
54.0514508510085, 25.0703901938793, 37.6933881305906, 36.6853653723001, 
53.6650555524679, 52.268438087776, 52.8621831466857, 43.1242291166037, 
46.6771236380788, 38.0328239221277, 40.0454611708371, 44.6406366176158, 
40.8238699987682, 51.9464749018547, 54.0302533272953, 37.9792331383524, 
48.3853988095525, 38.2093977349102, 42.2636098418388, 42.9876895407144, 
40.8018728193786, 40.1097096927465, 38.7432550253867, 39.2633283608111, 
43.4673723102812, 53.3740718733815, 49.1067921475768, 52.3002598744634, 
44.9847844953241, 44.3014423068017, 44.0191971364465, 47.0805245356855, 
55.0124134796556, 57.9938440244052, 62.8314454977068, 45.8093815891894, 
43.2300677500964, 39.4801550161538, 51.6253515591173), quality = structure(c(2L, 
2L, 2L, 1L, 2L, 2L, 1L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 2L, 1L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 1L), .Label = c("bad", "good"), class = "factor")), .Names = c("mol.wt", 
"rel.qtd", "quality"), row.names = c(10L, 14L, 47L, 16L, 57L, 
54L, 45L, 12L, 43L, 67L, 25L, 21L, 1L, 55L, 20L, 22L, 37L, 15L, 
8L, 38L, 46L, 64L, 51L, 65L, 52L, 61L, 63L, 32L, 50L, 27L, 19L, 
69L, 23L, 42L, 6L, 48L, 11L, 13L, 5L, 71L), class = "data.frame")
$\endgroup$
  • 1
    $\begingroup$ There is a lot to discuss here. But I'm caught up on the model issue. If you're planning on using this daily, are there models other than decision tree (rpart) that would be practical? Even a logistic model (unless you converted to a risk score) wouldn't seem practical. $\endgroup$ – charles Nov 13 '13 at 19:32
  • 3
    $\begingroup$ Based on "the decision has to be a box" you pretty much answered your own question: you need a decision tree. $\endgroup$ – Marc Claesen Nov 13 '13 at 19:40
  • $\begingroup$ @charles: Any model that produces simple rules so that any person doing those tests can apply them without the use of a modeling environment is fine. Ideally it should go something like: "The molecular weight of this band is within this region and the relative quantity within this region" -> The sample is "good", etc. That being said: The whole exercise is an evaluation of the method. Maybe the answer is simply that it's not possible to come up with some decision rules for this. $\endgroup$ – COOLSerdash Nov 13 '13 at 19:41
2
$\begingroup$

This is a really deep question. I am going to try to answer it for your specific case and make broader points at the same time.

Has anyone an idea on how to proceed here?

How to proceed from here is really a question of which method to use. The answer for your particular case seems to be CART (Classification and Regression Trees). CARTs will allow you to get nice, rectangular regions for prediction, but they are very noisy. That is why Random Forests and other similary algorithms were created. Random Forests and the like trade clarity for an improvement in prediction.

In a general sense, the choice of method depends on two things: what your goals are for the analysis and how well the model fits your data. Nothing is stopping you from trying a couple of methods and choosing which fits bets.

Note that trees can output a continuous probability. You can change the threshold for classifying an event (good in your case) to raise or lower sensitivity. Examine the ROC curve to see how the two relate before doing so.

Does it make sense to split the dataset into training- and test-set?

Yes and no. You should never measure the performance of the classification algorithm on the same data that it was fit on. You will end up drastically overfitting the data and over estimating.

In this case, a training and test set does not make sense because of the small sample size (~70). Instead, I would use Leave One Out Cross Validation (LOOCV).

The algorithm goes like this:

  1. Hold one observation out.
  2. Fit the model on the data except the hold out from 1.
  3. Classify the hold out from 1.
  4. Repeat 1-3 until all observations have been held out.
  5. Estimate fit based on the classifications from 3.

For LOOCV, the final model is the model fit on the entire dataset.

What proportion of the data should the training set be (I thought around a 60/40-split).

In general, 60/40 or 50/50 splits are good. If you have enough data, do 50/25/25 where the second 25% goes to a validation set. When you have a validation set, you fit the model on the test set then check its performance on the test set. If you think the model should be tweeked, do so and the retest on the test set until you are satisfied. Then, once the model is locked down, classify the data in the validation set. The results from the validation set will be those that you report.

For your case, I would recommend LOOCV.

How to avoid overfitting (i.e. boxes that are too small)?

Most algorithms have control parameters (e.g. cost for SVMs). In the tree package in R, there are several control parameters to help prevent overfitting (see tree.control).

What is a good statistic to assess the predictive performance in this case? AUC? Accurary? Positive predictive value? Matthews correlation coefficient?

It depends on the purpose. I would recomend at lease reporting Accuracy, Sensitivity, Specificity, Positive and Negative Predictive Values. Since you said the emphasis should be on sensitivity, that should be a focus. AUC is also commonly used but is noisy.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 for this detailed and practical answer. Thank you very much! I have one further question: Say I follow your advice and don't split my dataset (because of the rather small sample size). Instead, I apply LOOCV to measure the classification error. What model do I report in the end? As I didn't split my dataset, I will report the model I have built on the basis of the whole dataset, right? $\endgroup$ – COOLSerdash Nov 14 '13 at 16:07
  • 1
    $\begingroup$ You're welcome. I would use the model built on the whole dataset. $\endgroup$ – Christopher Louden Nov 14 '13 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.