A laboratory wants to evaluate whether a certain form of gel electrophoresis is suited as a classification method for the quality of a certain substance. Several gels were loaded, each with a clean sample of the substance and with a sample that contains impurities. In addition, a molecular marker was also loaded which serves as a reference. The following picture illustrates the setup (the picture doesn't show the actual experiment, I have taken it from Wikipedia for illustration):
Two parameters were measured for each gel and each lane:
- The molecular weight (that is how "high up" a compound wandered during the electrophoresis)
- The relative quantity. The total quantity of each lane is normalized to 1 and the density of each band is measured which results in the relative quantity of each band.
A scatterplot of the relative quantity vs. molecular weight is then produced which could look something like this (it's artificial data):
This graphic can be read as follows: Both the "good" (blue points) and "impure" (red points) substance exhibit two bands, one at around a molecular weight of 120 and one at around 165. The bands of the "impure" substance at a molecular weight around 120 are considerably less dense than the "good" substance and can be well distinguished.
The goal is to determine two boxed (see graphic below) which determine a "good" substance. These boxes will then be used for classification of the substance in the future into "good" and "impure". If a substance exhibits lanes that fall within the boxes it is classified as "good" and else as "impure".
These decision-rules should be simple to apply for someone in the laboratory. That's why it should be boxes instead of curved decision boundaries.
False-negatives (i.e. classify a sample as "impure" when it's really "good") are considered worse than false-positives. That is, an emphasis should be placed on the sensitivity, rather than on the specificity.
I'm am no expert in machine learning. I know, however, that there are quite a few machine learning algorithms/techniques that could be helpful: $k$-nearest neighbors (e.g.
R), classification trees (e.g.
ctree), support vector machines (
ksvm), logistic regression, boosting and bagging methods and many more.
One problem of many of those algorithms is that they don't provide a simple ruleset or linear boundaries. In addition, the sample size is around 70.
My questions are:
- Has anyone an idea of how to proceed here?
- Does it make sense to split the dataset into training- and test-set?
- What proportion of the data should the training set be (I thought around a 60/40-split).
- What, in general, is the workflow for such an analysis? Something like: Splitting dataset -> fit algorithm on the training set -> predict outcome for the test set?
- How to avoid overfitting (i.e. boxes that are too small)?
- What is a good statistic to assess the predictive performance in this case? AUC? Accurary? Positive predictive value? Matthews correlation coefficient?
Assume that I'm familiar with
R and the
caret package. Thank you very much for you time and help.
Here is an example dataset.
structure(list(mol.wt = c(125.145401455869, 118.210252208676, 165.048583787746, 126.003687476776, 170.149347112565, 127.761533014759, 155.523172614798, 120.094514977175, 161.234986765321, 168.471542655269, 156.522990530521, 154.377948321209, 165.365756398877, 167.965538771316, 116.132241687833, 115.143539160903, 156.696830822196, 162.578494491556, 136.830624758899, 123.886594633942, 124.247484227948, 126.257226352824, 160.684010454816, 166.618872115047, 126.599387146887, 165.690375912529, 159.786861142652, 114.520735974329, 125.753594471656, 157.551537154148, 157.320636890647, 171.5759136115, 158.580005438661, 125.647463565197, 130.404710783509, 127.128218318572, 162.144126888907, 161.804616951055, 167.917268243627, 168.582197247178), rel.qtd = c(57.68339235957, 54.0514508510085, 25.0703901938793, 37.6933881305906, 36.6853653723001, 53.6650555524679, 52.268438087776, 52.8621831466857, 43.1242291166037, 46.6771236380788, 38.0328239221277, 40.0454611708371, 44.6406366176158, 40.8238699987682, 51.9464749018547, 54.0302533272953, 37.9792331383524, 48.3853988095525, 38.2093977349102, 42.2636098418388, 42.9876895407144, 40.8018728193786, 40.1097096927465, 38.7432550253867, 39.2633283608111, 43.4673723102812, 53.3740718733815, 49.1067921475768, 52.3002598744634, 44.9847844953241, 44.3014423068017, 44.0191971364465, 47.0805245356855, 55.0124134796556, 57.9938440244052, 62.8314454977068, 45.8093815891894, 43.2300677500964, 39.4801550161538, 51.6253515591173), quality = structure(c(2L, 2L, 2L, 1L, 2L, 2L, 1L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 1L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L), .Label = c("bad", "good"), class = "factor")), .Names = c("mol.wt", "rel.qtd", "quality"), row.names = c(10L, 14L, 47L, 16L, 57L, 54L, 45L, 12L, 43L, 67L, 25L, 21L, 1L, 55L, 20L, 22L, 37L, 15L, 8L, 38L, 46L, 64L, 51L, 65L, 52L, 61L, 63L, 32L, 50L, 27L, 19L, 69L, 23L, 42L, 6L, 48L, 11L, 13L, 5L, 71L), class = "data.frame")