0
$\begingroup$

I have a data set of 200 data points and each data point has 49 features associated with and a label to predict. These 200 points are different proteins and these 49 features are each protein's pocket (where a small molecule can bind) properties. So I have 200 labels (druggability measure) for each point/protein.

Many times a protein has more than 1 solved crystal structure. In order to give maximum information for building a model, I included all the properties from all the eligible structures for each protein and gave same label to it. I now have 4195 data point for 200 proteins and labels. This kind of comes under data augmentation as I intended to introduce some noise (not random though).

Now my machine learning protocol involves looping through all proteins one by one and putting each protein and all its associated structures in test set for the first iteration. Everything but the test lines go into training set and a model is build using random forest for that iteration. This is repeated for the next protein and so on. So, in the end I have 200 random forest models that I am planning to use as an ensemble later on.

Now the problem is, all my models turn out to be conservative i.e. they predict close to mean of the distribution. I believe it is not over training issue as I am doing leave one protein out cross validation. Correct me if you think this is wrong. Any input on how to solve this issue will be highly appreciated.

There is the script https://github.com/ShipraMalhotra/Regressor/blob/master/Rrf_LOO_MAP95.R and data set files (2 files) https://github.com/ShipraMalhotra/Regressor/blob/master/set4 (200 data points) and https://github.com/ShipraMalhotra/Regressor/blob/master/set4_all_uniq (4195 data points).

Improve this question
$\endgroup$
4
  • $\begingroup$ the question is quite confusing..so your D.V has how many categories? You have 49 independent variables ? $\endgroup$
    – Rahul Agarwal
    Commented Sep 5, 2018 at 14:52
  • $\begingroup$ Voting to close/migrate to CrossValidated. (@MrFlick: you could do the same ...) Also: unless someone very familiar with this type of problem happens to have a good idea, it might take a lot of digging to figure this out. Do you know that your data actually contains any useful information in it ... ??? $\endgroup$
    – Ben Bolker
    Commented Sep 5, 2018 at 14:55
  • $\begingroup$ Yes @Ben-Bolker, data does contain useful information. It is a smaller data set I agree, but there are strong signals on information being useful. I am just not being able to model it in the right way. $\endgroup$
    – Shipra
    Commented Sep 5, 2018 at 15:03
  • $\begingroup$ Yes @Rahul-Agarwal data has 49 independent variable and 1 label. $\endgroup$
    – Shipra
    Commented Sep 5, 2018 at 15:15

1 Answer 1

Reset to default
0
$\begingroup$

Your approach is just wrong.

  1. You are not doing LOOCV since you are putting all the samples from one protein in the training set to build your model, so you are leaving more than one sample out.

  2. LOOCV does not give you 200 models, LOOCV returns the results from 200 models aggregated together, so you are definitely doing something wrong. You should take a look at some built-in functions that will solve this for you, for ex. train function in caret package.

  3. Why are you doing LOOCV? Why not K-fold CV? If you take out all the samples from one protein from your training set your model will struggle to predict this one unknown protein, since it has never seen it before.

  4. Why not use the RF OOB scores? Do you have a good reason not to use them?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.