I have a data set of 200 data points and each data point has 49 features associated with and a label to predict. These 200 points are different proteins and these 49 features are each protein's pocket (where a small molecule can bind) properties. So I have 200 labels (druggability measure) for each point/protein.
Many times a protein has more than 1 solved crystal structure. In order to give maximum information for building a model, I included all the properties from all the eligible structures for each protein and gave same label to it. I now have 4195 data point for 200 proteins and labels. This kind of comes under data augmentation as I intended to introduce some noise (not random though).
Now my machine learning protocol involves looping through all proteins one by one and putting each protein and all its associated structures in test set for the first iteration. Everything but the test lines go into training set and a model is build using random forest for that iteration. This is repeated for the next protein and so on. So, in the end I have 200 random forest models that I am planning to use as an ensemble later on.
Now the problem is, all my models turn out to be conservative i.e. they predict close to mean of the distribution. I believe it is not over training issue as I am doing leave one protein out cross validation. Correct me if you think this is wrong. Any input on how to solve this issue will be highly appreciated.
There is the script https://github.com/ShipraMalhotra/Regressor/blob/master/Rrf_LOO_MAP95.R and data set files (2 files) https://github.com/ShipraMalhotra/Regressor/blob/master/set4 (200 data points) and https://github.com/ShipraMalhotra/Regressor/blob/master/set4_all_uniq (4195 data points).