I have trained my random forest model on a 74,000 training examples where each example consists of two proteins Amino Acids sequence (20 characters) and some numeric values representing the similarity between each individual pair of sequences, and finally a numeric value representing the overall similarity between the two proteins, this is a regression model, so I wish for testing I can use just protein sequence and use my trained model to predict the distance between my test case and each of my training protein sequences. A sample of my test case is:

   test= G,Y,L,P,P,S, A,N,L,F,S,N, 1,-2,16,-4,-1,11, 21

where "G,Y,L,P,P,S," represent a 6 character fragment of the first protein (my testing) and "A,N,L,F,S,N," represent a 6 character fragment of the second protein (my training database) and the numbers "1,-2,16,-4,-1,11," each number represent the similarity between individual pairs of the 6 Amino Acids, e.g., the similarity between "G and A" is 1 and the similarity between "Y and N" is -2 and the similarity between "L and L" is 16, and so on offcourse the higher the number means the higher the similarity between the pairs of characters, finally the last number "21" represent the sum of the previous 6 numbers which represents the overall similarity between the two sequences.

when I trained my model on 74,000 of such training datasets the correlation between the predicted distance and actual distance was as high as 0.86, however, when I used the trained model in for testing the correlation was very low 0.17, I strongly believe that this over-fitting problem, however, I'm not sure is it due to the may be not good training datasets or that my features aren't strong enough to give a good prediction especially since the ranking of the features according to their importance was very high for all the features? the following is my features importance according to each node purity. any help on how to recognize the source of the overfitting is highly appreciated:

 V3      24564.326
 V4      22503.744
 V5      25030.450
 V6      24583.235
 V7      24661.309
 V8      20757.662
 V9      22985.824
 V10     22189.759
 V11     23875.170
 V12     23674.853
 V13     23339.595
 V14     19576.762
 V15     10169.309
 V16     19527.972
 V17      5430.600
 V18      4415.307
 V19     12897.114
 V20      3963.717
 V21     62614.692
  • $\begingroup$ have you considered removing everything but the raw data, and re-running it? Look at variable importances too. $\endgroup$ – EngrStudent Apr 16 '16 at 22:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.