I have trained my random forest model on a 74,000 training examples where each example consists of two proteins Amino Acids sequence (20 characters) and some numeric values representing the similarity between each individual pair of sequences, and finally a numeric value representing the overall similarity between the two proteins, this is a regression model, so I wish for testing I can use just protein sequence and use my trained model to predict the distance between my test case and each of my training protein sequences. A sample of my test case is:
test= G,Y,L,P,P,S, A,N,L,F,S,N, 1,-2,16,-4,-1,11, 21
where "G,Y,L,P,P,S," represent a 6 character fragment of the first protein (my testing) and "A,N,L,F,S,N," represent a 6 character fragment of the second protein (my training database) and the numbers "1,-2,16,-4,-1,11," each number represent the similarity between individual pairs of the 6 Amino Acids, e.g., the similarity between "G and A" is 1 and the similarity between "Y and N" is -2 and the similarity between "L and L" is 16, and so on offcourse the higher the number means the higher the similarity between the pairs of characters, finally the last number "21" represent the sum of the previous 6 numbers which represents the overall similarity between the two sequences.
when I trained my model on 74,000 of such training datasets the correlation between the predicted distance and actual distance was as high as 0.86, however, when I used the trained model in for testing the correlation was very low 0.17, I strongly believe that this over-fitting problem, however, I'm not sure is it due to the may be not good training datasets or that my features aren't strong enough to give a good prediction especially since the ranking of the features according to their importance was very high for all the features? the following is my features importance according to each node purity. any help on how to recognize the source of the overfitting is highly appreciated:
IncNodePurity V3 24564.326 V4 22503.744 V5 25030.450 V6 24583.235 V7 24661.309 V8 20757.662 V9 22985.824 V10 22189.759 V11 23875.170 V12 23674.853 V13 23339.595 V14 19576.762 V15 10169.309 V16 19527.972 V17 5430.600 V18 4415.307 V19 12897.114 V20 3963.717 V21 62614.692