While working on some problem in bioinformatics I applied bayesian network algorithm for classification purposes. As predictors I took a window of sequence of aminoacids and for dependent variable I used some feature of interest of a central acid.
For DAG estimation I used several different algorithms which gave me identical network structures, and all of them used BIC score during the DAG search. Obtained DAG structure consisted of two chains: X7-X5-X3-X1, and X6-X4-X2, while all Xs where connected with Y. Using this network I got my classification accuracy up to 65%.
The problem is that motivation for my small research was to check if Bayesian Networks can beat Markov chain procedure developed by my collegues for same problem. Basically my collegues obtained 81% accuracy by applying simple Bayesian procedure for model in which sequence of amino-acids was a Markov chain (they also used some pretty tricky empirical methods to obtain such a great result).
So, because my collegues showed that Markov Chain of second order gives best classification accuracy, I just modified my Bayesian Network by connecting two chains described above, so that each amino-acid was connected with two others (not one as before). I then used obtained structure and got classification accuracy 92% (which is pretty awesome for my task).
So, my questions are:
How can I justify changes of DAG I made to improve my model. Of course I can (and I will) mention results of my collegues, but still, its more intuitive explanation. Maybe there is better reasoning then intuition here
What can be a possible reason for algorithms not finding optimal structure? (My current explanation is that BIC has just too strong penalty for model's complexity and algorithms just got into some local extremum, and weren't able to get out of it because of BIC's complexity penalty. DO you think it's worth to try and develop some more accurate score for my particular problem, to obtain correct DAG structure? What may be the other explanations?)