Related answer: http://stats.stackexchange.com/a/66420/25433
If we choose to use these toolkits, besides that it is convenient, sometimes we might lose control of what is really happening.
This is not at all specific to open source toolkits. In fact, in open source programs it's easier to make changes where and when necessary. Additionally, if you add fundamental functionality you can become a contributor for the package to get a win-win situation.
If the learning system is entirely implemented by ourselves, there could not be that much confusion, since we know exactly what happens.
Probably. On the other hand, implementing from scratch takes a lot of knowledge, time and effort, especially if you want to do proper unit testing and what not (which most popular open source software solutions do provide). Also note that machine learning practitioners are often not software engineers, which makes (re)implementation far from trivial.
So what is the right attitude towards open source machine learning toolkits Just use them for the first phase, and then implement manually, without the help of learning toolkits in order to have more control over algorithms, or we can just rely on such kinds of learning toolkits?
The most important thing in my opinion is to be aware of what exists (for example here and here). More often than not, you can find what you need in open source (maybe after some fiddling here and there). Don't reinvent the hot water.
Never assume that open source software (or any software for that matter) is bug-free. If you get weird results, question the tools you used. The larger the userbase of a given package, the safer you are from critical bugs in the software. Users are inevitably the best testing bed for any software system. You can assume to be fairly safe from major issues in popular packages like weka, shogun, opencv, ...
This is another related question that might be particularly relevant for machine learning. A lot of academic open-source software is released as a proof-of-concept. Such implementations -- often at the cutting edge of algorithms with fairly few users -- should be treated with care. The smaller the userbase, the less scrutiny software has received.