When comparing standard deviation and entropy in context of statistics and related areas, I think that it is important to realize the difference between two notions of the entropy concept: entropy as a measure of variability, volatility, chaos (this meaning is usually implied in physics and similar domains) and entropy as a measure of an average information in a message (this meaning is usually implied in domains, based on Shannon's theory of information). However, despite obvious surface differences between the notions, the above-mentioned two dimensions reflect the close parallels between physics-based entropy and information theory-based entropy concepts. A discussion on this topic is beyond the scope of my answer, but this article is IMHO a good start.
The entropy formula, which you were confused about, comes from the theory of information (see this section) and is the basis for the use of entropy via the concept of information gain (notice the similarity in the formulas). If I understand correctly, all those types of entropy are particular (contextual) cases of a mathematics-based generalized concept of entropy in dynamic systems.
In terms of your particular question on potential use of standard deviation (SD) as a substitute measure for decision trees, I have to say the following:
- yes, it is possible to use SD as a substitute for entropy (information gain, to be more accurate);
- it seems that your statement about wanting higher SD as a criteria for attribute splitting is wrong - you need higher information gain, which can be substituted by standard deviation reduction, not SD itself. This nice page explains the idea and algorithm behind it rather well.
Finally, I would like to share two resources for reducing confusion and providing more details on the topic. First, this discussion is useful to see why your statements "higher the standard deviation, lesser the entropy" and "lesser the SD, more the entropy" [original style preserved] are incorrect. Second, this paper, despite its financial focus, presents potential reasons for preferring using entropy to standard deviation. Let me summarize them in the following list:
- entropy is a more general measure and supports a wider range of data types;
- entropy incorporates more information than SD (thus, making models more realistic);
- entropy is distribution-free, that is not dependent on a particular distribution (less errors);
- entropy satisfies the first order condition (used in optimization and econometric models);
- entropy also serves as a measure of dispersion (hence, playing an SD's role).
Reasons for not preferring using entropy to SD also should be noted and include the former's complexity and potential statistical bias, related to considered degrees of freedom of a model.