When should I use $E(X)$ and when to use $\mu$  for the mean? I see that $E(X)$ and $\mu_x$ are used to refer to the mean of some distribution or variate, but I am not clear on what circumstances each is appropriate.
$E(X)$ seems to always be more often in the context of a random variable
$\mu_x$ seems to appear when referring to the mean of some distribution. for example my books lists the formulas for all the standard distributions similar to for example Binomial mean : $\mu_x=np$
Some times the distinction seems to be whether its the subject of interest, or just being used as a valued variable. eg;
$$E(T_z) = \mu_T(1+\beta t)$$
in that case both $E(T_z)$ and $\mu_T$ both represent the expectation of the variable.
I guess in that sense one is a function that takes an argument, and the other is a variable to be assigned to, and read from.
 A: $E(X)$ explains, in the sense of defining, what the letter $\mu$ is being used to denote (assuming, modulo Wolfgang's comment, that there is something to denote).  In philosophical terminology, $\mu$ is a name whereas $E(X)$ is a definite description. 
That is, of course, a slightly idealised distinction.  Practically, the designation is, by convention, just more rigid for $E(X)$ than for $\mu$ in the sense that $\mu$ doesn't always and everywhere denote the expectation of a random variable, but $E(X)$ nearly always does.  
The utility of using $\mu$ is mostly in situations where you want to take advantage of this loose connection. The two classic examples are when it is a parameter, e.g. in $N(\mu,\sigma^2)$, and when you want to talk simply about an estimate, e.g. $\hat{\mu}$.
In both these contexts $\mu$ is operating more like a name because there is no reason that the parameters of a distribution must to be expectations of any sort, although they often are as in the example, and there is no reason that an expectation must be the thing being estimated.  The parameterisation and the estimation are the point, and a more explicit notation would obscure this.
That said, there is no mathematical reason I can see not to replace $\mu$ with $E(X)$ in a distribution statement.  But the pragmatic impact would be to say something like: "Notice that I have chosen to parameterise with two moments, rather than some other way".  I agree that would be weird, but only because that's not the convention.
This is much the same as the fact that people tend to prefer 
$$Y \sim Poisson(\lambda)$$
$$\log \lambda = \alpha + \beta X$$
rather than, say
$$Y \sim Poisson(\exp (\alpha + \beta X))$$
Because it keeps conceptually distinct claims separate and flags what you're meant to be taking away as a reader. 
I hope that gives some convention-based rather than mathematical guidance about when you might want to use $\mu$ rather than $E(X)$.
A: I would support your suggestion that it depends on what you want to do.
For example, assume that you want to "evaluate" the expected value of $X+\frac{1}{2}Y$, where $X$ and $Y$ are independent random variables. You would create very messy notation using the $\mu$-Notations, whereas 
$$E\left(X+\frac{1}{2}Y\right)=E(X)+\frac{1}{2}E(Y)$$
is as clear as it can get. 
Note that the expected value above is potentially unknown, or at least not explicitly given. 
However, when you refer to the property of the underlying distribution as in $X_i\sim N(\mu_{X_i}, \sigma)$ with $\mu_{X_i}\in\mathbb{R}$, your $\mu_{X_i}$ describes a value that is constant and assumed to be known beforehand. In this case, it would make no sense to write $X_i\sim N(E(X_i), \sigma)$. 
I would say that the "known/unknown value distinction" is probably a rough guideline. However, I can easily imagine situations where you would want to use them differently. (e.g. say your discussion repeats $E(X)$ 20 times within a few rows. Then it might be helpful to define $\mu_X:=E(X)$ and replace it all over that part to keep your notation clean).
As a last comment I have to say that I come with a strong mathematical background and not practical statistics. And as you know, practice is like theory, but different...
