I am looking for a reasonable estimate of how many total connections (weights) there are in a given neural net. My prior would be that if it is a vanilla ANN with 3 layers of 1200 neurons apiece it would be: (1200^2)*2 . Since each neuron on each layer connects to every other neuron there are 1200^2 between each layer. On a 3 layer ANN there are 2 "in-between" layers. Is this correct? Also, how would this change for RNN's with self-directed loops (and LSTM RNN's)?
I think you have 1200*2 in-between layers. So, total connection will be (1200^2)*2 (what you wrote is correct.) This is correct, if you are not considering any biased weights. If you add biased weights to each neuron then it would be (1200^2)*2+1200*2 number of connections.
In RNN, each node will be connected twice with itself (Node1 --> Node1 --> Node1). So, (1200^2)*2+(1200^2)*2 (without considering any biased weights). You may generalize it.