I am looking for ways to prove that the moment generating function of $X'AX$ given that $X \sim N(\vec{\mu}, \vec{\Sigma})$ and $A$ is symmetric is defined as:
$$M_{X'AX}(\vec{t})= \frac{1}{|I-2tA\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}\mu'[I-(I-2tA\Sigma)^{-1}]\Sigma^{-1}\mu} $$
I found similar texts stating this property but without a proof. But usually I have to show that $A \Sigma$ is symettric idempotent and would already know that $Y'AY$ is non central chi square.
I am quite sure @kjetil b halvorsen's answer at What is the moment generating function of the generalized (multivariate) chi-square distribution? reduces to the expression of MGF in this post on simplification.
A direct proof is not difficult either when $\Sigma$ is assumed to be positive definite.
The proof simply relies on the fact that for a symmetric positive definite matrix $B$, we have from the multivariate normal density
$$\int_{\mathbb R^p}\exp\left[-\frac12(x-\mu)' B^{-1}(x-\mu)\right] dx=(2\pi)^{p/2}(\det B)^{1/2}$$
Or,
$$\int_{\mathbb R^p}\exp\left[-\frac12 x' B^{-1}x+\mu' B^{-1}x-\frac12 \mu' B^{-1}\mu\right] dx=(2\pi)^{p/2}(\det B)^{1/2}$$
Taking $b'=\mu' B^{-1}$, this is same as
$$\int_{\mathbb R^p}\exp\left[-\frac12x' B^{-1}x+b' x\right]dx=(2\pi)^{p/2}(\det B)^{1/2}\exp\left(\frac12 b' B b\right) \tag{*}$$
For symmetric $A$,
\begin{align} M_{X' A X}(t)&=E\left[e^{tX' AX}\right] \\&=\frac1{(2\pi)^{p/2}(\det \Sigma)^{1/2}}\int_{\mathbb R^p} \exp(tx' Ax)\cdot \exp\left[-\frac12(x-\mu)' \Sigma^{-1}(x-\mu)\right] dx \\\\&=\frac{\exp(-\frac12 \mu'\Sigma^{-1}\mu)}{(2\pi)^{p/2}(\det \Sigma)^{1/2}}\int_{\mathbb R^p} \exp\left[-\frac12 x'(I-2tA\Sigma)\Sigma^{-1}x+\mu'\Sigma^{-1}x\right] dx \end{align}
We have $(I-2tA\Sigma)\Sigma^{-1}=\Sigma^{-1}-2tA$, which is assumed positive definite (it is already symmetric) for sufficiently small $|t|$. The MGF is now precisely of the form $(*)$.
Taking $B=(\Sigma^{-1}-2tA)^{-1}=\Sigma(I-2tA\Sigma)^{-1}$ and $b'=\mu'\Sigma^{-1}$ finally gives
$$M_{X'AX}(t)=(\det(I-2tA\Sigma))^{-1/2}\exp\left\{-\frac12 \mu'[I-(I-2tA\Sigma)^{-1}]\Sigma^{-1}\mu \right\}\,,$$
whenever the MGF exists. This also generalizes @whuber's answer here.
