My professor claimed that $\lim_{p\to\infty}\chi^2_p$ has a normal distribution. The claim was made on the basis of the Central Limit Theorem: as $p\to\infty$, we have a Normal$(p\mu, p^2\sigma^2)$. I do not see how this is valid nor true, as this claim would have a limit of $p$ on the left hand side, yet $p$ also appears on the right hand side. Furthermore, $\sigma^2$ and $\mu$ both depend on $p$...
What am I missing and how do I convince myself of this limit's distribution?
This property follows from the central limit theorem, using the fact that the chi-squared distribution is obtained as the distribution of a sum of squares of independent standard normal random variables. If you have a sequence of random variables $Z_1,Z_2,Z_3, ... \sim \text{IID N}(0,1)$ then you have:$^\dagger$
$$\chi_p^2 \equiv \sum_{i=1}^p Z_i^2 \sim \text{ChiSq}(p).$$
Now, the random variables $Z_1^2,Z_2^2,Z_3^2, ... $ are IID with mean $\mathbb{E}(Z_i^2) = 1$ and variance $\mathbb{V}(Z_i^2) = 2 < \infty$, so we have $\mathbb{E}(\chi_p^2) = p$ and $\mathbb{V}(\chi_p^2) = 2p$. Applying the classical central limit theorem you get:
$$\lim_{p \rightarrow \infty} \mathbb{P} \Bigg( \frac{\chi_p^2 - p}{\sqrt{2p}} \leqslant z \Bigg) = \Phi(z).$$
Another way of writing this formal limiting result is that:
$$\frac{\chi_p^2 - p}{\sqrt{2p}} \overset{\text{Dist}}{\rightarrow} \text{N}(0, 1).$$
That is the formal convergence result that holds for the chi-squared distribution. Informally, for large $p \in \mathbb{N}$ we have the approximate distribution:
$$\chi_p^2 \overset{\text{Approx}}{\rightarrow} \text{N}(p, 2p).$$
Though not strictly correct, sometimes this informal approximation is asserted as a kind of convergence result, informally referring to convergence where $p$ appears on both sides. (Or sometimes it is made strictly correct by adding an appropriate order term.) This is presumably what your professor was referring to.
In regard to this property, it is worth noting that the gamma distribution converges to the normal as the scale parameter tends to infinity; the convergence of the chi-squared distribution to the normal is a special case of this broader convergence result.
$^\dagger$ As a sidenote, one often proceeds more generally by writing:
$$\chi_{p}^{2} = \sum_{i = 1}^{p} \bigg( \frac{X_i - \mu_{i}}{\sigma_{i}} \bigg)^2,$$
where the $X_{i} \sim \text{N}(\mu_i, \sigma_i)$ are independent Gaussian random variables with arbitrary mean and variance. By setting $Z_i \equiv (X_i - \mu_{i})/\sigma_{i}$, we can write the above formula using standardised versions of any independent normal random variables instead.
I think I'm too late to the party, but I'm writing this answer only to find a way around the confusion caused by the appearance of $p$ on the left and the right sides, namely: in the terms of the chi-square distribution $\chi^2_p$ and the normal one $\mathcal{N}(p,2p),$ as pointed out in Ben's answer above. This point was raised in the OP and also commented upon by Dilip Sarwate in the comment section to the OP. Now here, one can still be interested in the difference of the two distributions$\chi^2_p$ and $\mathcal{N}(p,2p).$ Along this line, one can indeed guarantee here that:
$$ lim_{p \to \infty}\| \Phi_{\chi^2_p} - \Phi_{\mathcal{N}(p,2p)} \|_{L^{\infty}(\mathbb{R})}=0 .....(0)$$
where $\Phi_X$ denotes the CDF of the random variable $X.$ Note the uniform convergence of the difference above to zero, which is really the key point I'm making.
The proof is a one-step application of Berry-Eseén theorem, which translates to in our case as follows (check here that the hypothesis of this theorem is really true in our case!)
$$ | \Phi_{\frac{\chi^2_p - p}{\sqrt{2p}}}(x) - \Phi_{\mathcal{N}(0,1)}(x) | \le \frac{C}{\sqrt{p}} \forall x \in \mathbb{R}.....(1) $$
where $C$ is independent of $p.$ Note that this is stronger than CLT in this case, as the bound on the right hand side is independent of $x \in \mathbb{R},$ meaning the convergence of the difference between the two CDF's is uniform,which is not necessarily the case for convergence in distribution, as it's a pointwise convergence of the CDF's at the point of continuity of the limit CDF. Note that, the uniformity w.r.t. $x$ also implies:
$$ | \Phi_{\frac{\chi^2_p - p}{\sqrt{2p}}}(\frac{x-p}{\sqrt{2p}}) - \Phi_{\mathcal{N}(0,1)}(\frac{x-p}{\sqrt{2p}}) | \le \frac{C}{\sqrt{p}} \forall x \in \mathbb{R}.....(2) $$
Next, if we use the fact that $\forall a > 0,$
$$\Phi_{aX + b}(x) = \Phi_X(\frac{x-a}{b}) $$ $$\implies \Phi_{\chi^2_p}(x) = \Phi_{\frac{\chi^2_p - p}{\sqrt{2p}}}(\frac{x-p}{\sqrt{2p}}) , .....(3)$$
and similarly:
$$\Phi_{\mathcal{N}(p,2p)} = \Phi_{\mathcal{N}(0,1)}(\frac{x-p}{\sqrt{2p}}) .....(4)$$
Now the equations (1), (3) and (4) collectively implies the equation (0). This proves that even if the d.f. $p \to \infty,$ the difference between the CDF's of $\chi^2_p$ and $\mathcal{N}(p,2p)$ indeed goes to zero. N.B. to answer your question, we only used the fact the pointwise difference between the two CDF's in (1) is independent of the point $x$ and goes to zero as $p\to \infty,$ i.e. the convergence of the CDF's is uniform; we didn't really use the fact that the rate of convergence is $O(\frac{1}{\sqrt{p}}).$
I hope this helps!
