Thanks for the help from @whuber. Here is the answer to my own question.
The original statement is not true. It should be rephrased to this,
$$
For\ |\mu2-\mu1| < 2\sigma\\
p''(x) < 0, if \begin{cases}\sigma < x < (\mu2 - \mu1) + \sigma\\-\sigma < x < (\mu2 - \mu1) - \sigma\end{cases}
$$
Proof:
After some calculation, the second derivative is
$$
p''(x) = \frac{-1}{2\sqrt{2\pi}\sigma^3}[(1 - \frac{(x - \mu1)^2}{\sigma^2}) \exp^{\frac{-(x - \mu1)^2}{2\sigma^2}} + (1 - \frac{(x - \mu2)^2}{\sigma^2}) \exp^{\frac{-(x - \mu2)^2}{2\sigma^2}}]
$$
Set $y = \frac{1}{\sigma}(x - \frac{\mu1 + \mu2}{2})$
$$
p''(y) = \frac{-1}{2\sqrt{2\pi}\sigma^3}[(1 - (y + (\frac{\mu2 - \mu1}{2\sigma}))^2) \exp^{\frac{-1}{2}(y + \frac{\mu2 - \mu1}{2\sigma})^2} + (1 - (y - (\frac{\mu2 - \mu1}{2\sigma}))^2) \exp^{\frac{-1}{2}(y - (\frac{\mu2 - \mu1}{2\sigma}))^2}]
$$
By removing the constant, then the objective function is
$$
((y + (\frac{\mu2 - \mu1}{2\sigma}))^2 - 1) \exp^{-y(\frac{\mu2 - \mu1}{\sigma})} + ((y - (\frac{\mu2 - \mu1}{2\sigma}))^2 - 1) < 0
$$
We can organize this function into
$$
\exp^{-y(\frac{\mu2 - \mu1}{\sigma})} < \frac{(1 - (y - (\frac{\mu2 - \mu1}{2\sigma}))^2)}{((y + (\frac{\mu2 - \mu1}{2\sigma}))^2 - 1)}
$$
Because the exponential function must be positive, we should try to prove that
$$
(1 - (y - (\frac{\mu2 - \mu1}{2\sigma}))^2)((y + (\frac{\mu2 - \mu1}{2\sigma}))^2 - 1) > 0
$$
Because the 4 roots in this function are
$$
y = \frac{\mu2 - \mu1}{2\sigma}\pm1, -\frac{\mu2 - \mu1}{2\sigma}\pm1
$$
Finally, we can find the interval of y to make this function larger than 0
$$
\begin{alignedat}{2}
-\frac{\mu2 - \mu1}{2\sigma} - 1 < y < \frac{\mu2 - \mu1}{2\sigma} - 1\\
1 - \frac{\mu2 - \mu1}{2\sigma} < y < \frac{\mu2 - \mu1}{2\sigma} + 1
\end{alignedat}
$$
After substituing y with $\frac{1}{\sigma}(x - \frac{\mu1 + \mu2}{2})$, we can get
$$
\begin{alignedat}{2}
\mu1 - \sigma < x < \mu2 - \sigma\\
\sigma - \mu1 < x < \mu2 + \sigma\
\end{alignedat}
$$