Preamble
For any nonnegative random variable,
$$\begin{align}
E[X] &= \int_0^\infty xf(x)\,\mathrm dx \tag{1}\\
&=\int_0^\infty [1-F(t)]\,\mathrm dt\tag{2}\\
&= \int_0^\infty P\{X > t\}\,\mathrm dt\tag{3}
\end{align}$$
where the world will end if those square brackets in the integral on the right
side of $(2)$ are ever opened and the integral is written as the difference of the integrals of $1$ and $F(t)$ over $(0,\infty)$.
These integrals calculate the area between the line at height $1$ and the curve
$F(t)$ (this is what $(2)$ is telling us) in two different ways.
$(2)$ can be
interpreted as the
standard Riemann integral (measure theorists: please hold your fire,
I am not talking to you) which is finding the area by breaking it up into
vertical narrow strips of height $[1-F(t)]$ and width $\Delta t$, and then making
the strips become narrower and narrower, talking of upper and lower Riemann
sums, taking limits, etc.
But, another way of finding the area is to divide it into very narrow horizontal
strips with the strip of length $x$ whose lower side
extends all the way from $0$ on the
vertical axis to the point $(x, F(x))$ on the curve. The upper side of this
horizontal strip is from the horizontal axis to the point $(x+\Delta x), F(x+\Delta x))$, that is, the width of this strip is
$F(x+\Delta x) - F(x) \approx f(x)\Delta x$. Proceeding as before (making
the strips narrower etc,) we see
that the desired area can also be expressed as $(1)$.
End of preamble
From $(3)$, we can write
$$\begin{align}
E[X^2] &= \int_0^\infty P\{X^2 > t\}\,\mathrm dt\\
&= \int_0^\infty P\{X > \sqrt{t}\}\,\mathrm dt\\
&= \int_0^\infty 2yP\{X > y\}\,\mathrm dy&\scriptstyle{\text{substitute}~
y^2 = t, ~~2y\mathrm dy = \mathrm dt}\\
&= 2\int_0^\infty tG(t)\,\mathrm dt
\end{align}$$
So, it is not quite as simple as proving that $x^{2}f(x)dx = tG(t)dt$
as you were trying to do (presumably by some kind of change of variable);
we have to think about the matter a little
differently.