Your intuition is right - if you use the same variables and fit the same model. The results has to be identical.
Given that the results are different something must have changed.
In fact, you are getting two different results because of how you specify the model.
mod1 = lm(log(Girth)~Volume^2, trees)
trees$nY = log(trees$Girth)
trees$nX = trees$Volume^2
mod2 = lm(nY~nX, trees)
If we dig a little deeper and compare the data used to fit both models, we will see that they are in fact different
1 2.116256 10.3
2 2.151762 10.3
3 2.174752 10.2
4 2.351375 16.4
5 2.370244 18.8
6 2.379546 19.7
1 2.116256 106.09
2 2.151762 106.09
3 2.174752 104.04
4 2.351375 268.96
5 2.370244 353.44
6 2.379546 388.09
mod1 we see that Volume has not been squared!, hence this is the reason why you are seeing different RSEs. The
^ actually has a special meaning when specifying a model formula, you can read more about it here.
If you are trying to fit Volume^2, you can use the following command: