Sum of predicted values to the power of 10 [closed]

Question

When I take predicted values from a linear model to the power of 10, their sum is always a lot bigger than the original. Is it even allowed to sum, and does anybody have a reference for how this should be done?

E.g. lets say x is log(body.mass) and y is log(population.size). If I from the predicted values need to get the actual population size I would need sum(10^predicted.values) but this is always smaller than the original data.

Simple example in R code:

n = 10000
x = rnorm(n)
y = x + rnorm(n)
m = lm(y ~ x)
p = predict(m)

sum(10^y)/sum(10^p)

Gives results (if run a few times) from 6 to 40 times as many total individuals in the original data than the predicted.

Answer 1

Let's work with natural logarithms, instead of base 10.

You are tripped up by a common pitfall in the lognormal distribution: the expectation of the lognormal is not the exponential of the expectation $\mu$ on the log scale. You need to account for the heavy-tailedness by including the residual variance and calculate $e^{\mu+\sigma^2/2}$.

In R:

sum(exp(y))/sum(exp(p))
sum(exp(y))/sum(exp(p+summary(m)$sigma^2/2))

The last expression will come out around 1.

Answer 2

Differences in averages

You have a function f(x) such that $f(\overline{x}) \neq \overline{f(x)}$. So like the mean squared is not equal to the square of the means you also have that the mean of a power is not a power of the mean.

You estimate the model $y_i = \hat{y}_i +e_i$. Then you compare $\overline{10^{y_i}}$ with $\overline{10^{\hat{y}_i}}$. Or differently written you compare $10^{y_i} = 10^{\hat{y}_i+e_i}$ with $10^{\hat{y}_i}$.

The residual terms $e_i$ are obtained such that they average to zero for $y_i = \hat{y}_i +e_i$ but you do not get $10^{y_i} = 10^{\hat{y}_i} +e_i$ with $e_i$ average to zero. You will get that there is different scaling for negative and positive values because $10^{y+a}-10^y$ is a bigger difference than $10^{y-a}-10^y$.

So most often $\overline{10^{y_i}} > \overline{10^{\hat{y}_i}}$(even while $\overline{y_i} = \overline{\hat{y}_i}$ ), because the residuals do not 'count' the same after taking the power.

Simple example $2^1+2^{-1} = 0.5+2 = 2.5 > 2 = 2^0 + 2^0$

You will always get $\overline{10^{y_i}} > \overline{10^{\hat{y}_i}}$ when all $\hat{y}_i$ are the same. E.g. when you just model $\hat{y}_i = a$ instead of $\hat{y}_i = a + b x_i$.

Example when you do not get $\overline{10^{y_i}} > \overline{10^{\hat{y}_i}}$ is (note the extra data point with x=100):

set.seed(1)
n = 10000
x = c(rnorm(n),100)
y = x + rnorm(n+1)
m = lm(y ~ x)
p = predict(m)

sum(10^y)/sum(10^p)

giving 0.76 which is due to the data point at x=100 falling far below the line (the other 10000 points have much more weight) but contributing a lot when the power of 10 is taken (then the other 10000 points have much less weight)

What model/average to choose

The choice of the two different averages or the choice of model ($10^{\hat{y}_i} = 10^{a + b x_i + e_i}$ versus $10^{\hat{y}_i} = 10^{a + b x_i} + e_i$) will vary based on what weight you want to give to the different points (high versus low values).

See the below image for another example with the extra data points.

set.seed(1)
n = 200
x = c(rnorm(n),log(100*c(1:5)))
y = x + c(rnorm(n),rnorm(5,-1,0.1))
m = lm(y ~ x)
p = predict(m)

sum(10^y)/sum(10^p)

• One of the fit lines is according to a linear model:

  $$y_i = a x_i + b + e_i$$

• The other is according to a non linear model:

  $$(10^{y_i}) = 10^b (10^{x_i})^a +e_i $$

  or rewriting for simplicity $v_i = (10^{y_i})$, $u_i = (10^{x_i})$, and $c=10^b$

  $$v_i = c u_i^a +e_i$$

You see how the lines place different weights on different regions. In the first/left graph you see how the five points on the right have little weight on the linear model. In the second/right graph you see how the five points now have a much larger values (while the 200 points on the left are barely visible) and the residual terms get more weight.

It depends a lot on your goals which representation/model/average you want to choose as well on the original model that generates the data (how are the errors distributed).

Say you want to have a fitted curve to make predictions of $10^{Y}$ in the (entire) range $10^{X}$, then the non-linear model might be better, since the linear model put's more weight on the residuals of the smaller values.

---

What you want to do with the average of all $y_i$ or $10^{y_i}$ is unclear. To me it makes no sense because they depend on the $x_i$ which may differ from test to test (you say you are computing a population size, but what population is that if there are many $x_i$?) . The model parameters seem to be more relevant, but then again I don't know what you are doing with the average.