6
$\begingroup$

When I take predicted values from a linear model to the power of 10, their sum is always a lot bigger than the original. Is it even allowed to sum, and does anybody have a reference for how this should be done?

E.g. lets say x is log(body.mass) and y is log(population.size). If I from the predicted values need to get the actual population size I would need sum(10^predicted.values) but this is always smaller than the original data.

Simple example in R code:

n = 10000
x = rnorm(n)
y = x + rnorm(n)
m = lm(y ~ x)
p = predict(m)

sum(10^y)/sum(10^p)

Gives results (if run a few times) from 6 to 40 times as many total individuals in the original data than the predicted.

$\endgroup$
  • $\begingroup$ Could you tell us more on why exactly are you doing, what you're doing? $\endgroup$ – Tim Mar 26 '18 at 16:05
  • $\begingroup$ Yes. I am estimating the total population size from body mass, per species, for species with unknown population sizes but known body masses. And we are using the log(pop.size)~log(body.mass) as regression since that is a pretty strong relationship and body mass is known. $\endgroup$ – Rasmus Ø. Pedersen Mar 26 '18 at 16:12
  • 1
    $\begingroup$ I think you may be missing two steps between lines "y=..." and "m = ...". You should put in something like "x2 = log10(x)" and "y2 = log10(y)". Then your line "m" should be "m = lm(y2 ~ x2)". $\endgroup$ – EngrStudent - Reinstate Monica Mar 26 '18 at 18:43
  • $\begingroup$ You mean that the sum or predicted values is smaller, right? sum(10^p)<sum(10^y) $\endgroup$ – Sextus Empiricus Mar 26 '18 at 23:28
7
$\begingroup$

Let's work with natural logarithms, instead of base 10.

You are tripped up by a common pitfall in the lognormal distribution: the expectation of the lognormal is not the exponential of the expectation $\mu$ on the log scale. You need to account for the heavy-tailedness by including the residual variance and calculate $e^{\mu+\sigma^2/2}$.

In R:

sum(exp(y))/sum(exp(p))
sum(exp(y))/sum(exp(p+summary(m)$sigma^2/2))

The last expression will come out around 1.

$\endgroup$
  • $\begingroup$ Thank you. Does this mean that individual point estimates should also be corrected for the residual variance, if you are looking at a specific species? Further: the axes of the plot makes more intuitive sense by using base 10 for the log, is there a way to correct for residual variance here as well? $\endgroup$ – Rasmus Ø. Pedersen Mar 26 '18 at 16:32
  • $\begingroup$ Yes, if you wish to predict the expectation, then you need to correct with the variance. (Note that if you are working with quantiles, you don't need the variance - the exponential of the median on the log scale will come out correctly as the median on the original scale.) If you are plotting with log axes, then the data to be plotted are on the log scale, so no correction necessary. This earlier thread may be helpful. $\endgroup$ – Stephan Kolassa Mar 26 '18 at 16:35
  • $\begingroup$ I meant that i prefer to use the $log_{10}$ values in the model instead of $log_e$, is there a way to correct the prediction then? $\endgroup$ – Rasmus Ø. Pedersen Mar 26 '18 at 17:00
  • $\begingroup$ There may be. One would need to do the integration. This thread would be helpful. I don't really have the time at the moment, perhaps someone would like to chime in? Or you could post a follow-up question on calculating the expectation for a base 10 lognormal. Then again, since the difference is only a multiplicative factor, I personally would not go to the trouble (though I understand that some fields like the interpretability of base 10). $\endgroup$ – Stephan Kolassa Mar 26 '18 at 18:21
3
$\begingroup$

Differences in averages

You have a function f(x) such that $f(\overline{x}) \neq \overline{f(x)}$. So like the mean squared is not equal to the square of the means you also have that the mean of a power is not a power of the mean.

You estimate the model $y_i = \hat{y}_i +e_i$. Then you compare $\overline{10^{y_i}}$ with $\overline{10^{\hat{y}_i}}$. Or differently written you compare $10^{y_i} = 10^{\hat{y}_i+e_i}$ with $10^{\hat{y}_i}$.

The residual terms $e_i$ are obtained such that they average to zero for $y_i = \hat{y}_i +e_i$ but you do not get $10^{y_i} = 10^{\hat{y}_i} +e_i$ with $e_i$ average to zero. You will get that there is different scaling for negative and positive values because $10^{y+a}-10^y$ is a bigger difference than $10^{y-a}-10^y$.

So most often $\overline{10^{y_i}} > \overline{10^{\hat{y}_i}}$(even while $\overline{y_i} = \overline{\hat{y}_i}$ ), because the residuals do not 'count' the same after taking the power.

Simple example $2^1+2^{-1} = 0.5+2 = 2.5 > 2 = 2^0 + 2^0$

You will always get $\overline{10^{y_i}} > \overline{10^{\hat{y}_i}}$ when all $\hat{y}_i$ are the same. E.g. when you just model $\hat{y}_i = a$ instead of $\hat{y}_i = a + b x_i$.

Example when you do not get $\overline{10^{y_i}} > \overline{10^{\hat{y}_i}}$ is (note the extra data point with x=100):

set.seed(1)
n = 10000
x = c(rnorm(n),100)
y = x + rnorm(n+1)
m = lm(y ~ x)
p = predict(m)

sum(10^y)/sum(10^p)

giving 0.76 which is due to the data point at x=100 falling far below the line (the other 10000 points have much more weight) but contributing a lot when the power of 10 is taken (then the other 10000 points have much less weight)

What model/average to choose

The choice of the two different averages or the choice of model ($10^{\hat{y}_i} = 10^{a + b x_i + e_i}$ versus $10^{\hat{y}_i} = 10^{a + b x_i} + e_i$) will vary based on what weight you want to give to the different points (high versus low values).

See the below image for another example with the extra data points.

set.seed(1)
n = 200
x = c(rnorm(n),log(100*c(1:5)))
y = x + c(rnorm(n),rnorm(5,-1,0.1))
m = lm(y ~ x)
p = predict(m)

sum(10^y)/sum(10^p)

two different fits

  • One of the fit lines is according to a linear model:

    $$y_i = a x_i + b + e_i$$

  • The other is according to a non linear model:

    $$(10^{y_i}) = 10^b (10^{x_i})^a +e_i $$

    or rewriting for simplicity $v_i = (10^{y_i})$, $u_i = (10^{x_i})$, and $c=10^b$

    $$v_i = c u_i^a +e_i$$

You see how the lines place different weights on different regions. In the first/left graph you see how the five points on the right have little weight on the linear model. In the second/right graph you see how the five points now have a much larger values (while the 200 points on the left are barely visible) and the residual terms get more weight.

It depends a lot on your goals which representation/model/average you want to choose as well on the original model that generates the data (how are the errors distributed).

Say you want to have a fitted curve to make predictions of $10^{Y}$ in the (entire) range $10^{X}$, then the non-linear model might be better, since the linear model put's more weight on the residuals of the smaller values.


What you want to do with the average of all $y_i$ or $10^{y_i}$ is unclear. To me it makes no sense because they depend on the $x_i$ which may differ from test to test (you say you are computing a population size, but what population is that if there are many $x_i$?) . The model parameters seem to be more relevant, but then again I don't know what you are doing with the average.

$\endgroup$
  • $\begingroup$ I don't quite understand your answer. If they are only linearly related on a log-log scale how would you then start out by modelling $10^y \sim 10^x$ ? $\endgroup$ – Rasmus Ø. Pedersen Mar 26 '18 at 16:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.