# Weighted linear regression from log-likelihood?

I have 500 time-series and for each one, I compute the best (aka max likelihood) parameter α in a model that I'm testing, along with the corresponding log-likelihood.

Now I wish to uncover a linear relationship between my α and some other property of the time-series, say β. What are my options regarding linear regressors which would somehow take into account the log-likelihoods? In other words, how do I translate log-likelihood into some sort of weighting or error-bars in a rigorous and meaningful way?

In the example below, for instance, most high-alpha points have low log-likelihoods and the correct slope is therefore smaller than a simple linear regression would yield.

edit: For a more concrete example, let's assume my time-series are almost sinusoids with different periods $$T$$ and also have, say, a wavelength $$\lambda$$ assigned to them which should scale linearly with the period of oscillation. So my MLE routine returns $$T_{1} = 42$$ seconds with log-likelihood $$1000$$ and I know for this time series that $$\lambda_{1} = 521 \pm 5$$ nm. Some other series might return $$T_{2} = 33$$ seconds, log-likelihood $$10$$ and I know $$\lambda_{2} = 314 \pm 7$$ nm. How do I take the quality of the period estimate into account when fitting $$T$$ vs $$\lambda$$?

The log likelihood can be negative so you have to throw that idea out straight away.

You should probably use inverse variance weighting instead.

The idea is that if the $\alpha$ you estimated has a high variance, it should have a low weight. This might match your intuition since the likelihood of a linear regression model is a function of the variance.

In fact, inverse variance weighting in a linear regression model leads to the best linear unbiased estimate (BLUE) of the trend-line, if that matters at all (sometimes a little bias isn't so bad).

• Thanks that sounds good! What do you mean by "variance of $\alpha$" though - variance of the residuals of the model with parameter $\alpha$? Sorry I'm a stats noob! Sep 12, 2018 at 6:11
• @PetrJakubčík More precisely, it's the standard error of $\hat{\alpha}$. I can't tell you more. You only told us you estimated $\alpha$ with maximum likelihood. If it's from a linear model, then I think the answer to your question is yes. But any old MLE has some variance. You may have to do more digging. Sep 12, 2018 at 12:17
• Cool, I am using L-BFGS-B from Python's scipy.optimize.minimize and as it turns out, I can use the Hessian of the likelihood function at the maximum to estimate the standard error as described here: stackoverflow.com/questions/19836744/… . Thanks for your help. Sep 13, 2018 at 10:17

I don't think you should try to use log-likelihood as some kind of weighting, the reason is that you're comparing different sets of data.

If $\alpha_1$ is the max likelihood estimate (MLE) for data set 1 and $\alpha_2$ is the MLE for data set two, then these are the best for their data. The values of the likelihood may be different but they are the best parameters for their data.

If you're after error bounds on your $\alpha$ vs $\beta$ regression, bootstrap your data.

Better yet, you can set up a hierarchical model where you model the dependence between $\alpha$ and $\beta$ explicitly. Without more information it's hard to give a concrete example though.

• Hi, thanks for your ideas - I have now added a concrete example to the post. Any clue which of your proposed methods would be best in this case? Sep 11, 2018 at 18:16
• the inverse variance weighting AdamO suggested is similar in spirit to a hierarchical model and is probably simpler so I would go with that actually. Sep 11, 2018 at 21:20