1
$\begingroup$

I found by accident the nonparametric Theil-Sen Estimator as a replacement for standard OLS linear Regression. How well does it perform with autocorrelated data, non-normal residuals and heteroskedasticity?

$\endgroup$
2
$\begingroup$

It should cope with non-normal errors without difficulty and is robust to both y and x-outliers (influential observations), and so isn't badly affected by influential outliers (unlike L1 regression; see the example here).

However, it can't handle more than about 29% gross outliers (in the worst case) or a bit less in small samples.

Estimation wise it should behave reasonably in the presence of heteroskedasticity or autocorrelation, but the performance of hypothesis tests ad intervals would be affected by autocorrelation and I think by at least some kinds of heteroskedasticity (but I haven't investigated this).

$\endgroup$
  • $\begingroup$ The interesting part of the question concerns autocorrelation. Standard estimators of the uncertainty in the intercept and slope estimates will be wrong in that circumstance, perhaps grievously so. It might therefore help to articulate more clearly the sense in which the Theil-Sen estimator "should behave reasonably" in such cases. $\endgroup$ – whuber May 8 '18 at 14:02
  • 1
    $\begingroup$ I meant only that the point estimates themselves would tend to be reasonable (assuming we're not dealing with extremely large autocorrelation); the issues with p-values and intervals that I mentioned are of course affected by (/largely determined by) the impact on standard error. I'll probably come back and put in some examples, time permitting. $\endgroup$ – Glen_b -Reinstate Monica May 8 '18 at 15:37
1
$\begingroup$

I happen to have Theil's original two conference presentations on my desk. Even though I rarely use it as an estimator, it is an important limiting case in a number of problems.

So, let's begin with the simple case where all of the assumptions of ordinary least squares holds, then it has 85% relative asymptotic efficiency with ordinary least squares. This does not mean you should use Theil's estimator in lieu of OLS as a standard practice, but let us assume that when you are constructing your research and about to collect data, you come to believe that some assumption is likely to be violated. Then it is an excellent estimator, the best among those that Sen(1968) tested.

Sen, Pranab Kumar. Estimates of the Regression Coefficient Based on Kendall's Tau. Journal of the American Statistical Association. Vol 63(324). December 1968. Pp. 1379-1389

Now let us move into time series. To begin with, for the simple AR(1) equation $$x_{t+1}=\beta{x_t}+\epsilon_{t+1},|\beta|<1$$ you should use standard methods for the same reason you should use OLS above. If the assumptions are met, then a distribution-free method is simply less powerful.

If you believe, on the other hand, you have some issue that you cannot solve with a stationary series, then Theil's method of regression should be used. You should pick up the original two conference proceedings because they will help you understand what is being attempted as it is a distribution-free method for polynomial regression of any order $N.$ Autocorrelation won't be an issue either. This is because the testing process is strictly distribution-free. In a sense, the presence of autocorrelation alters the test distribution. That is not a problem here. Autocorrelation falls out the window.

Now let us consider the case of $$x_{t+1}=\beta{x}_t+\epsilon_{t+1},|\beta|>{1}.$$ This is the explosive roots case. Note that Theil's regression works for all AR(N) models. In that case, you should not consider Theil's regression as the Bayesian alternative is leaps and bounds better. This is primarily due to the prior. Since $\hat{\beta}-\beta$ is a symmetric distribution, in this case, some of the solutions from any Frequentist regression will end up in the region $|\beta|<1$, which is known not to be possible if you can constrain the model by knowing that $|\beta|\ge{1}.$ In that case, you simply shouldn't consider a non-Bayesian solution. So for models like the stock market or cancer growth you won't use this, even though it would produce an unbiased solution.

There is only one special case you would consider it, and that is when you truly have a sharp null hypothesis, such as $\beta=k$. In that case, because Bayesian methods don't do well with a sharp null hypothesis, you should use Theil's regression. In that case, where the goal is not prediction, but epistemological, then you should prefer Theil's regression to the Bayesian alternative, despite the much less accurate result.

Now let us consider the general case, where $$x_{t+1}=\beta{x_t}+\epsilon_{t+1},\forall\beta\in\mathbb{R}.$$ I have yet to run simulations on this. There is a Bayesian solution to this, and there isn't a distribution dependent parametric solution for this in Frequentist or Maximum Likelihood Methods. The non-existence of a solution is in the literature.

Nonetheless, because Theil's method does not depend upon the distributions involved, it is free of that problem. The only question is admissibility relative to the Bayesian estimator. There are a couple of questions I have not approached yet. For example, if there is a generalized Bayesian solution to allow for flat priors, then, as a guess, Theil's method should be no less efficient. This is due to the fact that order statistics are always sufficient. My reservation is that it is not admissible in the case of $|\beta|<1$ and it isn't admissible in the case of $|\beta|\ge{1}$. Still, the boundary is information in and of itself. I think the only way to solve it is through simulations and I haven't had the time to get there yet.

Because the result is a distribution-free result, homoskedasticity, heteroskedasticity and askedasticity (eg. Cauchy distribution) is not an issue. Likewise, non-normality in residuals doesn't matter. Indeed, if the assumptions of OLS are strongly violated, then Theil's method has a higher relative asymptotic efficiency. In the case of regression on distributions on variables drawn from the Cauchy distribution, the asymptotic relative efficiency is infinitely better.

Theil's method of polynomial regression should be in anyone's quiver of tools. It probably shouldn't be used much because the set of tools available is approaching mindnumbing levels, but it really is a great second best alternative and a first best in some special cases.

$\endgroup$
  • $\begingroup$ Interesting answer and great effort. I am not competent enough to judge the result fully, though. Some notes: (1) explosive roots require a strict inequality; (2) heteroskedasticity is tangential to distributional assumptions, so I doubt that a distribution-free result would make it a non-issue; (3) what do you mean by "$\beta-\hat\beta$ is a symmetric distribution"?; what is $\hat\beta$ there?; (4) you seem to imply that frequentists are not allowed to consider restricted estimation with inequality restrictions, is that right? $\endgroup$ – Richard Hardy May 9 '18 at 14:02
  • $\begingroup$ I have to admit I spent quit to a lot of times with ARIMA Models and OLS with standard correction tools such as Newey West, but I never managed to understand it or get a good valid Model, because the order of correlation was unknwon and had to analyse quickly 80 different time series. This is why I went for this Estimator and pray it will do the trick for my thesis $\endgroup$ – Neon67 May 9 '18 at 23:20
  • $\begingroup$ @Dave Harris, could you also have a look at this related question: stats.stackexchange.com/questions/346014/… $\endgroup$ – Neon67 May 13 '18 at 19:44
  • $\begingroup$ @RichardHardy thanks for the correction on 1. I am still thinking about 2. For 3 the sampling distributions of the estimators are known and symmetric. 4) I actually have constructed the restricted estimator but it has poles in it and special functions. I am going to pass it to a friend whose specialty is real analysis with special functions to see if he has a nice way out, otherwise, I also think there may be a solution to the problem via geometry, but I haven't spent a lot of time thinking about it. A similar geometry problem appears in the literature several hundred years ago. $\endgroup$ – Dave Harris May 23 '18 at 20:57
  • $\begingroup$ Thank you! Regarding (3), once again, what exactly is $\hat\beta$? $\endgroup$ – Richard Hardy May 24 '18 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.