3
$\begingroup$

The purpose of loess is to create an 'average' value of the response for any $x$, by using points in the region of $x$ to create a local regression line.

From what I understand, picking the smoothing parameter essentially chooses the proportion of closest data points to use in the local regression. The response values at these data points are then weighted by distance from $x$.

What happens if the data is not uniformly distributed in $x$? For example, in this dataset the points are less dense in the region $x \in [0, 20]$. (This is scatter.smooth in R with span = 0.25 and degree = 0, a weighted moving average).

loess smoothed scatterplot with degree 0 and span 0.25

For reference the kernel density estimate is:

enter image description here

Does this mean that individual points in the sparse region have more weight on the local regression at that point? What implications does this have for picking the smoothing parameter, to take into account regions of dense data and sparse data?

Are there any other concerns I need to worry about?

$\endgroup$
2
$\begingroup$

Putting it simply: If the data is not uniformly distributed in $x$ you get a wiggly estimate!

In effect your estimate at point $x^*$, $\mu(x= x^*)$ is still unbiased but it suffers it terms of variance. Think of it as the weighted regression estimate around point $x^*$ but lacking any close neighbours. You still estimate something but you are more uncertain about it.

The weight for an individual point in the particular region (sparsely or densely populated is irrelevant) in not proportional to the number of neighbours it has. It is only determines by the kernel and the bandwidth used. Commonly it has the form $K(\frac{x - x^*}{b})$, ie. you just care how far you are from all the other points and you normalize that by the bandwidth $b$. (Kernel functions are symmetric in the sense $K(-u)=K(u)$ so don't get bogged down by the sign).

The obvious implication when using a smoothing parameter is that if your bandwidth is too small you will base your estimate $\mu(x^*)$ in a small number of points. Would you believe a regression with 3 data points for example? In addition some kernel functions (most commonly the Gaussian kernels) have infinite support. That means that you will always be able to get some minute information from distant points even if you have a relatively small bandwidth. The Epanechnikov or the rectangular (uniform) kernel functions have finite supports; practically, information outside the bandwidth range is ignored. You might be OK with that if you have a slightly dense design but if you have irregularly sampled data you might end up some holes in your estimates.

I think a decent rule of thumbs is a $k$-NN (nearest neighbour) approach. You use the smallest bandwidth that guarantees at least $k$ points at each point of estimation $x^*$ (usually this is used to find the minimum bandwidth). You can then CV your way to better bandwidth's $b$! These address the general work behind local weighted linear regression.

For the loess algorithm in particular you will determine a neighbourhood $N(x^*)$ of $k$-NN to the estimation point $x^*$. Then based on that $N$ you will determine the largest distance $D_{x^*}$ between your point $x^*$ and its neighbours. Following that (given the kernel $K$) you will get the weights as $K(\frac{x - x^*}{D_{x^*}})$ and based on these weights you will do your local weighted linear regression. To revisit your particular question: yes, individual points at sparse regions that are close to $x^*$ will know have greater weights than if the region was more densely populated. That is because the distance $D_{x^*}$ increases driving the magnitude of the kernel input down. In the general instance of local weighted linear regression $b$ is fixed so it is agnostic of sparsity. In the case of loess, $b$ (or $D^*$ in the particular example) changes in sparse designs. That is as you have already assess instead of specifying a specific bandwidth loess' span is expressed as the percentage of the data inside the window used.

The Chapt. 8 on Non-parametric regression from the book Regression by Fahrmeir et al. has a nice worked introduction on the topic.

$\endgroup$
  • $\begingroup$ While I believe that the first half of your post is correct, I currently disagree with your assertion, maybe because you are not precise in what you mean by 'small number of points': "The obvious implication when using a smoothing parameter is that if your bandwidth is too small you will base your estimate μ(x∗)μ(x∗) in a small number of points. Would you believe a regression with 3 data points for example?" As far as I know, the loess algorithm uses the same number of points to estimate at each value of $x$. $\endgroup$ – Alex May 6 '16 at 6:18
  • $\begingroup$ @Alex: I am sorry for the misunderstanding, you are right. I focused on the general notion of local weighted linear regression regarding the choice of $b$ and not the the loess algorithm in particular. Yes, that algorithm does use a particular neighbourhood $N(x^*)$. I hope it is clear now. $\endgroup$ – usεr11852 May 6 '16 at 7:27
  • $\begingroup$ thanks, the clarification is helpful. Do you know of any rule of thumbs to determine the best span to use when the data is variable dense? I guess maybe some sort of cross validation approach could work. $\endgroup$ – Alex May 6 '16 at 7:42
  • $\begingroup$ Yes, as I mention in my answer CV is the obvious way to go. GCV is also commonly used, it is usually faster to evaluate too. See the seminal paper from Golub, Heath and Wahba, 1979 on Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter, it should set you on the right track. $\endgroup$ – usεr11852 May 6 '16 at 7:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.