# Clustering in data

I have a set of data points $(x,y)$ where $y = f(x)$. My goal is to fit the function $f$ using ols. The choice of function $f$ is quadratic due to domain know-how.The independent variable $x$ exhibits clustering, i.e. a lot of observations in a particular region and relatively sparse in others (shown is a plot of $y$ versus $x$).

I am looking to address this clustering because I think it is biasing the results. One way to solve this problem is to try robust regression, as I think y it is a weighting problem: in other words, the clustered data points are weighted more than they should be. I am curious if there are best practices on how to address this problem and what should I be aware of.

The distribution of $X$ is not an assumption in OLS regression. Thus the 'clustering' in $X$ is not necessarily a problem.

In addition, data tend to have less leverage* over the fitted regression line the closer they are to the mean of $X$. I note that the high density cluster is roughly where the mean would have been if $X$ had been uniformly distributed. Therefore, if you are worried that the cluster will have too much influence, I suspect those worries are misplaced and no weighting need be done.

On the other hand, if you are worried the cluster won't have enough influence (I don't interpret this as your concern), then you could use WLS (weighted least squares). You would develop a weighting scheme from some prior theoretical understanding that would make the clustered points more influential than the more sparse points to the sides.

For example, if you repeatedly sample the same x-value ($x_j$), you should get an ever better approximation of the vertical position of $f(x_j)$. As a general rule of study design, you should try to oversample locations in $X$ where you are most interested in knowing $f(X)$. However, note that if you oversample in an area you don't care about and undersample where you are primarily interested, and your functional form is misspecified (e.g., you include only a linear term, when a quadratic is required), then you could end up with biased estimates of $f(X)$ in your locations of interest.

If you are worried that the nature of the function $f(X)$ might be different inside the cluster than outside it, @user777's suggestion to use splines will take care of that.

* Also see my answer here: Interpreting plot.lm().

• Gung comes to the rescue! This is the answer I was looking for, many thanks.I mean what threw me off, is imagine that I am repeatedly sampling same x for the same y, and add that to my sample data, is it not tantamount to increasing the weight on that particular x, something similar is happening here? I guess what you mean is that the weight may he high, but the error term would be low as we are close to the mean?
– gbh.
Commented Nov 30, 2015 at 17:49
• You're welcome, @gbh. (See edit for response to comment.) Commented Nov 30, 2015 at 17:58
• Thanks gung. As a follow up, if I sample an x over and over again, that is same as weighing that observation more. Whether or not the sampling changes the regression depends on the leverage of x right. If leverage of x = 0, I can sample it infinitely more often and it wont change anything righjt?
– gbh.
Commented Nov 30, 2015 at 19:26
• Not exactly. If you sample the mean of X a million times, and other x-values 5 times each, that won't have any effect on the slope but will affect the intercept; w/ a quadratic term things will get more complicated. Commented Nov 30, 2015 at 19:42
• OK. How do you prove this, "For example, if you repeatedly sample the same x-value (xj), you should get an ever better approximation of the vertical position of f(xj)"?
– gbh.
Commented Nov 30, 2015 at 20:54

Or is the shape more complicated? Restricted cubic splines fit flexible functions to different sections of the data; cubic spline design matrices can be estimated with OLS. MARS accomplishes a similar effect (different models for different subsets of points), but discovers the subset locations on its own.

Splines build up from this idea: a data set can be partitioned into different intervals, and each of those intervals can be modeled with some degree of accuracy as a constant. Alternatively, you can model each interval as a linear function. Still more elaborately, you can make the linear functions continuous at the interval boundaries. These boundaries are "knots." This solution is a piece-wise linear model. Restricted cubic splines further enforce that the function have several orders of differentiability at the knots, and that the function be linear in the intervals $(-\infty, k_1],[k_n,\infty)$, i.e. "outside" of the last knots. This linearity requirement is motivated by a desire to not over-extrapolate in very data-sparse regions. Restricted cubic splines allow for non-monotonic functions, e.g. the function can increase and then decrease and then increase again.

How to select knot locations is a sticky wicket, since it will influence what the ultimate model looks like. Frank Harrell's book Regression Modeling Strategies has some recommendations based on quantiles.

• I am fitting a quadratic function and yes, they do have that physical and visual relationship.
– gbh.
Commented Nov 30, 2015 at 16:43
• How do splines get over the problem of clustering in general?
– gbh.
Commented Nov 30, 2015 at 16:45