6
$\begingroup$

I'd like to perform least-squares fit to data which is unevenly distributed on the x-axis.

For example, if I was to bin the data, it would be something like

x = 0~5: 10 data points

x = 5~10: 20 data points

x = 10~15: 2 data points

x = 15~20: 4 data points

I want to fit a best-fit to this to predict future values, but the model of course performs poorly for high values of x.

I can bin the data as above and then assign weights to the data based on the density (sparsity) of values in each bin. These weights can then be used to weight the contribution to the total error by each point, but I wonder: Is this the standard way of dealing with this problem, or is there another method?

Incidentally, I will be implementing this using a python library, so any advice about practical implementation is also appreciated.

Thank you for any advice!

Edit: Added a picture for illustration.

Left: Well-distributed points leading to a reasonable estimation for all x.

Right: Points distributed to lower-x values, leading to poor estimation of y at high-x values

Left: Well-distributed points leading to a reasonable estimation for all x. Right: Points distributed to lower-x values, leading to poor estimation of y at high-x values

$\endgroup$
6
  • $\begingroup$ The standard way is to use ordinary least squares with no weights at all. In what sense does this "perform poorly" for the higher values of $x$? (In fact, it will tend to fit those observed values more closely than it will fit the lower values. But that does not mean more poorly.) $\endgroup$
    – whuber
    Feb 9, 2015 at 23:43
  • $\begingroup$ Thanks for the comment! Since there are more points at low x, the total error can be minimised by making the linear trend fit well at low x at the expense of high x, which only have a couple of points and therefore contribute only a little to the total error. If the data were truly linear then unweighted fitting in this case may be fine, but my data is not really linear. So perhaps rather than worrying about weighting, a better step would be to try fitting with a non-linear function? $\endgroup$
    – Gerhard
    Feb 9, 2015 at 23:57
  • $\begingroup$ Also, I wonder if you could clarify further what you mean by "it will tend to fit those [higher x] observed values more closely than it will fit the lower [x] values. But that does not mean more poorly."? (Sorry if this is an elementary question.) $\endgroup$
    – Gerhard
    Feb 10, 2015 at 0:14
  • $\begingroup$ Is X the dependent or independent variable? $\endgroup$ Feb 10, 2015 at 2:06
  • $\begingroup$ x is independent. I've added a picture to make things clearer. $\endgroup$
    – Gerhard
    Feb 10, 2015 at 3:08

3 Answers 3

1
$\begingroup$

Assuming that some data is redundant simply because they are similar in value (on the x-axis) then your approach is correct (if ignoring issues of outliers). The technique you are looking for is called Kernel density estimation. The kernel bandwidth should be chosen based on the context of the data. If x-values within a certain distance are to be considered redundant then the bandwidth of the kernel should be wide enough to include any two values within that distance. Then you can use the inverse of the density estimation as the weights used in the least-squared regression.

Two dimensional kernel's can be used if two points can be considered non-redundant even though they have the same x-value. (e.g. a single point in the top left of your graph would have the same weight as the values in the top right.)

$\endgroup$
1
  • $\begingroup$ KDE does not perform regression--even in 2D--so it's difficult to see how this post answers the question. $\endgroup$
    – whuber
    Apr 19, 2021 at 23:29
1
$\begingroup$

For this problem, I found this article to be of great interest. For what I understood, there is usually 2 ways to deal with such non well distributed dataset :

  • either we resample the dataset, usually by deleting some data to transform the dataset into a well distributed one,
  • or we weight the data points, for example here by accounting for the density of points along x axis.

The second approach is the one they propose in the article. One advantage compared to the first method is that we do not delete data points so we can say in a way that we are more representative. They implement the method in a python package called denseweight. It is based on a kernel density estimation which allows you to get the weight for each point.

Then you can use a weighted linear regression with these weights, for example scipy.optimize.curve_fit with sigma parameter to account for the uncertainty of each data point. Setting sigma=1/weights will make isolated points more certain than dense groups of points according to their weight.

Here is an example with my dataset (x, y), either using denseweight module, or directly trying to use gaussian kernel density estimation:

from scipy.optimize import curve_fit
from denseweight import DenseWeight
from scipy.stats import gaussian_kde

f = lambda x, a, b : a * x + b

fig, ax = plt.subplots()
ax.plot(x, y, '+')

# Standart linear regression :
popt, pcov = curve_fit(f, x, y)
xfit = np.array(ax.get_xlim())
yfit = popt[0] * xfit + popt[1]
ax.plot(xfit, yfit, '-', label='standard linear fit')

# Weighted linear regression with denseweight module' :
dw = DenseWeight(alpha=1)
weights = dw.fit(x)
popt, pcov = curve_fit(f, x, y, sigma=1/weights)
yfit = popt[0] * xfit + popt[1]
ax.plot(xfit, yfit, '-', label='weighted linear fit with denseweight module')

# Weighted linear regression with gaussian_kde :
kde = gaussian_kde(x)
popt, pcov = curve_fit(f, x, y, sigma=kde.pdf(x))
yfit = popt[0] * xfit + popt[1]
ax.plot(xfit, yfit, '-k', label='weighted linear fit with gaussian_kde')

ax.legend()

enter image description here

$\endgroup$
10
  • $\begingroup$ Welcome to CV and thank you for the effort you put into posting a clear answer. However, I find it difficult to compare two fits that are so obviously bad. What would be the point of using any of the lines shown here to describe these data? There is a real disconnect between performing any kind of regression and reweighting the explanatory variable, because they have completely different aims and regression is always conditional on the explanatory variable, which implies it doesn't need or use any weights or make distributional assumptions about that variable. $\endgroup$
    – whuber
    Mar 14, 2023 at 16:06
  • $\begingroup$ Thank you @whuber for your comment. My dataset represents a 2D space trajectory of a particle going from left to right in the above figure. The particle has an increasing velocity, but the points are recorded with a fix time step. And for the last time steps (right side of the figure) the particle is too poorly detected. This is why we have more points in the left side of the graph. I know from other observations that the real trajectory is quite linear. What I want is to recover the global slope of this linear trajectory which in fact is closest to the black/green curve than the red curve. $\endgroup$
    – battsong
    Mar 14, 2023 at 19:36
  • $\begingroup$ You seem to be saying you know the data are not what they are! It is difficult to justify characterizing these (many) data points as "quite linear." $\endgroup$
    – whuber
    Mar 14, 2023 at 20:02
  • $\begingroup$ I know the global trajectory shape just by watching the recording images of the particle, that is why I can say that at the scale of the entire path the particle has taken, the trajectory is "quite linear". $\endgroup$
    – battsong
    Mar 14, 2023 at 20:33
  • $\begingroup$ It is common to have some datasets which have non well distributed data which makes simple regression to not be representative of the reality. I gave an example with my particle, but we could think of other cases, for example if you take a car accelerating in a roundabout, and you record the scene with a drone from the top with a fix frame rate, this will give you way more data points at the beginning of the trajectory where the car is slow enough than at the end of the trajectory where the car goes too fast. This leads to wrong estimation of the entire trajectory. How should we deal with it ? $\endgroup$
    – battsong
    Mar 14, 2023 at 20:57
0
$\begingroup$

You can use cubic splines. One advantage of splines over standard polynomial regression is that data points influence is more local. This is due to irregularity at knots.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.