Questions tagged [least-absolute-deviations]

A regression that minimizes the sum of absolute errors (instead of the sum of squared errors).

Filter by
Sorted by
Tagged with
3 votes
1 answer
43 views

How can I fit a least absolute deviation model with random intercepts?

I have some data and a model formulation that I believe describes the data generating process. The model looks like this $$y_{i,t} = \beta_1x_{i,t} + \beta_2x_{i,t-1}+\alpha_t+\epsilon_{i,t}$$ where $\...
Chechy Levas's user avatar
  • 1,255
0 votes
0 answers
26 views

Gauss-Markov-Theorem not holding true when error terms not normally distributed? [duplicate]

I tried to simulate some stuff around the Gauss-Markov-Theorem when I stumbled over something I cannot explain. I tried to simulate the density of an OLS estimator and a LAD (least absolute deviations)...
Sebastian Geis's user avatar
0 votes
0 answers
81 views

Is Centering Data Around Their Medians in Least Absolute Deviation Regression Model (No Intercept), a Good Robust Practice For Smaller Data Sets?

Per the regression model: $\mathbf{y} = f(\mathbf{x},\mathbf{\beta}) + \mathbf{\epsilon}$ Where the $\beta$ estimate of LAD regression is given by: $ \hat{\beta}_{LAD} = \text{argmin}_{ b} \sum_{i=1}^...
AJKOER's user avatar
  • 2,208
2 votes
0 answers
62 views

Asymptotic for LAD - Problem 16, p.84 Van der Vaart

From Van der Vaart's Asymptotics Statistics, we have the derivation of the asymptotics for the least square regression (Example 5.27). Now, the problem 16 of the same section regards the asymptotics ...
Eryna's user avatar
  • 309
3 votes
2 answers
174 views

Quantile regression necessarily has a solution with $r$ residuals equal to 0: why/how

Given data $\{(Y_i,X_{1,i},X_{2,i}\dots X_{p,i}):1\le i\le n\}$. let $\theta\in(0,1)$ and $\beta=(\beta_1,\beta_2\dots\beta_p)^T$. Then, the quantile regression problem $$\underset{\alpha,\beta}{\min}\...
reyna's user avatar
  • 365
0 votes
0 answers
135 views

Orthogonal regression that minimizes absolute error instead of squared error

Fitting a line through 3D points is usually done by orthogonal least squares (aka "total least squares"), i.e., by minimizing the mean squared orthogonal distance between the points and the ...
cdalitz's user avatar
  • 5,132
0 votes
1 answer
41 views

Interpretation of LAD under model misspecification

Suppose we have $n$ i.i.d. draws of $y_i$ and $x_i$ and consider a linear equation: $$y_i =\beta_0 +\beta_1x_{1i}+...+\beta_kx_{ki}+u_i =x_i'\beta+u_i $$ A common assumption in OLS is $E[y|x]=x_i'\...
Michael Gmeiner's user avatar
12 votes
2 answers
763 views

Residual using absolute loss linear regression

For ordinary least square linear regression, we have sum of residuals as zero, what about the sum of residuals for linear regression calculated using absolute loss?
william007's user avatar
  • 1,087
2 votes
1 answer
210 views

Question about Huber loss when k=0 in Casella and Berger

In Casella and Berger (page484), the following Huber loss is defined. Then on the next page, Table 10.2.1 shows the Huber estimator for different $k$: In particular, $k=0$ gives the median, which ...
ExcitedSnail's user avatar
  • 2,778
0 votes
0 answers
349 views

Regression coefficients while minimizing absolute error

I understand that we like working with square error instead of absolute error because it makes the calculus easy. But I was wondering about the parameters of Linear Regression minimized for absolute ...
Vishal Balaji's user avatar
0 votes
0 answers
175 views

well maintained lad-regression solution

https://www.real-statistics.com/multiple-regression/lad-regression/ I am trying to understand what is the best package (in R or python) for lad-regression. But the following one says "but least ...
user1424739's user avatar
2 votes
1 answer
373 views

Minimizing Huber Loss

Huber Loss is given as follows: I'd like to proof $\gamma=median\{y_1,...y_N \}$ minimizes the Huber Loss so i've taken its first derivate for $\gamma\neq y_i$: I've tried to proof that first ...
Davi Américo's user avatar
0 votes
0 answers
107 views

Least Absolute Values Regression

I have the data: x <- c(0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0) y <- c(2.06, 2.12, 2.32, 2.02, 2.76, 3.04, 2.83, 3.15, 3.36, 3.68, 3.96) I ...
Epsilon Delta's user avatar
7 votes
1 answer
1k views

Why is there no improvement when training Xgboost with pseudo-Huber loss?

In this StackOverflow post I asked if there was something wrong with my syntax when training an XGboost model (in R) with the native pseudo-Huber loss ...
PaulG's user avatar
  • 1,267
0 votes
0 answers
118 views

Bayesian least absolute deviations

Ordinary Least Squares have a bayesian equivalent. For instance, fitting the linear model $$ y \sim a + bx + \epsilon $$ using OLS is equivalent to defining the following Bayesian linear model: $$ y \...
David Masip's user avatar
1 vote
1 answer
228 views

Does this exist: Recursive Least Absolute Deviation-Regression?

For least squares regression there exists the Recursive Least Squares algorithm which allows to find the least squares solution online. Does something similar exist for Least Absolute Deviation ...
phinz's user avatar
  • 131
40 votes
7 answers
16k views

Why is using squared error the standard when absolute error is more relevant to most problems? [duplicate]

I recognize that parts of this topic have been discussed on this forum. Some examples: Is minimizing squared error equivalent to minimizing absolute error? Why squared error is more popular than the ...
Ryan Volpi's user avatar
  • 1,898
1 vote
0 answers
35 views

Generalisation of Absolute Deviation and Least Squares Deviation: Statistical Use?

This linked answer defines Least Absolute Deviation (LAD) and Least Squares Deviation (LSD), as follows: $$ LAD = \Sigma^n_{i=1} |y_i-f(x_i)| $$ And: $$ LSD = \Sigma^n_{i=1} (y_i-f(x_i))^2 $$ ...
Colm Bhandal's user avatar
4 votes
1 answer
234 views

Reference for doing linear regression with mean absolute deviation?

I am looking for a resource that goes over how to derive the coefficients for a linear regression model while minimizing the mean absolute deviation. I am hoping for both a mathematical and ...
confused's user avatar
  • 3,253
2 votes
0 answers
97 views

Efficiency of OLS versus Quantile regression estimator

If I have a linear model $ y_i = x_i'\boldsymbol\beta + \epsilon_i $ and I assume that OLS estimator of $\boldsymbol\beta$ is unbiased and consistent and Least absolute deviation (LAD) estimator of $...
doremi's user avatar
  • 147
1 vote
0 answers
555 views

Consistency of least absolute deviation estimator (LAD) vs OLS

If we know that OLS estimator of beta in linear regression model is unbiased and consistent and we don't have any further assumptions (on errors or anything), will LAD estimator of beta be also ...
doremi's user avatar
  • 147
2 votes
0 answers
57 views

Comparing the robustness least absolute deviation with OLS

My professor told us that the OLS estimator can be influenced by outliers because $$\hat{\beta}_{OLS}=\text{argmin}\left\lVert y - X^T \beta\right\rVert_2^2 $$ implies that the first order ...
Stan Shunpike's user avatar
2 votes
1 answer
112 views

Convergence of weighted regression solution to the solution of L1 regression

I have read in this paper that the weighted regression solution converges to the solution of L1 regression, for weights, $$W_i=1/|y_i-\hat y_i|$$ I worked this out but unfortunately, I lost. Could ...
jeza's user avatar
  • 2,089
1 vote
0 answers
47 views

least absolute deviation version of deming [closed]

Based on what I know, deming minimizes the sum of square of perpendicular distance to the regression line. Is there a package in R that can run regression that minimize the sum of absolute value of ...
CABLE's user avatar
  • 111
1 vote
0 answers
17 views

Why would the sequence $\hat{x}_{n+1} = \left( A^T E(\hat{x}_n) A\right)^{-1} A^T E(\hat{x}_n)b$ from Least absolut deviation regression converge

Consider the problem $$ \min_x f(x) = \min_x \|b - Ax\|_{1} = \min_x \sum_i \left| b_i - \sum_j a_{ij}x_{j} \right| $$ where $x,b$ are vectors and $A$ matrix of some dimension. Then $$ \frac{\...
Olba12's user avatar
  • 123
3 votes
4 answers
3k views

How regression trees split, when all the Features and target have only continuous values

Can anyone please explain how splitting is performed in regression trees when we only have continuous features. I have referred to different papers, but all I could find is formulas or theorems. Can ...
Sim's user avatar
  • 31
4 votes
0 answers
195 views

Most efficient LAD solver

What is the most efficient way to solve linear Least absolute deviation regression problem? I know it can be solved using linear programming, is there a better/faster method? Edit: I'm interested ...
Meni's user avatar
  • 183
5 votes
3 answers
4k views

Is there any library for least absolute deviation (LAD) regression with regularization terms?

We know that LASSO and ridge and ElasticNet all apply regularization terms on the coefficients of least squares regression. However, I have not yet found any R / python libraries that compute ...
felixc's user avatar
  • 51
7 votes
0 answers
254 views

Is there any geometric intuition on least absolute deviation regression?

There are a lot of geometric intuitions for regression with least square, e.g., projection, orthogonal, etc. (This and this answers are good examples.) Is there similar geometric intuition for least ...
Haitao Du's user avatar
  • 36.9k
4 votes
0 answers
573 views

Interpretation of the least absolute deviations linear regression coefficient

In linear regression of $y$ onto $x$, one finds a $\beta_0$ and $\beta_1$ minimizing $\sum \|y - (\beta_1 x + \beta_0)\|^2$. One can show that $$\beta_1 = \rho(x,y) \frac{\sigma(y)}{\sigma(x)},$$ ...
Apprentice's user avatar
1 vote
1 answer
3k views

Can I use gradient descent for Least Absolute Deviation Regression?

I've read in some posts that the absolute function is not differentiable. However, it is not differentiable only at 0. Can I not check the value at each training point to calculate the derivative. ...
racerX's user avatar
  • 111
3 votes
2 answers
4k views

Model that optimizes mean absolute error always gives same prediction

My gradient boosting regression model (GBM) is trained to minimize mean absolute error (MAE) but gives the same prediction for every record on my highly skewed dataset. I believe there is a quick fix ...
Ryan Zotti's user avatar
  • 6,647
1 vote
0 answers
34 views

Fisher Information in LAD model for ML estimator

Considering an i.i.d. sample from a linear model $y_i=\alpha x_i+u_i$ (both $y$ and $x$ are centered with respect to their means) errors are homoscedastic and are distributed as: $$u\sim\frac{1}{\sqrt{...
GabMac's user avatar
  • 267
1 vote
1 answer
478 views

For syntax of least absolute deviation from package 'L1pack' [closed]

I am trying to use the least absolute deviation regression from my dataset which has one column of dependent variable and multiple columns of independent variables. I tried using the following syntax ...
Eric's user avatar
  • 506
1 vote
0 answers
1k views

Is it possible to force least absolute deviations (LAD) regression to return the 'median' value when infinite solutions are possible?

I have a problem where LAD regressions is not giving me a solution as the R package (L1pack) errors whenever there is an infinite number of possible solutions*. ...
Alex's user avatar
  • 4,392
7 votes
1 answer
2k views

What is the maximum likelihood/GLM version of least absolute deviations for robust linear regression?

Robust linear regression from minimising the absolute deviationresults in a regression line of medians conditional on covariates, instead of means using the standard least squares methodology: Is ...
Alex's user avatar
  • 4,392
13 votes
2 answers
6k views

How to solve least absolute deviation by simplex method?

Here is the least absolute deviation problem under concerned: $ \underset{\textbf{w}}{\arg\min} L(w)=\sum_{i=1}^{n}|y_{i}-\textbf{w}^T\textbf{x}|$. I know it can be rearranged as LP problem in ...
southdoor's user avatar
  • 231
1 vote
0 answers
312 views

Why is Coordinate Descent not used to solve Least Absolute Deviation?

I have recently been looking into why Least Absolute Deviation (LAD) is not used in place of OLS for machine learning, and it appears the primary reason is due to difficulty in computing a solution ...
Chris Wood's user avatar
3 votes
1 answer
2k views

$R^2$ for least absolute deviation regression

I know that $R^2$ is for the least square regression. Is there an analogous measure of fit to $R^2$ in LAD (Least Absolute Deviations) regression? Here I am concerned with the "fitting quality".
CCKKLLGG's user avatar
1 vote
0 answers
120 views

What is the standard measure of fit quality for least absolute deviation regression (the analog of $R^2$) [duplicate]

When one runs an OLS regression, one often prefers to look at $R^2$ rather than the mean squared error, though both measure goodness of fit. $R^2$ is dimensionless and has some advantageous properties:...
kuzzooroo's user avatar
  • 121
8 votes
1 answer
418 views

When does Least Square Regression (LSQ) line equal to Least Absolute Deviation (LAD) line?

I have the following question at hand. Suppose $(x_1,y_1),(x_2,y_2),\cdots,(x_{10},y_{10})$ represents a set of bi-variate observations on $(X,Y)$ such that $x_2=x_3=\cdots =x_{10}\ne x_1.$ Under ...
Qwerty's user avatar
  • 1,267