15

TLDR; when nothing is known about actual cost of error to the user of the model, MSE is a better default option compared to MAE because, in my opinion, it is easier to manipulate analytically and is more likely to match the actual cost of error. It's a great question. I like that you start with desire to make your loss function match actual costs. This is ...


10

I think the reason is more sociological that statistical. Short version: We do it this way because we always have. Longer version: Historically, we could not do many of the things we now take for granted. Many things are computer intensive and Ronald Fisher was born before Alan Turing. So, people did OLS regression - a lot. And people read those ...


9

Some hints to help you gain some insight Make up or generate some data consistent with the conditions in the question. Try $x_1=0, y_1=0$ and $x_2..x_{10}=1$ (choosing some values for $y_i$, $i=2,...,10$). Where do the lines pass relative to the first point? Now start as above but try placing $y_i$, $i=2,...,10$ at say 1,2,3,4,5,6,7,8,9 respectively. Where ...


6

The first 5 answers fail to distinguish between estimation loss and prediction loss, something that is crucial in answering the question. A priori, there is no reason that the two should coincide. I will discuss both types of loss in the context of point prediction using linear regression. The discussion can be extended to models other than linear regression ...


5

You want an example for solving least absolute deviation by linear programming. I will show you an simple implementation in R. Quantile regression is a generalization of least absolute deviation, which is the case of the quantile 0.5, so I will show a solution for quantile regression. Then you can check the results with the R quantreg package: rq_LP <- ...


5

This difficulty can be addressed using subgradients. A subgradient of a convex function $f$ at some point $x$ is any value $v$ that satisfies $f(y)-f(x)\ge v\cdot (y-x)$ for all $y$ in the domain. Note that if $f$ is differentiable at $x$ then the subgradient is just the gradient. The subgradient method is very similar to gradient descent in concept; update ...


4

minimize the sum of squared errors $\sum_i \epsilon_i(\theta)^2$ with $\epsilon_i(\theta) = y_i - \mathbf x_i^\top \theta$ which is simply ordinary least squares estimation for $\theta$. But clearly $$\sum_i \epsilon_i(\theta)^2 = \sum_i \lvert\epsilon_i(\theta)\lvert^2$$ now use weights and observe that $\lvert y_i - \hat y_i(\theta)\lvert = \lvert \...


3

Check out rqPen and hqreg packages in R which claim to perform quantile regression with lasso and elastic net respectively. Maybe you know this already but least absolute deviation regression is median regression or quantile regression at the 50% percentile. Minimizing the absolute deviation results in the median (with potential problem of multiple solutions)...


3

There's no GLM (no natural exponential family model) that corresponds to L1 (Least absolute value) regression.* Note that if you're doing MLE then a density of form $\frac{c}{\phi}\exp(-g(\frac{y-\mathbf{x}\beta}{\phi}))$ with have log-likelihood $-n\log(\phi)-\sum_i g(\frac{y_i-\mathbf{x}_i\beta}{\phi})$. Now maximizing likelihood with respect to the ...


2

Linear Programming can be generalized with convex optimization, where in addition to simplex, many more reliable algorithms are available. I would suggest you to check The Convex Optimization Book and the CVX toolbox they provided. Where you can easily formulate least absolute deviation with regularization. https://web.stanford.edu/~boyd/cvxbook/...


2

The definition of "analogue" is not clear. If you view $R^2$ as a metric to measure the "goodness of the fit" in regression setting. Then, likelihood can be used to evaluate the "goodness of the fit" for LDA (Linear Discriminant Analysis). EDIT: I think OP was trying to ask LAD (Least absolute deviations regression) but not LDA. Here is my answer to LAD. ...


2

Someone's going to complain that this question belongs to other stackoverflow forum. But the quick take on your question is, you need to follow the standard R idiom. That is, for function like lad, it wants a formula like y ~ x1 + ... and a data object. To use your syntax, you want l1fit which accepts a x and y. Just read the documentation carefully.


2

Phillips, R.F. (2002). Least absolute deviations estimation via the EM algorithm. Statistics and Computing 12: 281-285 Dielman, T (2005). Least absolute value regression: recent contributions. Journal of Statistical Computation and Simulation 75: 263-286 Roger Koenker has done brilliant work on Quantile Regression, I don't know whether any that might be ...


2

If errors are independent and follow the normal distribution (of any variance but consistent), then the sum of squared errors corresponds to their joint probability/likelihood. $\Pi e^{-x_i^2}=e^{-\Sigma x_i^2}$ So under those conditions minimizing the sum of square errors is the same as maximizing the likelihood. If a cost-minimizing prediction is needed ...


2

I think it's worth taking a step back and considering what the two losses imply. Looking at it from a probabilistic point of view, the loss function is equivalent to the assumed log-likelihood function and thus should correspond to how we think our measurements are distributed around their unknown 'true' values. As you say, in the case of OLS this is ...


1

The problem is different from OLS since LAD generates regression hyperplanes that necessarily fit m data points (in m-dimensional space). So, upon the arrival of a new point, LAD will either “jump” to a new hyperplane or stay where it was. This gives it a distinct discreet flavor. I imagine this in a 3D animation where points arrive and I watch the ...


1

In my opinion, it boils to that the squared error guarantees a unique solution, easier to work with and hence much more intuition. By only two main assumptions (and linearity of the error term), a quadratic loss function guarantees that the estimated coefficient is the unique minimized. Least-absolute deviations does not have this property. There is ...


1

Suppose one rolls one die (numered 1-6), and wants to compute its average deviation from the average value of 3.5. Two rolls would differ by 0.5, two by 1.5, and two by 2.5, for an average deviation of 1.5. If one takes the average of the squares of the values, one would have one deviation of 0.25, one of 2.25, and one of 6.25, for an average of 2.916 (35/...


1

Tree-based models perform recursive binary splits to optimize some metric, like information gain or Gini impurity. If you have continuous variables, then at each step, the algorithm will look for the variable/cutoff combination that is 'best' according to the metric used. In case of a discrete outcome variable, this relates to the number of correctly ...


1

Probably, you can log transform (or any other scale transform) the target variable and then use RMSE. It might remove the high outlier impact.


1

Please correct me if I am wrong, but it sounds like you are looking for a sweet spot between the sharpness of the L1 penalty and the smoothing of the L2 penalty. The elastic net might do the job. Slide 9 of https://web.stanford.edu/~hastie/TALKS/enet_talk.pdf seems to be capturing the issue you mention


Only top voted, non community-wiki answers of a minimum length are eligible