Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

83
votes
2answers
36k views

When to use regularization methods for regression?

In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS? In case this helps steer the discussion, my main interest is improving ...
57
votes
3answers
6k views

Why does ridge estimate become better than OLS by adding a constant to the diagonal?

I understand that the ridge regression estimate is the $\beta$ that minimizes residual sum of square and a penalty on the size of $\beta$ $$\beta_\mathrm{ridge} = (\lambda I_D + X'X)^{-1}X'y = \...
49
votes
3answers
41k views

Mean absolute error OR root mean squared error?

Why use Root Mean Squared Error (RMSE) instead of Mean Absolute Error (MAE)?? Hi I've been investigating the error generated in a calculation - I initially calculated the error as a Root Mean ...
44
votes
3answers
3k views

Where does the misconception that Y must be normally distributed come from?

Seemingly reputable sources claim that the dependent variable must be normally distributed: Model assumptions: $Y$ is normally distributed, errors are normally distributed, $e_i \sim N(0,\sigma^2)...
42
votes
6answers
25k views

What algorithm is used in linear regression?

I usually hear about "ordinary least squares". Is that the most widely used algorithm used for linear regression? Are there reasons to use a different one?
41
votes
5answers
66k views

Regression when the OLS residuals are not normally distributed

There are several threads on this site discussing how to determine if the OLS residuals are asymptotically normally distributed. Another way to evaluate the normality of the residuals with R code is ...
37
votes
5answers
16k views

Is minimizing squared error equivalent to minimizing absolute error? Why squared error is more popular than the latter?

When we conduct linear regression $y=ax+b$ to fit a bunch of data points $(x_1,y_1),(x_2,y_2),...,(x_n,y_n)$, the classic approach minimizes the squared error. I have long been puzzled by a question ...
34
votes
8answers
34k views

Is it valid to include a baseline measure as control variable when testing the effect of an independent variable on change scores?

I am attempting to run an OLS regression: DV: Change in weight over a year (initial weight - end weight) IV: Whether or not you exercise. However, it seems reasonable that heavier people will lose ...
34
votes
5answers
22k views

Why sigmoid function instead of anything else?

Why is the de-facto standard sigmoid function, $\frac{1}{1+e^{-x}}$, so popular in (non-deep) neural-networks and logistic regression? Why don't we use many of the other derivable functions, with ...
33
votes
5answers
32k views

How to derive the ridge regression solution?

I am having some issues with the derivation of the solution for ridge regression. I know the regression solution without the regularization term: $$\beta = (X^TX)^{-1}X^Ty.$$ But after adding the ...
32
votes
3answers
41k views

Maximum likelihood method vs. least squares method

What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimaton (LSE) ? Why can't we use MLE for predicting $y$ values in linear regression and vice versa? Any ...
28
votes
4answers
31k views

Why squared residuals instead of absolute residuals in OLS estimation? [duplicate]

Why are we using the squared residuals instead of the absolute residuals in OLS estimation? My idea was that we use the square of the error values, so that residuals below the fitted line (which are ...
24
votes
3answers
18k views

How to perform orthogonal regression (total least squares) via PCA?

I always use lm() in R to perform linear regression of $y$ on $x$. That function returns a coefficient $\beta$ such that $$y = \beta x.$$ Today I learned about ...
24
votes
2answers
12k views

Why is RSS distributed chi square times n-p?

I would like to understand why, under the OLS model, the RSS (residual sum of squares) is distributed $$\chi^2\cdot (n-p)$$ ($p$ being the number of parameters in the model, $n$ the number of ...
24
votes
1answer
12k views

Proof that the coefficients in an OLS model follow a t-distribution with (n-k) degrees of freedom

Background Suppose we have an Ordinary Least Squares model where we have $k$ coefficients in our regression model, $$\mathbf{y}=\mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}$$ where $\mathbf{\beta}$ ...
23
votes
6answers
12k views

Why do we usually choose to minimize the sum of square errors (SSE) when fitting a model?

The question is very simple: why, when we try to fit a model to our data, linear or non-linear, do we usually try to minimize the sum of the squares of errors to obtain our estimator for the model ...
22
votes
5answers
7k views

When is quantile regression worse than OLS?

Apart from some unique circumstances where we absolutely must understand the conditional mean relationship, what are the situations where a researcher should pick OLS over Quantile Regression? I don'...
22
votes
3answers
11k views

What does “all else equal” mean in multiple regression?

When we do multiple regressions and say we are looking at the average change in the $y$ variable for a change in an $x$ variable, holding all other variables constant, what values are we holding the ...
22
votes
1answer
5k views

Linear regression prediction interval

If the best linear approximation (using least squares) of my data points is the line $y=mx+b$, how can I calculate the approximation error? If I compute standard deviation of differences between ...
21
votes
2answers
5k views

Correlation between OLS estimators for intercept and slope

In a simple regression model, $$ y = \beta_0 + \beta_1 x + \varepsilon, $$ the OLS estimators $\hat{\beta}_0^{OLS}$ and $\hat{\beta}_1^{OLS}$ are correlated. The formula for the correlation ...
21
votes
1answer
18k views

Equivalence between least squares and MLE in Gaussian model

I am new to Machine Learning, and am trying to learn it on my own. Recently I was reading through some lecture notes and had a basic question. Slide 13 says that "Least Square Estimate is same as ...
21
votes
4answers
19k views

ANOVA vs multiple linear regression? Why is ANOVA so commonly used in experimental studies?

ANOVA vs multiple linear regression? I understand that both of these methods seem to use the same statistical model. However under what circumstances should I use which method? What are the ...
21
votes
4answers
3k views

Why does the least square solution give poor results in this case?

There is an image in page 204, chapter 4 of "pattern recognition and machine learning" by Bishop where I don't understand why the Least square solution gives poor results here: The previous paragraph ...
20
votes
1answer
8k views

How to calculate the prediction interval for an OLS multiple regression?

What is the algebraic notation to calculate the prediction interval for multiple regression? It sounds silly, but I am having trouble finding a clear algebraic notation of this.
20
votes
2answers
6k views

Least squares logistic regression

I have seen it claimed in Hosmer & Lemeshow (and elsewhere) that least squares parameter estimation in logistic regression is suboptimal (does not lead to a minimum variance unbiased estimator). ...
19
votes
2answers
8k views

Is there any advantage of SVD over PCA?

I know how to calculate PCA and SVD mathematically, and I know that both can be applied to Linear Least Squares regression. The main advantage of SVD mathematically seems to be that it can be applied ...
18
votes
2answers
3k views

Proof that F-statistic follows F-distribution

In light of this question : Proof that the coefficients in an OLS model follow a t-distribution with (n-k) degrees of freedom I would love to understand why $$ F = \frac{(\text{TSS}-\text{RSS})/(p-1)...
18
votes
2answers
3k views

How does it make sense to do OLS after LASSO variable selection?

Recently I have found that in the applied econometrics literature, when dealing with feature selection problems, it is not uncommon to perform LASSO followed by an OLS regression using the selected ...
18
votes
0answers
524 views

Bound for Arithmetic Harmonic mean inequality for matrices?

NOTE: This question has originally been posted in MSE, but it did not generate any interest. It was first posted there, because the question itself is a pure matrix-algebra question. Nevertheless, ...
17
votes
4answers
1k views

Why is Ordinary Least Squares performing better than Poisson regression?

I'm trying to fit a regression to explain the number of homicides in each district of a city. Although I know that my data follows a Poisson distribution, I tried to fit an OLS like this: $log(y+1) = ...
17
votes
2answers
83k views

What happens when I include a squared variable in my regression?

I start with my OLS regression: $$ y = \beta _0 + \beta_1x_1+\beta_2 D + \varepsilon $$ where D is a dummy variable, the estimates become different from zero with a low p-value. I then preform a ...
17
votes
3answers
6k views

Can there be multiple local optimum solutions when we solve a linear regression?

I read this statement on one old true/false exam: We can get multiple local optimum solutions if we solve a linear regression problem by minimizing the sum of squared errors using gradient ...
16
votes
1answer
6k views

Question on how to normalize regression coefficient

Not sure if normalize is the correct word to use here, but I will try my best to illustrate what I am trying to ask. The estimator used here is least squares. Suppose you have $y=\beta_0+\beta_1x_1$, ...
16
votes
3answers
3k views

Why not use the “normal equations” to find simple least squares coefficients?

I saw this list here and couldn't believe there were so many ways to solve least squares. The "normal equations" on Wikipedia seemed to be a fairly straight forward way: $$ {\displaystyle {\begin{...
16
votes
6answers
3k views

Intuitive explanation of the $(X^TX)^{-1}$ term in the variance of least square estimator

If $X$ is full rank, the inverse of $X^TX$ exists and we get the least squares estimate: $$\hat\beta = (X^TX)^{-1}XY$$ and $$\operatorname{Var}(\hat\beta) = \sigma^2(X^TX)^{-1}$$ How can we ...
16
votes
1answer
583 views

Reversing ridge regression: given response matrix and regression coefficients, find suitable predictors

Consider a standard OLS regression problem$\newcommand{\Y}{\mathbf Y}\newcommand{\X}{\mathbf X}\newcommand{\B}{\boldsymbol\beta}\DeclareMathOperator*{argmin}{argmin}$: I have matrices $\Y$ and $\X$ ...
16
votes
1answer
7k views

MLE vs least squares in fitting probability distributions

The impression that I got, based on several papers, books and articles that I've read, is that the recommended way of fitting a probability distribution on a set of data is by using maximum likelihood ...
15
votes
1answer
1k views

Why does this regression NOT fail due to perfect multicollinearity, although one variable is a linear combination of others?

Today, I was playing around with a small dataset and performed a simple OLS regression which I expected to fail due to perfect multicollinearity. However, it didn't. This implies that my understanding ...
15
votes
2answers
2k views

What is the “partial” in partial least squares methods?

In partial least squares regression (PLSR) or partial least squares structural equation modelling (PLS-SEM), what does the term "partial" refer to?
15
votes
1answer
2k views

Definition and Convergence of Iteratively Reweighted Least Squares

I've been using iteratively reweighted least squares (IRLS) to minimize functions of the following form, $J(m) = \sum_{i=1}^{N} \rho \left(\left| x_i - m \right|\right)$ where $N$ is the number of ...
14
votes
2answers
4k views

Measures of residuals heteroscedasticity

This wikipedia link lists a number of techniques to detect OLS residuals heteroscedasticity. I would like to learn which hands-on technique is more efficient in detecting regions affected by ...
14
votes
2answers
10k views

Why is a projection matrix of an orthogonal projection symmetric?

I am quite new to this, so I hope you forgive me if the question is naïve. (Context: I am learning econometrics from Davidson & MacKinnon's book "Econometric Theory and Methods", and they do not ...
13
votes
3answers
734 views

Why trace of $I−X(X′X)^{-1}X′$ is $n-p$ in least square regression when the parameter vector $\beta$ is of p dimensions?

In the model ${y} = X \beta + \epsilon$, we could estimate $\beta$ using the normal equation: $$\hat{\beta} = (X'X)^{-1}X'y,$$ and we could get $$\hat{y} = X \hat{\beta}.$$ The vector of residuals ...
13
votes
4answers
53k views

Whats the relationship between $R^2$ and F-Test?

I was wondering if there is a relationship between $R^2$ and a F-Test. Usually $$R^2=\frac {\sum (\hat Y_t - \bar Y)^2 / T-1} { Y_t - \bar Y)^2 / T-1}$$ and it measures the strength of the linear ...
13
votes
3answers
2k views

Why are there large coefficents for higher-order polynomial

In Bishop's book on machine learning, it discusses the problem of curve-fitting a polynomial function to a set of data points. Let M be the order of the polynomial fitted. It states as that We ...
13
votes
1answer
1k views

I have a line of best fit. I need data points that will not change my line of best fit

I'm giving a presentation about fitting lines. I have a simple linear function, $y=1x+b$. I'm trying to get scattered data points that I can put in a scatter plot that will keep my line of best fit ...
13
votes
1answer
5k views

Is bootstrapping standard errors and confidence intervals appropriate in regressions where homoscedasticity assumption is violated?

If in standard OLS regressions two assumptions are violated (normal distribution of errors, homoscedasticity), is bootstrapping standard errors and confidence intervals an appropriate alternative to ...
13
votes
3answers
6k views

Utility of the Frisch-Waugh theorem

I am supposed to teach the Frish Waugh theorem in econometrics, which I have not studied. I have understood the maths behind it and I hope the idea too "the coefficient you get for a particular ...
13
votes
3answers
1k views

Linear regression: any non-normal distribution giving identity of OLS and MLE?

This question is inspired from the long discussion in comments here: How does linear regression use the normal distribution? In the usual linear regression model, for simplicity here written with ...
13
votes
1answer
4k views

R-squared in linear model verses deviance in generalized linear model?

Here's my context for this question: From what I can tell, we cannot run an ordinary least squares regression in R when using weighted data and the survey package. ...