Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

16
votes
1answer
6k views

Question on how to normalize regression coefficient

Not sure if normalize is the correct word to use here, but I will try my best to illustrate what I am trying to ask. The estimator used here is least squares. Suppose you have $y=\beta_0+\beta_1x_1$, ...
41
votes
5answers
66k views

Regression when the OLS residuals are not normally distributed

There are several threads on this site discussing how to determine if the OLS residuals are asymptotically normally distributed. Another way to evaluate the normality of the residuals with R code is ...
22
votes
1answer
5k views

Linear regression prediction interval

If the best linear approximation (using least squares) of my data points is the line $y=mx+b$, how can I calculate the approximation error? If I compute standard deviation of differences between ...
24
votes
3answers
18k views

How to perform orthogonal regression (total least squares) via PCA?

I always use lm() in R to perform linear regression of $y$ on $x$. That function returns a coefficient $\beta$ such that $$y = \beta x.$$ Today I learned about ...
33
votes
5answers
32k views

How to derive the ridge regression solution?

I am having some issues with the derivation of the solution for ridge regression. I know the regression solution without the regularization term: $$\beta = (X^TX)^{-1}X^Ty.$$ But after adding the ...
34
votes
8answers
34k views

Is it valid to include a baseline measure as control variable when testing the effect of an independent variable on change scores?

I am attempting to run an OLS regression: DV: Change in weight over a year (initial weight - end weight) IV: Whether or not you exercise. However, it seems reasonable that heavier people will lose ...
22
votes
3answers
11k views

What does “all else equal” mean in multiple regression?

When we do multiple regressions and say we are looking at the average change in the $y$ variable for a change in an $x$ variable, holding all other variables constant, what values are we holding the ...
21
votes
1answer
18k views

Equivalence between least squares and MLE in Gaussian model

I am new to Machine Learning, and am trying to learn it on my own. Recently I was reading through some lecture notes and had a basic question. Slide 13 says that "Least Square Estimate is same as ...
57
votes
3answers
6k views

Why does ridge estimate become better than OLS by adding a constant to the diagonal?

I understand that the ridge regression estimate is the $\beta$ that minimizes residual sum of square and a penalty on the size of $\beta$ $$\beta_\mathrm{ridge} = (\lambda I_D + X'X)^{-1}X'y = \...
24
votes
2answers
12k views

Why is RSS distributed chi square times n-p?

I would like to understand why, under the OLS model, the RSS (residual sum of squares) is distributed $$\chi^2\cdot (n-p)$$ ($p$ being the number of parameters in the model, $n$ the number of ...
13
votes
3answers
2k views

Why are there large coefficents for higher-order polynomial

In Bishop's book on machine learning, it discusses the problem of curve-fitting a polynomial function to a set of data points. Let M be the order of the polynomial fitted. It states as that We ...
83
votes
2answers
36k views

When to use regularization methods for regression?

In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS? In case this helps steer the discussion, my main interest is improving ...
49
votes
3answers
41k views

Mean absolute error OR root mean squared error?

Why use Root Mean Squared Error (RMSE) instead of Mean Absolute Error (MAE)?? Hi I've been investigating the error generated in a calculation - I initially calculated the error as a Root Mean ...
42
votes
6answers
25k views

What algorithm is used in linear regression?

I usually hear about "ordinary least squares". Is that the most widely used algorithm used for linear regression? Are there reasons to use a different one?
4
votes
1answer
965 views

Understanding linear algebra in Ordinary Least Squares derivation

In ordinary least squared there is this equation (Kevin Murphy book page 221, latest edition) $$NLL(w)=\frac{1}{2}({y-Xw})^T(y-Xw)=\frac{1}{2}w^T(X^TX)w-w^T(X^T)y$$ I am not sure how the RHS equals ...
37
votes
5answers
16k views

Is minimizing squared error equivalent to minimizing absolute error? Why squared error is more popular than the latter?

When we conduct linear regression $y=ax+b$ to fit a bunch of data points $(x_1,y_1),(x_2,y_2),...,(x_n,y_n)$, the classic approach minimizes the squared error. I have long been puzzled by a question ...
24
votes
1answer
12k views

Proof that the coefficients in an OLS model follow a t-distribution with (n-k) degrees of freedom

Background Suppose we have an Ordinary Least Squares model where we have $k$ coefficients in our regression model, $$\mathbf{y}=\mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}$$ where $\mathbf{\beta}$ ...
28
votes
4answers
31k views

Why squared residuals instead of absolute residuals in OLS estimation? [duplicate]

Why are we using the squared residuals instead of the absolute residuals in OLS estimation? My idea was that we use the square of the error values, so that residuals below the fitted line (which are ...
4
votes
1answer
1k views

Derivation of the effect of unmodeled confounders on OLS estimates

I'd like to see how not adjusting for confounders affects estimated coefficients in linear regression. Here is what my lecture notes say: Un-adjusted model: $E[Yi] = \beta_0 + \beta_1X_i$ Adjusted ...
13
votes
4answers
53k views

Whats the relationship between $R^2$ and F-Test?

I was wondering if there is a relationship between $R^2$ and a F-Test. Usually $$R^2=\frac {\sum (\hat Y_t - \bar Y)^2 / T-1} { Y_t - \bar Y)^2 / T-1}$$ and it measures the strength of the linear ...
5
votes
3answers
11k views

Difference between fixed effects dummies and fixed effects estimator?

I started to read about panel regression models. However, I am a bit confused about the different model specifications in the fixed effects model: Does a fixed effects panel regression always mean ...
11
votes
3answers
6k views

Perform linear regression, but force solution to go through some particular data points

I know how to perform a linear regression on a set of points. That is, I know how to fit a polynomial of my choice, to a given data set, (in the LSE sense). However, what I do not know, is how to ...
13
votes
1answer
1k views

I have a line of best fit. I need data points that will not change my line of best fit

I'm giving a presentation about fitting lines. I have a simple linear function, $y=1x+b$. I'm trying to get scattered data points that I can put in a scatter plot that will keep my line of best fit ...
14
votes
2answers
4k views

Measures of residuals heteroscedasticity

This wikipedia link lists a number of techniques to detect OLS residuals heteroscedasticity. I would like to learn which hands-on technique is more efficient in detecting regions affected by ...
11
votes
3answers
783 views

What are the merits of different approaches to detecting collinearity?

I want to detect whether collinearity is a problem in my OLS regression. I understand that variance inflation factors and the condition index are two commonly used measures, but am finding it ...
11
votes
1answer
5k views

Omitted variable bias in logistic regression vs. omitted variable bias in ordinary least squares regression

I have a question about omitted variable bias in logistic and linear regression. Say I omit some variables from a linear regression model. Pretend that those omitted variables are uncorrelated with ...
13
votes
3answers
6k views

Utility of the Frisch-Waugh theorem

I am supposed to teach the Frish Waugh theorem in econometrics, which I have not studied. I have understood the maths behind it and I hope the idea too "the coefficient you get for a particular ...
19
votes
2answers
8k views

Is there any advantage of SVD over PCA?

I know how to calculate PCA and SVD mathematically, and I know that both can be applied to Linear Least Squares regression. The main advantage of SVD mathematically seems to be that it can be applied ...
12
votes
5answers
4k views

Recursive (online) regularised least squares algorithm

Can anyone point me in the direction of an online (recursive) algorithm for Tikhonov Regularisation (regularised least squares)? In an offline setting, I would calculate $\hat\beta=(X^TX+λI)^{−1}X^TY$...
21
votes
2answers
5k views

Correlation between OLS estimators for intercept and slope

In a simple regression model, $$ y = \beta_0 + \beta_1 x + \varepsilon, $$ the OLS estimators $\hat{\beta}_0^{OLS}$ and $\hat{\beta}_1^{OLS}$ are correlated. The formula for the correlation ...
3
votes
1answer
1k views

Objective function of canonical correlation analysis (CCA)

Given two vectors of random variables $X$ and $Y$, Canonical Correlation Analysis (CCA) finds the transformation matrices $A$ and $B$ so that $\operatorname{corr}(A_{1*} X, B_{1*} Y)$ is first maximal,...
5
votes
2answers
2k views

Estimation of unit-root AR(1) model with OLS

Given a random walk $x_t$, $$x_t=x_{t-1}+\varepsilon_t,$$ consider estimating the slope coefficient $\beta$ in $$x_t=\beta x_{t-1}+\varepsilon_t$$ by OLS. This question and the following answer ...
4
votes
2answers
3k views

Is least squares the standard method to fit a 3 parameters Gaussian function to some x and y data?

A participant in one experiment needs to decide whether a flash and a sound are simultaneous or not for many possible asynchronies between the flash and the sound (x in seconds). For each asynchrony, ...
2
votes
1answer
3k views

Common trends with Difference in differences

For OLS to have good properties, we should assume that $E(u|X)=0$. When using DiD, is it then this assumption which requires the so-called common trend assumption (need for the DiD to have good ...
8
votes
3answers
6k views

Why is OLS estimator of AR(1) coefficient biased?

I am trying to understand why OLS gives a biased estimator of an AR(1) process. Consider $$ \begin{aligned} y_{t} &= \alpha + \beta y_{t-1} + \epsilon_{t}, \\ \epsilon_{t} &\stackrel{iid}{\...
5
votes
2answers
639 views

Intuition for recursive least squares

The least squares formula, $\beta = (X'X)^{-1}X'Y$ can be recursively formulated as \begin{align} \beta_t &= \beta_{t-1} +\frac{1}{t}R_t^{-1}x_t'(y_t-x_t\beta_{t-1}),\\ R_t &= R_{t-1}+\frac{1}...
4
votes
1answer
617 views

Algorithm for minimization of sum of squares in regression packages

I'd like to know what technique or techniques are used by regression packages, (in particular the lm function of R) to minimize the sum of squares. Does it use ...
8
votes
1answer
5k views

OLS vs. logistic regression for exploratory analysis with a binary outcome

In the idealized logistic model, we obtain an S-shaped curve linking each continuous IV to the DV. But in practice this S-shape infrequently occurs, making the logistic approach seem a little less ...
4
votes
2answers
2k views

Is the model wrong if a coefficient changes from minus in correlation table to plus in OLS?

Perhaps a very basic question but one that has me confused. Say, in a correlation table the relationship between A and the DV (B)...
1
vote
2answers
179 views

Full-Rank design matrix from overdetermined linear model

I'm trying to create a full-rank design matrix X for a randomized block design model starting from something like the example from page 3/8 of this paper . It's been suggested that I can go about ...
44
votes
3answers
3k views

Where does the misconception that Y must be normally distributed come from?

Seemingly reputable sources claim that the dependent variable must be normally distributed: Model assumptions: $Y$ is normally distributed, errors are normally distributed, $e_i \sim N(0,\sigma^2)...
21
votes
4answers
19k views

ANOVA vs multiple linear regression? Why is ANOVA so commonly used in experimental studies?

ANOVA vs multiple linear regression? I understand that both of these methods seem to use the same statistical model. However under what circumstances should I use which method? What are the ...
20
votes
1answer
8k views

How to calculate the prediction interval for an OLS multiple regression?

What is the algebraic notation to calculate the prediction interval for multiple regression? It sounds silly, but I am having trouble finding a clear algebraic notation of this.
11
votes
3answers
37k views

Assumptions to derive OLS estimator

Can someone briefly explain for me, why each of the six assumptions is needed in order to compute the OLS estimator? I found only about multicollinearity—that if it exists we cannot invert (X'X) ...
17
votes
2answers
83k views

What happens when I include a squared variable in my regression?

I start with my OLS regression: $$ y = \beta _0 + \beta_1x_1+\beta_2 D + \varepsilon $$ where D is a dummy variable, the estimates become different from zero with a low p-value. I then preform a ...
12
votes
4answers
16k views

How to choose initial values for nonlinear least squares fit

The question above says it all. Basically my question is for a generic fitting function (could be arbitrarily complicated) which will be nonlinear in the parameters I am trying to estimate, how does ...
13
votes
3answers
1k views

Linear regression: any non-normal distribution giving identity of OLS and MLE?

This question is inspired from the long discussion in comments here: How does linear regression use the normal distribution? In the usual linear regression model, for simplicity here written with ...
12
votes
1answer
5k views

Proof of LOOCV formula

From An Introduction to Statistical Learning by James et al., the leave-one-out cross-validation (LOOCV) estimate is defined by $$\text{CV}_{(n)} = \dfrac{1}{n}\sum\limits_{i=1}^{n}\text{MSE}_i$$ ...
13
votes
1answer
5k views

Is bootstrapping standard errors and confidence intervals appropriate in regressions where homoscedasticity assumption is violated?

If in standard OLS regressions two assumptions are violated (normal distribution of errors, homoscedasticity), is bootstrapping standard errors and confidence intervals an appropriate alternative to ...
7
votes
3answers
19k views

Selecting best model based on linear, quadratic and cubic fit of data

I have a Java code that performs a linear regression on a set of data using the Gauss-Jordan elimination. It calculates a linear, quadratic and cubic functions using the least squares method. My ...