# Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

1,629 questions
2answers
36k views

### When to use regularization methods for regression?

In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS? In case this helps steer the discussion, my main interest is improving ...
3answers
6k views

2answers
3k views

### How does it make sense to do OLS after LASSO variable selection?

Recently I have found that in the applied econometrics literature, when dealing with feature selection problems, it is not uncommon to perform LASSO followed by an OLS regression using the selected ...
2answers
86k views

### What happens when I include a squared variable in my regression?

I start with my OLS regression: $$y = \beta _0 + \beta_1x_1+\beta_2 D + \varepsilon$$ where D is a dummy variable, the estimates become different from zero with a low p-value. I then preform a ...
6answers
4k views

### Intuitive explanation of the $(X^TX)^{-1}$ term in the variance of least square estimator

If $X$ is full rank, the inverse of $X^TX$ exists and we get the least squares estimate: $$\hat\beta = (X^TX)^{-1}XY$$ and $$\operatorname{Var}(\hat\beta) = \sigma^2(X^TX)^{-1}$$ How can we ...
3answers
6k views

### Can there be multiple local optimum solutions when we solve a linear regression?

I read this statement on one old true/false exam: We can get multiple local optimum solutions if we solve a linear regression problem by minimizing the sum of squared errors using gradient ...
1answer
7k views

### MLE vs least squares in fitting probability distributions

The impression that I got, based on several papers, books and articles that I've read, is that the recommended way of fitting a probability distribution on a set of data is by using maximum likelihood ...
1answer
7k views

### Question on how to normalize regression coefficient

Not sure if normalize is the correct word to use here, but I will try my best to illustrate what I am trying to ask. The estimator used here is least squares. Suppose you have $y=\beta_0+\beta_1x_1$, ...
3answers
4k views

5answers
887 views

### Bias towards natural numbers in the case of least squares

Why do we seek to minimize x^2 instead of minimizing |x|^1.95 or |x|^2.05. Are there reasons ...
3answers
2k views

### Why are there large coefficents for higher-order polynomial

In Bishop's book on machine learning, it discusses the problem of curve-fitting a polynomial function to a set of data points. Let M be the order of the polynomial fitted. It states as that We ...
1answer
1k views

### I have a line of best fit. I need data points that will not change my line of best fit

I'm giving a presentation about fitting lines. I have a simple linear function, $y=1x+b$. I'm trying to get scattered data points that I can put in a scatter plot that will keep my line of best fit ...
1answer
6k views

### Is bootstrapping standard errors and confidence intervals appropriate in regressions where homoscedasticity assumption is violated?

If in standard OLS regressions two assumptions are violated (normal distribution of errors, homoscedasticity), is bootstrapping standard errors and confidence intervals an appropriate alternative to ...