Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

3
votes
0answers
4k views

What to do when ovtest and linktest in Stata suggest model misspecification?

I have a sample that consists of 50 observations. The base model of the OLS-Regression with three control variables, two of them significant, has a $R^2=0.50$ and its F-Value is 7. Both ...
1
vote
2answers
115 views

Fitting a continuous result to categorical predictors semiparametrically

Suppose one has a relatively large number of observations, each of which consists of a continuous result and a small number (2 or perhaps 3) of categorical variables, each of which has a large ...
5
votes
1answer
720 views

Why is semipartial correlation cited so seldom?

In OLS regression, I find the semipartial correlation (a.k.a. part correlation) to be a very useful indicator. When squared, it shows each predictor's unique contribution to explained variance in the ...
1
vote
0answers
467 views

Predictive power (or $R^2$) adjusted for certain variables

I will frame this question for Ordinary Least Square (OLS) regression, but my question is for both OLS and Logistic. Let's say we data over 10000 different individuals. For each person we have three ...
14
votes
2answers
10k views

Why is a projection matrix of an orthogonal projection symmetric?

I am quite new to this, so I hope you forgive me if the question is naïve. (Context: I am learning econometrics from Davidson & MacKinnon's book "Econometric Theory and Methods", and they do not ...
0
votes
0answers
54 views

Summation representation for multivariate regressions (or other time-saving techniques) [duplicate]

Possible Duplicate: Efficient online linear regression Is there a summation representation for multivariate regressions? For example, if I regress $y$ on $X$ instead of using $\hat \beta = (X'X)^...
3
votes
1answer
91 views

Does including both raw and per capita measures as predictors reduce significance of either predictor?

I'm running a regression on independent variables, some of which are measured in different units, for example: The amount of broadband connections in a country The amount of broadband connections in ...
8
votes
2answers
172 views

Is it possible for $R^2$ of a regression on two variables be higher than the sum of $R^2$ for two regressions on the individual variables?

In OLS, is it possible for the $R^2$ of a regression on two variables be higher than the sum of $R^2$ for two regressions on the individual variables. $R^2(Y \sim A + B) > R^2(Y \sim A) + R^2(Y \...
6
votes
2answers
422 views

How to apply a soft coefficient constraint to an OLS regression?

I would like to estimate an ordinary least squares regression of the form $$ y = X\beta + \varepsilon, \ $$ except that, instead of minimizing the sum of squared residuals, $$ SSR(b)=(y-Xb)'(y-...
8
votes
4answers
8k views

Why do people often run a regression with and without control variables?

I often run regressions from a low-n dataset (~100 observations). Often the results are only significant with the inclusion of control variables. However, I often see journal articles where people (...
2
votes
3answers
1k views

How to interpret categorical variables in an OLS when only one category is statistically significant?

I am running a simple OLS. Dependent Variable: Population Change In A Congressional District After An Election Independent Variable: Who won the election: Democrat, Republican, or, Independent (...
34
votes
8answers
34k views

Is it valid to include a baseline measure as control variable when testing the effect of an independent variable on change scores?

I am attempting to run an OLS regression: DV: Change in weight over a year (initial weight - end weight) IV: Whether or not you exercise. However, it seems reasonable that heavier people will lose ...
3
votes
1answer
235 views

Why does the OLS estimator simplify as follows for the single regressor case?

I was reading in "A Guide to Econometrics" that given $Y = X \beta + \epsilon$, the variance covariance matrix of $\beta^\text{OLS}$ is given by $\sigma^2 (X' X)^{-1}$ where $\sigma^2$ is the ...
1
vote
1answer
74 views

Measuring tariff evasion before and after tariff cut

I have data on tariff rates and a proxy for tariff evasion (that is common in the literature). The data spans a couple of years before the country I'm studying implements a tariff reform and lowers ...
2
votes
1answer
2k views

prcomp() vs lm() results in R [duplicate]

I have a simple matrix: [,1] [,2] [,3] [1,] 1 2 3 [2,] 4 5 6 [3,] 7 8 9 [4,] 10 11 12 I have to calculate linear regression ...
7
votes
3answers
5k views

Estimating linear regression with OLS vs. ML

Assume that I'm going to estimate a linear regression where I assume $u\sim N(0,\sigma^2)$. What is the benefit of OLS against ML estimation? I know that we need to know a distribution of $u$ when we ...
24
votes
3answers
18k views

How to perform orthogonal regression (total least squares) via PCA?

I always use lm() in R to perform linear regression of $y$ on $x$. That function returns a coefficient $\beta$ such that $$y = \beta x.$$ Today I learned about ...
21
votes
1answer
18k views

Equivalence between least squares and MLE in Gaussian model

I am new to Machine Learning, and am trying to learn it on my own. Recently I was reading through some lecture notes and had a basic question. Slide 13 says that "Least Square Estimate is same as ...
2
votes
2answers
3k views

Tool to confirm Gaussian fit

I have a series of (x1,y1) points. I'm using a 3rd party software tool to which I feed these points. The tool then provides a mechanism for me to get back a series of (x2,y2) points that are on a ...
4
votes
2answers
323 views

What to do with p-values when standard errors are obviously biased

What am I supposed to do when I want to interpret significances, although I know that standard errors are biased because of wrong error term assumptions? I know that there is the possibility to use ...
2
votes
2answers
1k views

Is there a GLS estimator that has lower variance than OLS for sum of parameters in linear model under Gauss-Markov conditions?

I have a model $$Y=\beta_0 + \beta_1 x_1 + \beta_2x_2 +\epsilon$$ I would like the minimum variance unbiased estimate of $\gamma=\beta_1 + \beta_2$. Assuming the Gauss Markov conditions hold, but $...
4
votes
3answers
1k views

Dependent variable is a function of independent variables; can I sensibly include them in a regression?

We've created a survey asking students, among other things, their GPA (=weighted average of grades) and their marks in some specific courses (which count towards GPA). We wanted to see which ...
5
votes
1answer
2k views

Obtaining standard error on a data point obtained from linear regression

I have data with standard error, included below for clarity, ...
4
votes
2answers
258 views

Modeling number of phone calls with OLS

Have anyone tried modeling number of phone calls using OLS? The dataset is number of calls per months for each customer's account. The dependent variable is number of calls or average number of calls, ...
7
votes
2answers
585 views

Justification for using geometric weights in linear regression

In practical application, I have witnessed often the following practice. One observes a pair $(x_t, y_t)$ over time. Under the assumption that they are linearly related, we regress one against the ...
5
votes
0answers
1k views

Methods to best test lead/lag relationships

I was wondering if you can share your experiences on what you feel is the best method to test lead / lag relationships between I(1) time series variables (i.e stock prices) and advantages and ...
6
votes
3answers
420 views

Computing $(X^TX)^{-1}X^Ty$ in OLS

Let $A\in\mathbb{R}^{n \times n}$ be a dense symmetric positive-definite matrix (the $X^TX$ from here) and $b$ a vector in $\mathbb{R}^n$. I need to compute $A^{-1}b$. Two questions: Could you ...
9
votes
2answers
4k views

Influence functions and OLS

I am trying to understand how influence functions work. Could someone explain in the context of a simple OLS regression \begin{equation} y_i = \alpha + \beta \cdot x_i + \varepsilon_i \end{equation} ...
2
votes
1answer
772 views

What is the magnitude of bias in censored regression when OLS is applied?

If my dataset comprises few censored variables (<1%) and I fit the OLS regression using a heteroscedastic resistant estimator (the residuals are not terribly heteroscedastic to begin with)- are the ...
5
votes
3answers
2k views

Effect of missing data and outliers on least square estimation

Why is it that "missing data" and "outliers" can affect the performance of least square estimation?
83
votes
2answers
36k views

When to use regularization methods for regression?

In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS? In case this helps steer the discussion, my main interest is improving ...
9
votes
3answers
3k views

Why vertical distances?

Why does OLS estimation involve taking vertical deviations of the points to the line rather than horizontal distances?
3
votes
1answer
91 views

Estimating k in d=kv

This example was taken from Mathematical Statistics : A Unified Introduction (ISBN 9780387227696), page 58, under the section 'The Principle of Least Squares'. I think my problem has more to do with ...
42
votes
6answers
25k views

What algorithm is used in linear regression?

I usually hear about "ordinary least squares". Is that the most widely used algorithm used for linear regression? Are there reasons to use a different one?
12
votes
5answers
877 views

Bias towards natural numbers in the case of least squares

Why do we seek to minimize x^2 instead of minimizing |x|^1.95 or |x|^2.05. Are there reasons ...
12
votes
4answers
9k views

How large should a sample be for a given estimation technique and parameters?

Is there a rule-of thumb or even any way at all to tell how large a sample should be in order to estimate a model with a given number of parameters? So, for example, if I want to estimate a least-...