Questions tagged [least-squares]
Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.
2,721
questions
24
votes
1
answer
9k
views
Linear regression prediction interval
If the best linear approximation (using least squares) of my data points is the line $y=mx+b$, how can I calculate the approximation error? If I compute standard deviation of differences between ...
- 241
4
votes
1
answer
3k
views
"Studentized" bootstrap confidence interval for variance of OLS error terms
Suppose that one has the usual regression model $\mathbf{y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\varepsilon}$, where each $\varepsilon_t$ is iid distributed with $\mathbb{E}(\varepsilon_t) = ...
- 1,147
5
votes
1
answer
201
views
Aggregating all measurements per x-value in least-square fitting
We want to select one out of a given set of (continuous) functions that best matches a set of observations $\{(s_i,c_i) \mid 1 \leq i \leq N\} \subseteq\mathbb{N} \times \mathbb{N}$ (input size and ...
- 153
16
votes
2
answers
5k
views
Measures of residuals heteroscedasticity
This wikipedia link lists a number of techniques to detect OLS residuals heteroscedasticity. I would like to learn which hands-on technique is more efficient in detecting regions affected by ...
- 4,288
3
votes
1
answer
114
views
When can we add a statistical touch to least square optimization problems?
I am trying to connect the dots between statistics and linear algebra/optimization.
As you know, Least Square problems are linear algebra and optimization problems. But they also can be connected to ...
- 2,315
4
votes
1
answer
424
views
How can you relate the coefficients of a multivariate regression of ${\bf Y} \sim {\bf X}$ to the coefficients of ${\bf X} \sim {\bf Y}$?
Is anyone aware of a orthogonal multiple regression library that is implemented in say R, Scipy, Matlab, Octave, etc.? (Or even fortran/C...) If I'm not mistaken, it would not be difficult to write ...
- 799
3
votes
0
answers
279
views
Is is correct to compare t-statistics of different pairs of cointegrated timeseries?
I am testing for cointegration all the pairs from a set of 100 stocks. I run an Ordinary Least Square Regression on each pair and then I test for the existence of unit roots in the residuals. I am ...
5
votes
1
answer
2k
views
Does a regression tree strictly dominate OLS in prediction?
Since OLS tries to measure E[Y|X], and regression trees try to partition the data into different branches, then take means of the response under different branches, is it reasonable to say that ...
- 1,692
1
vote
1
answer
190
views
Confusion regarding least squares method
I am having some confusion regarding least squares method. Actually, least squares method is for minimizing the square of the $L_2$ norm of $Ax-b$ as given in this video lecture. However I am confused ...
- 1,491
8
votes
0
answers
190
views
Scope of non-linear least squares
edit: tl;dr: I can coerce a lot of optimization problems to take the form of a non-linear least squares problem, but does it make sense to do so?
Suppose we have some empirical data $P=\{(x_i', y_i')\...
- 323
58
votes
5
answers
102k
views
Regression when the OLS residuals are not normally distributed
There are several threads on this site discussing how to determine if the OLS residuals are asymptotically normally distributed. Another way to evaluate the normality of the residuals with R code is ...
- 4,288
8
votes
1
answer
6k
views
OLS vs. logistic regression for exploratory analysis with a binary outcome
In the idealized logistic model, we obtain an S-shaped curve linking each continuous IV to the DV. But in practice this S-shape infrequently occurs, making the logistic approach seem a little less ...
- 11.8k
6
votes
4
answers
167
views
Could logistic regression be used to detect large errors in least squares regression?
I have the following linear model:
$$w^*=\text{arg min}_w\sum_{i=1}^N \bigg(Y_i-\sum_{j=1}^M X_{i,j}\times w_j\bigg)^2$$
Let $T \in N^*$ and $e_i=|Y_i-\sum_{j=1}^M X_{i,j}\times w_j|$.
It's ...
- 91
2
votes
1
answer
2k
views
Least squares with non-linear constraints
I have the following problem:
I want to minimize a least square problem with non-linear restrictions. The start model has the following form:
$$w^*=\text{min arg}_w \sum_{i=1}^{N} (y_i-\sum_{j=1}^3 ...
- 91
2
votes
1
answer
966
views
How can I minimize this least squares problem with inequality constraints?
I have a least square problem with two different inequality problems. I can not use NNLS because its just solves least square problem with equality and inequality problems or just one inequality ...
- 195
5
votes
1
answer
399
views
Nonlinear regression model linear in some parameters
In a set of lecture notes that I stumbled upon online, the author discusses a nonlinear regression model, which is linear in some parameters, like this
$$
y = \theta_1 + \theta_2\exp \left( {\...
- 51
3
votes
0
answers
398
views
Standard errors in weighted least squares
My data is a panel of countries by year. Suppose my main RHS variable is a country's GDP and my main LHS variable classifies countries by whether the country is a democracy.
Is it desirably to ...
- 815
18
votes
1
answer
8k
views
MLE vs least squares in fitting probability distributions
The impression that I got, based on several papers, books and articles that I've read, is that the recommended way of fitting a probability distribution on a set of data is by using maximum likelihood ...
- 343
3
votes
1
answer
2k
views
Deriving OLS estimates using method of moments
I've worked the slope all the way down to $\sum [x_i(y_i - \bar{y})] = \hat\beta_1 \sum[x_i(x_i - \bar{x})]$
But I can not figure out how to show the steps for:
$\sum[x_i(y_i - \bar{y})] = \sum(x_i -...
- 771
5
votes
2
answers
10k
views
2 period difference-in-differences fixed effects versus OLS
I have a question on the difference-in-differences estimator. Suppose my data consists of two periods and the treatment is administered to some of the individuals in period $t = 2$. I estimate this ...
- 118
20
votes
2
answers
7k
views
Least squares logistic regression [duplicate]
I have seen it claimed in Hosmer & Lemeshow (and elsewhere) that least squares parameter estimation in logistic regression is suboptimal (does not lead to a minimum variance unbiased estimator). ...
- 211
3
votes
0
answers
887
views
Pregibon test for linearity vs. Ramsey's RESET test
Does every Ordinary Least Squares (OLS) regression model have to pass both the Pregibon Test for Linearity (sometimes called link-test) and the Ramsey RESET tests?
I am working on an OLS model and it ...
- 193
4
votes
2
answers
517
views
Significance testing of slopes with replicates
I've made replicate measurements of a parameter (fluorescence) which is expected to increase with time, and I'm having a hard time understanding how to test for significance of the slope of the ...
- 684
5
votes
1
answer
3k
views
Quadratic term and variance inflation factor in OLS estimation
One question I have carried around with me for a while is related to including quadratic terms for model specifications. I wonder why it is considered ok to include linear and quadratic terms into OLS ...
- 193
1
vote
1
answer
3k
views
How can I get residuals output in a variable or to act like a data frame in R?
I have an object:
noise.lm
it's just a simple linear model with an X and Y.
when I type in
resid(noise.lm)
it produces ...
- 771
4
votes
1
answer
263
views
Least angle regression for a set of vectors?
As far as I know, LARS solves the following problem (using the same notation as Efron et al. Least angle regression):
Given a vector y, and a matrix X. Pick some column vectors from X, and express ...
- 41
5
votes
2
answers
2k
views
What are similarity measures between a line and a set of points?
A colleague discussed about a concept of similarity between two different entities line and a set of points. My first guess for solution was considering distances squared ($d^2$) as in LS. So for the ...
- 1,416
2
votes
1
answer
145
views
Regression with complex causal structure
I have already run a whole bunch of OLS and found the following
Regress P= B1*L+e_1, found b1<0
Regress X= B2*L+e_2, found b2>0
Regress X= B3*P+e_3, found b3<0
I want to build a case with ...
- 21
5
votes
2
answers
380
views
Formula for single linear regression for dataset that has uncertainties on both x and y
I'm going to teach classes on Physics Laboratory on First year of Bachelor studies. In most of the excercises during data analysis students will have to fit a line to measurements they have taken.
I ...
- 1,120
5
votes
2
answers
1k
views
Multiple regression and OLS. How to choose the best "non-linear" specification?
Let's say I have to make a multiple regression like:
$ Y_i = \beta_0 + \beta_1 x_i + \beta_2 w_i + ... +\beta_3 z_i + \epsilon_i $
Then I run a Ramsey RESET test upon it and discover that my linear ...
- 307
8
votes
3
answers
28k
views
Selecting best model based on linear, quadratic and cubic fit of data
I have a Java code that performs a linear regression on a set of data using the Gauss-Jordan elimination. It calculates a linear, quadratic and cubic functions using the least squares method.
My ...
- 81
0
votes
0
answers
303
views
Error amplification in linear regression with uncertainties in design matrix
I am running a least squares regression, and I am trying to estimate the uncertainty on the fit solution. The problem is, I have an uncertainty in my design matrix coefficients as well as in my ...
- 11
3
votes
1
answer
2k
views
How do I handle very different weights in a least squares fit?
I'm performing a weighted linear least squares fit, where the weights correspond to the number of counts of a specific observation. Due to the nature of the data, it is possible that a small handful ...
- 1,668
1
vote
3
answers
2k
views
Least squared regression where the coefficients switch signs upon the addition of new variable
I am conducting a least square regression using the python library numpy. When I run OLS (ordinary least square regression) over just the variable IE6 I get this output (with the key takeaway being ...
- 221
45
votes
3
answers
30k
views
Why is RSS distributed chi square times n-p?
I would like to understand why, under the OLS model, the RSS (residual sum of squares) is distributed $$\chi^2\cdot (n-p)$$ ($p$ being the number of parameters in the model, $n$ the number of ...
- 20.8k
3
votes
0
answers
6k
views
What to do when ovtest and linktest in Stata suggest model misspecification?
I have a sample that consists of 50 observations. The base model of the OLS-Regression with three control variables, two of them significant, has a $R^2=0.50$ and its F-Value is 7. Both ...
- 31
1
vote
2
answers
117
views
Fitting a continuous result to categorical predictors semiparametrically
Suppose one has a relatively large number of observations, each of which consists of a continuous result and a small number (2 or perhaps 3) of categorical variables, each of which has a large ...
- 151
7
votes
1
answer
842
views
Why is semipartial correlation cited so seldom?
In OLS regression, I find the semipartial correlation (a.k.a. part correlation) to be a very useful indicator. When squared, it shows each predictor's unique contribution to explained variance in the ...
- 11.8k
1
vote
0
answers
504
views
Predictive power (or $R^2$) adjusted for certain variables
I will frame this question for Ordinary Least Square (OLS) regression, but my question is for both OLS and Logistic.
Let's say we data over 10000 different individuals. For each person we have three ...
- 559
19
votes
2
answers
16k
views
Why is a projection matrix of an orthogonal projection symmetric?
I am quite new to this, so I hope you forgive me if the question is naïve. (Context: I am learning econometrics from Davidson & MacKinnon's book "Econometric Theory and Methods", and they do not ...
- 1,147
0
votes
0
answers
59
views
Summation representation for multivariate regressions (or other time-saving techniques) [duplicate]
Possible Duplicate:
Efficient online linear regression
Is there a summation representation for multivariate regressions?
For example, if I regress $y$ on $X$ instead of using $\hat \beta = (X'X)^...
- 1,241
3
votes
1
answer
106
views
Does including both raw and per capita measures as predictors reduce significance of either predictor?
I'm running a regression on independent variables, some of which are measured in different units, for example:
The amount of broadband connections in a country
The amount of broadband connections in ...
- 173
9
votes
2
answers
245
views
Is it possible for $R^2$ of a regression on two variables be higher than the sum of $R^2$ for two regressions on the individual variables?
In OLS, is it possible for the $R^2$ of a regression on two variables be higher than the sum of $R^2$ for two regressions on the individual variables.
$R^2(Y \sim A + B) > R^2(Y \sim A) + R^2(Y \...
- 203
6
votes
2
answers
669
views
How to apply a soft coefficient constraint to an OLS regression?
I would like to estimate an ordinary least squares regression of the form
$$
y = X\beta + \varepsilon, \
$$
except that, instead of minimizing the sum of squared residuals,
$$
SSR(b)=(y-Xb)'(y-...
- 223
8
votes
4
answers
11k
views
Why do people often run a regression with and without control variables?
I often run regressions from a low-n dataset (~100 observations). Often the results are only significant with the inclusion of control variables. However, I often see journal articles where people (...
- 611
2
votes
3
answers
2k
views
How to interpret categorical variables in an OLS when only one category is statistically significant?
I am running a simple OLS.
Dependent Variable: Population Change In A Congressional District After An Election
Independent Variable: Who won the election: Democrat, Republican, or, Independent (...
- 611
47
votes
9
answers
47k
views
Is it valid to include a baseline measure as control variable when testing the effect of an independent variable on change scores?
I am attempting to run an OLS regression:
DV: Change in weight over a year (initial weight - end weight)
IV: Whether or not you exercise.
However, it seems reasonable that heavier people will lose ...
- 611
3
votes
1
answer
389
views
Why does the OLS estimator simplify as follows for the single regressor case?
I was reading in "A Guide to Econometrics" that given $Y = X \beta + \epsilon$, the variance covariance matrix of $\beta^\text{OLS}$ is given by $\sigma^2 (X' X)^{-1}$ where $\sigma^2$ is the ...
- 31
1
vote
1
answer
79
views
Measuring tariff evasion before and after tariff cut
I have data on tariff rates and a proxy for tariff evasion (that is common in the literature). The data spans a couple of years before the country I'm studying implements a tariff reform and lowers ...
- 57
2
votes
1
answer
2k
views
prcomp() vs lm() results in R [duplicate]
I have a simple matrix:
[,1] [,2] [,3]
[1,] 1 2 3
[2,] 4 5 6
[3,] 7 8 9
[4,] 10 11 12
I have to calculate linear regression ...
- 2,427