Questions tagged [least-squares]
Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.
2,721
questions
4
votes
2
answers
94
views
Help: Partitioned samples efficiency in OLS compared to one sample regression
As usual, we can estimate by OLS the model (in matrix form) $Y=\alpha+\beta*X+u$ with a sample of $n+m$ observations. The OLS estimator is $\hat{\beta}=(X^{T}X)^{-1}X^{T}Y$. Now, if we partition our ...
- 43
5
votes
1
answer
197
views
Does Frisch–Waugh–Lovell theorem work on panel data?
We consider the following panel regression model
\begin{align}
Y_{i} = X_{1,i}\beta_1 + X_{2,i}\beta_2 + \epsilon_{i} , \ i=1,...,N,
\end{align}
where $Y_i := [y_{i1},...,y_{iT}]'$, $X_{k,i}:= [x_{k,...
- 105
0
votes
0
answers
12
views
Interpreting the time series linear regression - differences before and after collapsing data
Consider the following time series:
The coefficient on the linear regression makes sense: each additional year, the variable Y increases by 3 percentage points.
Now, the problem occurs when I'm ...
0
votes
0
answers
23
views
Weighted Least Squares -- zero mean for error makes sense, but why require zero mean for measurements?
In Introduction to Linear Algebra, Strang shows \hat{x} from equation 12 in the attached screenshot is the best linear unbiased estimator for Ax = b. One of the assumptions used to derive \hat{x} is ...
0
votes
0
answers
22
views
OLS interpretation – if x decreases
Suppose I have a multivariate linear regression where the coefficient on Beta1 equals 0.05. All independent variables as well as the dependent variable are in percent.
So the normal interpretation is:
...
2
votes
1
answer
127
views
Linear regression with convex combination of the parameters
I am looking for a method to solve the following linear regression problem:
$$
y_i=\sum_{j=1}^Kx_{ij}\beta_j+\varepsilon_i
$$
with all $\beta_j\geq0$ and $\sum \beta_j=1$. I am familiar with ...
2
votes
1
answer
54
views
How to establish relationship between regressions on subsets of data?
From classical OLS, the regression of $y\in\mathbb{R}^n$ on $X\in\mathbb{R}^{n\times k}$ yields $\beta = (X^TX)^{-1} X^Ty$. Suppose we were to partition $X$ into two blocks as: $X = \begin{pmatrix} ...
- 597
1
vote
1
answer
127
views
Asymptotic variance of linear regression with homoskedasticity assumption (Wooldridge Panel book Eq. (4.10))
Jeffrey M. Wooldridge
Econometric Analysis of Cross Section and Panel Data
Chapter 4 The Single-Equation Linear Model and OLS Estimation
Section 4.2 Asymptotic Properties of OLS
Subsection 4.2.2 ...
- 11
0
votes
0
answers
38
views
Prediction with uncertainty after least-square estimation
I've fit a model (the solution to differential equations or some other non-linear functions) to observational data to estimate the best-fitted parameters and their uncertainty by least-square methods (...
- 11
0
votes
0
answers
73
views
The Literature on the impact of outliers on ordinary least square (OLS) regression
I remembered I have encountered a paper in 1960s or 1970s that explore the impact of outliers on ordinary least square (OLS) regression. In the paper, it is shown that just adding one outlier will ...
0
votes
0
answers
12
views
If linear regression is parametric, do we need normality of the features and/or target? [duplicate]
From what I know, linear regression is a parametric model (as mentioned in here). Parametric tests requires normality of the variables. My first question is that this is an assumption of the linear ...
3
votes
1
answer
46
views
Linear regression model with a distribution over regression equations
Suppose that the observations $(y_t, x_t, k_t)_{t=1}^N$ satisfy the linear regression equation:
\begin{equation}
\begin{split}
y_t = \begin{cases}
x_t \beta + e_t & w.p. \; \theta \\
k_t \gamma + ...
- 202
2
votes
0
answers
83
views
How to manually calculate the variance of the least squares estimator in R [closed]
As stated in the title, how do you manually calculate the variance of the least squares estimator in R?
I know that the least estimates have the following formula:
$$\hat{\beta}=(X^TX)^{-1} X^T Y, $$
...
- 147
1
vote
0
answers
18
views
Independent variables affected by size, how I resolve this problem?
I am doing a research about the impact of trade unions on permanent contracts. The dependent variable is trade union presence (0,1), the independent variable is number of permanent contracts ...
5
votes
1
answer
332
views
Log of a log-transformed variable
I have been suggested to use the log of a log-transformed independent variable (i.e., log(log healthcare expenditure)). I am not sure how would this make sense. Is this a standard practice (in the ...
- 61
1
vote
0
answers
69
views
How can I estimate the sum of coefficients
I am trying to estimate the cumulative effect. When I have an ols regression with many dummies as explanatory variables, can I sum the coefficients to find the cumulative effect?
If yes, how do I find ...
- 11
0
votes
0
answers
13
views
Confidence bounds for coefficients of a fit of data set obtained with another fit
I fitted an equation to a set of data points. Then I substracted the fit previously obtained to another set of data points. After that, I fitted another equation to this new data (result of the ...
- 1
1
vote
0
answers
20
views
Geometric Interpretation of GLS with endowment of new norm
I was reading a very short passage about GLS (Generalized Least Squares Regression) provided with insufficient reference. I understand the derivation process of the BLUE $\hat{\beta_G} = (X^TV^{-1}X)^{...
- 23
0
votes
0
answers
42
views
OLS After LASSO to remove negative coefficients
I have a regression model with many predictors and not that many instances. (~70 predictors and 150 instances) I would like to use the model for inference, and therefore need to identify the sparse ...
- 203
1
vote
0
answers
53
views
Potential problem with multicollinearity?
I am trying to figure out what happens to my results if I unintentionally introduced multicollinearity.
I have an unadjusted version of this regression that includes an interaction term. The ...
0
votes
0
answers
90
views
Proc Mixed LSM standard error doesn't equal manual calculations
My experiment has 3 raw samples, and 17 derived products, made with different processing conditions. The raw samples were done in triplicate, where as the derived products were made in duplicate (i.e.,...
1
vote
0
answers
56
views
Does including an irrelevant explanatory variable have more consequences for OLS estimation results than the omission of an explanatory variable [closed]
“The inclusion of an irrelevant explanatory variable in a regression has more serious consequences for OLS estimation results than the unjustified omission of an explanatory variable.” Explain whether ...
- 11
0
votes
0
answers
107
views
What can OLS with a Box-Cox transformed dependent variable tell me?
Just to ellaborate: I’m doing an OLS-test to determine the following things:
Do my independent variables have a significant effect on the dependent variable?
What’s the direction of the effect of my ...
- 1
3
votes
1
answer
291
views
What is the relationship between linear regression and z score regression?
So I'm taking a stat's class that has introduced z-score regression. According to my professor, z-score regression gives us the "line of best fit" when the data has a linear structure.
I've ...
- 140
0
votes
0
answers
72
views
Composite Null hypothesis in a Linear Regression
Suppose that we have an OLS estimator $\hat{\beta}$.
Also, assume that we know $\hat{\beta}\sim N(\beta, \Omega)$ and have an estimator for $\Omega$, say $\hat{\Omega}$.
Here, I want to test the ...
- 755
1
vote
1
answer
44
views
standard error derivation using FWL theorem: Is this correct?
I'm reading a paper (https://scholar.google.com.br/scholar?oi=bibs&cluster=986729284887040990&btnI=1&hl=en), and there's this derivation in the article which I'm not sure is correct, so I ...
- 21
3
votes
1
answer
31
views
Accuracy of coefficients on log-linear regressiosn
Consider the following two regression models using data from the table below.
$ln(wage)=\beta_0+\beta_1 female+u$
$ln(wage)=\gamma _0+\gamma_1 male+v$
wage
female
male
10
1
0
20
1
0
30
1
0
40
1
...
- 31
1
vote
0
answers
47
views
Confidence intervals for non-negative least squares
Can we use the non-parametric bootstrapping to compute the confidence intervals for the regression coefficients estimated from non-negative least squares?
I wonder whether this has the same issues ...
- 651
1
vote
1
answer
104
views
How to interpret an index of values between -2.5 and +2.5 (an independent variable) in a regression?
I am in the proccess of writing my Master's Thesis and I'm performing a multivariate regression (OLS). One of my independent variables is Chinn-Ito Index (financial openness index) which takes values ...
- 11
2
votes
1
answer
148
views
why not using sample variance (instead of MSE) to estimate the error variance in linear regression?
Assuming the true equation for Y is linear as below:
$$Y_i =\beta_1X_i +\beta_0 + \epsilon_i$$
Assuming X is fixed, then the variance of each Y is:
$$var(Y_i )=var(\epsilon_i)=\sigma^2$$
In order to ...
- 319
1
vote
1
answer
100
views
What if error of linear regression is uncorrelated between different observations, but dependent?
One of the assumptions of classic linear regression is that error is uncorrelated between different observations. But it is obvious that uncorrelated is not independent. I have already learn the ...
- 11
0
votes
1
answer
87
views
Convexity of linear least squares problem when rank-deficient matrix
A linear least squares problem is always convex as explained mathematically here https://math.stackexchange.com/questions/483339/proof-of-convexity-of-linear-least-squares. However, a linear LS can ...
- 307
4
votes
1
answer
185
views
Handling multicollinearity with Restricted Least Squares
The dummy variable trap - including a dummy variable
for every category and including a constant term in the regression together guarantees
perfect multicollinearity - is most commonly resolved by ...
- 73
2
votes
0
answers
74
views
Question about Classification with Least Squares
According to "Statistical Pattern Recognition", from Bishop on page 184, we can solve a Classification problem of classifying $K$-Classes using a Least Squares approach. Now, this certainly ...
- 465
1
vote
1
answer
64
views
How exactly is OLS derived from assumption of normally distributed residuals?
Ordinary least square solution of linear regression can be derived from the assumption of normally distributed residuals:
$$
e_i=y_i-\hat{y_i}\\
e_i\sim N(0, \sigma^2)
$$
What I don't quite understand ...
- 21
2
votes
0
answers
36
views
Fixed effects or independent models?
Suppose we are running a model, let's say a linear regression model or a logistic regression.
Suppose also that we have gathered data for three cities: City A (4000 surveys), City B (5000 surveys) and ...
- 121
0
votes
0
answers
44
views
Resolve Lord's Paradox with principal components analysis (orthogonal distance regression)?
I've been reading Judea Pearl's description of Lord's Paradox in his 2018 book The Book of Why: The New Science of Cause and Effect, in which he presents the following plot:
In this special case ...
- 5,342
0
votes
1
answer
156
views
Prove OLS consistency
Consider the linear model
$$
Y={\underbrace{X_i}_{K\times 1 }}^\top\beta+U_i
$$
and assume
(0) There is no intercept in the model
(1) $E(X_i U_i)=0_K$ [orthogonality]
(2) $E(X_i X_i^\top)$ has rank $K$...
- 580
5
votes
1
answer
80
views
Expected length of vectors after orthogonalization
Suppose I take $k$ vectors randomly sampled from surface of unit sphere in $d$ dimensions.
$$v_1, v_2, v_3,\ldots ,v_k$$
I apply Gram-Schmidt orthogonalization (but not orthonormalization) to obtain ...
- 5,977
1
vote
0
answers
41
views
What would happen if the error term isn't independent of X in OLS?
I was reading Introduction to Statistical Learning 2nd edition chapter 3.1.2 when I came across the following sentence:
We typically assume that the error term is independent of X.
Which made me ...
- 111
1
vote
2
answers
75
views
How is zero conditional mean assumption "stronger" than uncorrelated assumption?
I am trying to understand what the zero conditional mean assumption ($\mathbb{E}[u\vert X]=0 $) encompasses in addition to a zero-correlation assumption ($\text{Corr}(X,u)=0$). I assume it must be &...
- 401
1
vote
1
answer
364
views
Dummy variable as both dependent and independent variable
I'm trying to replicate NBER's business cycle dating which consists of a binary dummy variable with 0 = expansion, 1 = recession.
The way I've done this is by taking the 6 underlying economic ...
0
votes
0
answers
42
views
How to find a decomposition of multivariate X along which y varies the most?
I'm looking for an existing algorithm which carries out the task shown in the title.
My use-case in other words: I have a set of continuous independent variables (X) and a continuous dependent ...
- 123
4
votes
2
answers
787
views
MAE vs MSE for Linear regression
Several articles says that MAE is robust to outliers but MSE is not and MSE can hamper the model if errors are too huge. My question is that MSE and MAE both are error matrices, our priority is to ...
- 41
5
votes
2
answers
350
views
Why is error of OLS not zero?
Consider ordinary least squares (OLS): We have $n$ real datapoints $\mathbf{x}\in\mathbb{R}^d$ ($d$ features) organized in an ($n \times d$) matrix $X = (\mathbf{x}_1, \dots, \mathbf{x}_n)^T$ and we ...
- 163
0
votes
0
answers
8
views
Clustering by firm id (year, country, industry dummies) control variables to tackle heterogeneity==> enough?
I’m focusing on managers characteristics (age education and narcissism) relationship with narratives reporting by firms.
(400 observations over 5 years , unbalanced , 6 countries )
Is doing the above ...
- 1
1
vote
1
answer
36
views
What is $Cov(\mu(X), X\beta^{*})$? The projection of a nonlinear function onto the linear span of the features $X$
Consider a data generating mechanism of $Y_i = \mu(X_i)+ \epsilon_i$, $\epsilon_i \sim N(0,1)$. $\mu(X)$ is a nonlinear function. Suppose $X$ is also a random variable with $E[X]=0$ and $Var[X] = \...
- 23
0
votes
0
answers
67
views
How do I represent an OLS regression model as a mathematical equation?
I have created an OLS regression model as part a pre-test/post-test experimental design with a single factor (group membership).
In order to investigate the interaction between pre-test and group ...
- 175
0
votes
0
answers
18
views
Can I use OLS techniques to analyse trends in documents from irregularly spaced time intervals
Apologies in advance as this is a lengthy query. I have a few questions involving research looking at the differences in level of ambition of language of documents on environmental issues from Group ...
- 1
0
votes
1
answer
31
views
Multivariate regressions with qualitative features
In the example from An Introduction to Statistical Learning Book about interpretations of coefficients in existence and regression with quantitative and qualitative variables (Example south, east and ...
- 69