Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

2
votes
1answer
48 views

Is constrained (nonnegative) least squares a form a regularisation?

Is nonnegative least squares already a form of regularisation? By adding a constraint that $\beta \geq 0$ (the coefficients), does it make sense to add another regularisation term as in LASSO or ridge ...
6
votes
1answer
71 views

What is the difference between $E[\varepsilon\mid X]=0$ and $E[\varepsilon X]=0$ in OLS regression?

Why is the assumption $E[\varepsilon X]=0$ weaker than $E[\varepsilon\mid X]=0$?
0
votes
1answer
145 views

Showing that the minimum-variance estimator is the OLS estimator

Recap of required theory Consider the following regression: $$y_i = \alpha + \beta x_i + u_i \tag{1}$$ where $y_i$ are iid and $x_i$ are deterministic (i.e. fixed). We know that the OLS estimator $...
1
vote
1answer
51 views

How to use White-test to check if the heteroscedasticity has been effectively dealt with by a WLS?

Consider an OLS model with $n$ observations and $p$ explanatory variables (including an intercept term) $$y=X\beta + \epsilon$$ We may use a White test to (approximately) check for the presence of ...
2
votes
1answer
63 views

Transform data while keeping the mean constant

I am using an OLS regression to fit a model to some data. The estimated response is given by the usual $$\mathbf{\hat{y}} = \mathbf{Py} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}...
5
votes
1answer
309 views

Sufficient Statistic for $\beta$ in OLS

I have the classical regression model $$y = \beta X + \epsilon$$ $$\epsilon \sim N(0, \sigma^2)$$ where $X$ is taken to be fixed (not random), and $\hat\beta$ is the OLS estimate for $\beta$. It is ...
0
votes
1answer
91 views

Estimating the variance of the GLS estimator

Consider the linear regression model where $y = XB + u$. Assume that $\mathrm{E}[u \mid X] = 0$. Assume that $\mathrm{V}[u \mid X] = \sigma^2I$. This is the simple linear model. Now relax the ...
2
votes
0answers
167 views

Leave-one-out cross validation for a linear model through the origin

An Introduction to Statistical Learning by James et al. defines the leave-one-out cross-validation estimate for least-squares as: $$CV_{(n)}=\frac{1}{n}\sum_{i=1}^n \biggr(\frac{y_i-\hat y_i}{1-h_i}\...
2
votes
0answers
52 views

Linear Prediction and Linearity of CEF

I am revisiting the basic notions of linear regression and stumbled upon the following idea in Cameron and Trivedi's Microeconometrics book: However, for the conditional mean to be linear in x, so ...
0
votes
1answer
28 views

Is there a systematic reason why a model trained on a subset of data does better out-of-sample than the same model trained on the full dataset?

I trained a linear regression model using 3000 data points. (OLS regression, no regularization.) Then I trained another model with the same predictors (about 25), but with a subset ($n=700$) of the ...
3
votes
1answer
63 views

Why is individual R-squared higher than overall R-squared?

I ran a ridge regression model on a set of data across 6 groups. As you can see, the overall R-squared is low. Because groups A and B make up the most of the data, I would think the overall R-...
1
vote
0answers
54 views

What statistics can I use to compare OLS to an ordered probit

I am trying to justify to the use of an ordered probit, my dependent variable is a survey response on a likert scale so is likely ordinal, but I wanted to provide a goodness of fit stat to back up my ...
0
votes
0answers
19 views

Interpretation of Economic Significance of Coefficients in Linear Probability Model and Reversing the Dependent Variable

I have the following regression: open_regime = B1 GovernmentSupport + B2 ln(R&D Budget) + other_controls + FirmFE + TimeFE + ProjectCategoryFE where ...
5
votes
2answers
139 views

calculating the p-values in a constrained least squares

I have been using Matlab to perform unconstrained least squares (ordinary least squares) and it automatically outputs the coefficients, test statistic and the p-values. My question is, upon ...
0
votes
2answers
115 views

ANOVA multicollinearity adjustment

I am using the statsmodel.ols module to compute an omnibus (ANOVA) F-test for three within-subjects factors; 2*3*2 levels design. The Cond. No. of the omnibus test (...
1
vote
1answer
26 views

Regression doubt on ols estimator

I understand that beta-1 and beta-2 are parameters/OLS Estimators. But why is the variance of the error term a parameter?
1
vote
0answers
113 views

Fast way to obtain SSR (Sum of Squares residuals) from QR in least square model?

I am using a linear regression, yet the only output I need is the Sum of Squared Residuals (SSR), I don't care about the coefficients. (Context is a non-linear LS, which is linear given an extra ...
1
vote
2answers
67 views

Linear model with two columns where one column is a linear transformation of another

If we have a linear model $y_t = β_1x_{1t} + β_2x_{2t} + e_t$ where the errors $e_t$ are i.i.d normally distributed: $e_t$ ~ $IIN(0, σ^2)$, t = 1,...,T. The vectors of random regressors $X_1$ and $...
2
votes
1answer
47 views

In OLS, while using log-log and linear-log transforamtions, is valid to transform some regressors only?

In OLS I was wondering if it is valid to log-transform some regressors only. Specifically, continuous regressors, because it is advised not to transform binary or categorical variables. For instance, ...
4
votes
1answer
183 views

VIF Drops Significantly When I Delete Some Dummy Variables

Is my model valid even with the high VIF? Does it matter which dummy variable I drop as the reference point? I have a a category variable (Fruit) that I converted ...
1
vote
2answers
70 views

Variance of $\hat{\beta}$ when assuming heteroskedastic error terms

I am wondering why, when we are assuming heteroskedastic error terms: $Var(\hat{\beta}|X) = (X'X)^{-1}X'E[\varepsilon \varepsilon'|X] X(X'X)^{-1}$ simplifies to $Var(\hat{\beta}|X) = (\sum{x_ix_i'...
2
votes
0answers
89 views

Robust Regression in MATLAB's robustfit: what is the optimal weight function to tackle heteroskedasticity?

I'm currently performing a linear regression analysis and encountered a fair amount of heteroskedasticity. Increases in predicted values go along with decreases in residual variance. Otherwise, the ...
0
votes
0answers
58 views

Stuck on a term in $\operatorname{Var}\left[ \widehat{\beta}_0 \right]$ proof

So I was trying to prove that $\operatorname{Var}[\hat{\beta}_0] = \dfrac{\sigma^2n^{-1} \sum{(x_i)^2}}{\sum{(x_i-\bar{x}})^2}$ And I got stuck with the part $\dfrac{-2\bar{x}}{\sum{(x_i-\bar{x})^2}} ...
0
votes
0answers
10 views

Determining the Relationship Between Monte Carlo Breaks and Model Volatility

I'm looking for a statistical test to understand the relationship (if any) between the model volatilities of a stochastic process, and the occurrence of a'break', defined as an instance when an ...
0
votes
1answer
78 views

Expected value of squared least squares estimator

I am trying to prove $E(\hat{\beta} '\hat{\beta}) = \beta'\beta+\sigma^2 *\sum_{k=1}^K\lambda_k^{-1}$ where $\lambda_k$ denotes the eigenvalues of the matrix $(X'X)$ with dimensions $K\times K$. $\...
0
votes
0answers
27 views

Regression equation passing through the origin

I'm going through Multiple Choice Questions of Basic Econometrics by Gujarati. There is a question which states that: It is a simple two-variable regression: Any regression equation written in its ...
1
vote
0answers
33 views

OLS R-Squared Drops by 15% When Constant / Intercept is Removed?

I ran an Ordinary Least Squares model and found the constant / intercept is the more significant than all the other features. When constant is included, the R-squared is 45%, when I remove the ...
1
vote
1answer
374 views

How to find the OLS estimator of variance of error

Given the Linear Regression model $y=X\beta+\epsilon$, where $\epsilon \sim D(0_n,\sigma^2 I_n)$ and $D$ is some distribution. How to find the OLS estimator of $\sigma^2$. I know that the sum of ...
1
vote
0answers
366 views

Proof Verification: $\tilde{\beta_1}$ is an unbiased estimator of $\beta_1$ obtained by assuming intercept is zero

Consider the standard simple regression model $y= \beta_o + \beta_1 x +u$ under the Gauss-Markov Assumptions SLR.1 through SLR.5. Let $\tilde{\beta_1}$ be the estimator for $\beta_1$ obtained by ...
1
vote
0answers
19 views

Using fitted lagged variable as dependent variable in a new regression?

Suppose I have regression like this; $y[t] = a * exp(b*y[t-1])$ From this regression, I get; $\bar y = y - residuals$ What happens if I regress a new regression like this? $y[t] = c * exp(d*ybar[t-...
2
votes
2answers
41 views

Intuitive explanation from regression coefficient estimate formula

Can someone provide an intuitive explanation of why the OLS regression estimate, of y=a+bx, b have the form b=cov(x,y)/V(x). How intuitively are the covariance and variance related in this?
1
vote
1answer
36 views

Assumptions of linear fit; linearity and homoscedasticity

I'm reading about the assumptions of taking a linear fit between two variables from here, and that source says: For diagnosing non-linearity: nonlinearity is usually most evident in a plot of ...
1
vote
1answer
40 views

Unbiasedness and Consistency of DID estimator - pooled cross sections over time

Consider two time periods, where In time-period 1: a random sample is collected for group 1 (control) and a random sample is collected for group 2 (treated). In time-period 2: a random sample is ...
0
votes
1answer
47 views

What is the best way to test and validate a multivariate regression using OLS?

I am implementing a multivariate regression from scratch using Ordinary Least Squares to get the weights. I noticed that this method does not have any hyperparameters to tweak, so I am not sure what I ...
1
vote
0answers
192 views

Expected value of the residuals

How would one prove that the expected value of the residuals from OLS regression is zero? I will make two cases. In the first case I treat $X_i$ as random and in the second case I treat it is non-...
0
votes
0answers
29 views

ANOVA tables are exactly the same using OLS and Poisson regression?

I have fit two models in R, as follows: m1 = lm(y ~ x * z) m2 = glm(y ~ x * z, family="poisson") Where y is the dependent ...
0
votes
1answer
51 views

Degrees of freedom correction in estimation of AR(p) process

Assume that I have a process $y_t$ such that $$y_t = c + \phi_1 y_{t-1} + \ldots + \phi_p y_{t-p} + u_t$$ where $u_t$ is i.i.d. white noise such that $E[u_t] = 0, \forall t$ and $E[u_t u_s]$ is equal ...
1
vote
0answers
38 views

Is fixed effects (within estimator) estimated using OLS?

In a paper I submitted to a journal, I estimated a fixed effects panel model with regional fixed effects. The reviewer of my paper in his comments said my estimation was bad because I "simply used OLS ...
2
votes
1answer
38 views

Does regularization in regression help with numerics when the data matrix is not full rank?

I am trying to get some intuition around regression when the data matrix $A$ is not full rank in the following regression/least squares problem: $$y=Ax+b$$ where $y \in \mathbb{R}^n$, $A \in \mathbb{...
1
vote
1answer
31 views

Decomposition of vector into product of a function on a matrix and a function on a vector - Possible? [closed]

Say I have access to $N$-dim vector $Y$, $N \times p$ matrix $X$, and $q$-dim vector $Z$. Ultimately, I would like to learn the functions $g,f$ in: $\underset{N\times1}{\underbrace{Y}}=\underset{N\...
1
vote
0answers
24 views

Long T Small N causal time series ols regression analysis ? can i use panel data? [closed]

I have N=28 AND T=260 WEEKLY DATA and i want to check impact of two variables on stock volatility.i proposed that one variable moderates others effect on stock volatility . I was using time series ...
3
votes
3answers
101 views

Finding OLS estimator for $\beta$ where $y_i=\beta+ 2 \beta x_i+\epsilon_i$

Consider the following model with the usual OLS assumptions: $\epsilon_i$ are uncorrelated random variables with mean zero and constant variance $\sigma^2$. $$y_i=\beta+ 2 \beta x_i+\epsilon_i$$ $(...
1
vote
0answers
19 views

Normality assumption for t/z tests

I have what is probably a silly question regarding normality assumptions for t/Z tests. As I understand, t/z tests require that sample data was obtained from populations following a normal ...
2
votes
1answer
44 views

Show that solution to cubic smoothing spline reduces to regular least squares minimization as $\lambda$ approaches infinity

I am asked to show that the solution to a smoothing splines problem of the form $$ \text{PRSS}(f,\lambda) = \sum_{i=1}^N\left[y_i-f(x_i)\right]^2 + \lambda \int f''(t)^2 dt, $$ with $$ f(x) = \sum_{j=...
0
votes
0answers
17 views

R/S-Analysis: Hurst Coefficient Using Least Squares Method

The application I am working on (it is an image processing program) needs to calculate, given an m-by-m matrix of integers, the so-called Hurst coefficient for that matrix, considering it as a time-...
1
vote
1answer
57 views

Role of random sample assumption in consistency of OLS estimator

I guess in part what this all amounts to is what does the assumption {(x_i,y_i) : i=1,2,...,n} being i.i.d. imply about the i.i.d-ness of functions of it? I am confused because for example I have ...
0
votes
0answers
14 views

Measurement error in one regressor amongst k regressors

Above I have set out the assumptions and a true claim made by Wooldridge (2002). I am having trouble proving that claim and was wondering if anyone could provide a step-by-step solution of the claim. ...
0
votes
0answers
66 views

In LLSE $e_i$ are orthogonal random variables

If we have a sequence of zero-mean random variables $(y_0,y_1,...,y_{i-1})$ and $$e_i =y_i-\hat{y}_{i|i-1},\ \ \&\ \ \ \hat{y}_{0|-1}=0 $$ $\hat{y}_{i|i-1}=$ LLSE (Linear least square estimate) ...
0
votes
2answers
29 views

Linear Least square estimate of $x^3$ given $x$ and the moments

I have been struggling to find a direction on how to proceed with the following problem. Given that $x$ is a zero mean (non-Gaussian) random variable with moments E$(x^n)=\mu_n$. I need to find the ...
0
votes
1answer
81 views

Calculation of intercept in multiple linear regression (OLS)

While researching OLS, I found out the equation to calculate coefficients as: $$ \beta = (X^\top X)^{-1}X^\top y $$ (Ref: https://en.wikipedia.org/wiki/Linear_least_squares) However it does not ...