Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

2
votes
0answers
37 views

Kernel and regularization parameter of James–Stein estimator

Consider a FIR model of the form $y= Ug_0+e$ with $e$ white noise with variance $\sigma^2$. We assume that we have collected N input-output measurements $y$ and $U$. The James–Stein estimator is ...
1
vote
0answers
47 views

Minimal Sufficient Statistic for Gaussians with different means

I have the following problems on my Statistics course (using Casella and Berger's book) problem set: 1) Let $Y_{i} = X_{i}'\theta + U_{i}$ where $\theta \in \mathbb{R}^k$ and $U_{i}$ are iid $N(0,...
0
votes
0answers
19 views

Estimation procedure for VAR(P) to specific model ie VAR(2)

I am having trouble linking the least squares estimation method of a generalized VAR(P) to a specific VAR model. So this is what I know so far: Y=(Y1,......Yt) KT B=(V,A1,.....Ap) K(KP+1) Zt= (...
0
votes
0answers
13 views

Reducing Least Squares Regression Error in Predictive Model

I have been iterating on a predictive modeling framework for my marketing team to predict the output of leads we can expect for each day remaining in the current month. I started the process by ...
0
votes
0answers
13 views

Significant test for comparison of two (or more) OLS models

How may I compare two (or more) possibly not-nested OLS models that I may say one is significantly better than the other? Were they nested, I know I might use some kind of F-test. To select not the ...
0
votes
0answers
29 views

Estimating price elasticity where prices are determined by online seller

I have a dataset that contains information about purchases of different items at different times with different prices. I want to calculate both price and cross-price elasticity for each item. So in ...
1
vote
2answers
114 views
3
votes
3answers
127 views

Why shrinkage estimators?

Iam trying to understand the usage of lasso and ridge regression. The advantage of both methods is that we get a lower variance in comparisson to the ols estimation and thus we get a better prediction....
3
votes
1answer
257 views

Why don’t we need strict exogeneity for OLS consistency? [closed]

I know how to show that OLS only requires orthogonality between regressor and error for consistency, so the title is maybe a misnomer (couldn’t think of a better one) But consider the following ...
4
votes
1answer
105 views

Coefficient of determination invariant to centering and rescaling of variables

Can someone provide a proof why the coefficient of determination given by $$R^2:=1-\dfrac{||y-\hat{y}||_2^2}{||y-\overline{y}||_2^2}$$ in a multiple linear regression setting $$y=X\beta+\epsilon$$ ...
0
votes
0answers
52 views

Derive the bias and variance for linear regression with zero intercept

I am to derive the bias and variance for the following linear regression with zero intercept using least squares. $y_t = at +\epsilon_t$. I have already found that $\hat{a}=\frac{\sum_{t=1}^ny_tt}{\...
1
vote
0answers
32 views

Heteroscedasticity and weighted least square estimator

"In presence of heteroscedasticity, OLS estimators are unbiased but inefficient" Showing the unbiased part is relatively easy. Some authors have explained the inefficiency with the help of new ...
2
votes
0answers
16 views

How to derive a recursive version of a regularised cost function

I am to derive a recursive version of the following cost function and examine for which choice of D can we have a estimator windup $V(\theta) = \frac{1}{2}\sum_{t=1}^n(y(t)-\phi(t)^T\theta)^2 + \...
0
votes
1answer
67 views

Multiple values in a variable for linear regression

Say I have a data set that I am trying to perform a linear least squares regression on. Suppose that the end goal is to predict y from x. The training data set I am working with has the form ...
0
votes
0answers
74 views

Asymptotic distribution of beta in ols

I have a proof of the asymptotic distribution of the OLS estimator but I don't understand how we end up with the highlighted part of the following screenshot:
0
votes
0answers
16 views

What can we learn from residual of least squares?

problem description Consider the following simple least square matrix problem $$ AX=B$$ where $A,X,B$ are all matrices. Clearly the solution to $$\min_{X} \lVert AX-B \rVert_F$$ would undoubtfully ...
0
votes
1answer
43 views

Is the multiple correlation coefficient the correlation between $y$ and $\hat y$?

Wikipedia states [Multiple correlation] is the correlation between the variable's values and the best predictions that can be computed linearly from the predictive variables. But lecture notes ...
1
vote
1answer
69 views

Differentiation of RSS

I have the equation $RSS\left ( \beta \right )=\left ( y-X\beta \right )^{T}\left ( y-X\beta \right )$ that I'm trying to differentiate w.r.t. $\beta$. I used chain rule to get to $\frac{\partial ...
0
votes
0answers
44 views

How to solve a well fitted model - Model Misspecification

I am currently writing a paper, analysing the impact of goldprice movements on the capital structure of gold mining firms. My basic model is a simple OLS model with (y=leverage and x=ln(goldprice)). ...
0
votes
0answers
18 views

Why does the vectorization of the sum of variance-weighed least square equal $R^{T}\Sigma R$?

The sum of the variance-weighed least square errors of $n$ independent observations is given by $$\sum_{i=1}^{n}\frac{(y_i-\hat{y}_i)^2}{\sigma_i^2}$$ where $\begin{cases} y_i \mathrm{\,\,is\,\,the\,\...
2
votes
1answer
82 views

How does average treatment effect relate to averages of individual treatment effects? [closed]

I have a balanced panel with two groups, A and B, and run a standard fixed effects regression. I want to derive how the $\beta$ coefficients of these two regressions are related: \begin{eqnarray} y_{...
0
votes
0answers
14 views

Generalised superior ADF test misspecification

If I run an GSADF test of Phillips et al. (2015) to identify multiple unit-root periods, what I'm doing is basically hypothesis testing a recursive regression such as the following against the null $...
2
votes
1answer
61 views

Least Squares with respect to a Matrix

There is a nice geometric interpretation of ordinary least squares where $\beta \in\mathbb{R}^p$ is to be learned/ estimated from data given by $y \in\mathbb{R}^n$ and $x \in\mathbb{R}^{n\times p}$ ...
1
vote
0answers
20 views

Standard errors of OLS estimate if regressor is a stochast?

Assume the model classical linear regression model (with for simplicity only one regressor) $$y=X\beta +u,$$ with $u$, $X$ independent, and $\operatorname{Var}(u|X)=\sigma^2I_n$. Assume for ...
3
votes
2answers
136 views

Adding a Constant to Every Column of X (OLS)

In OLS, if I have design matrix X (an NxK matrix of full column rank) and I add a constant, such as 2, to every entry of X, how does that change my estimators? Let's denote $\tilde{X} = X + 2$. I ...
4
votes
0answers
240 views

Is the OLS estimator the UMVUE (assuming Normality)?

Suppose $$ \mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e} \, , \\ \mathbf{e} \sim \mathcal{N}(0,\mathbf{I}_P) \, . $$ We know that $\mathbf{\hat{b}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \...
0
votes
0answers
21 views

Creating dummies for panel data in OLS

I have a panel data for 12 countries ( 6 developed and 6 developing) and 10 years. my dependent variable is Exchange rate, while the independent variables are GDP, Openness of economy, Inflation, ...
0
votes
0answers
48 views

Principal Component Regression and its relation to linear regression

I am reading Elements of Statistical Learning. On page 79 it is stated that the principal component regression is defined as: $\hat{\bf{y}}_{(M)}^{\text{pcr}}=\bar{y}{\bf{1}}+\sum_{m=1}^M \hat{\theta}...
0
votes
0answers
50 views

Why is $\mathbb{E}\left[\frac{1}{n-2}\sum\limits_{i=1}^n e_i^2\right] = \sigma^2$?

The context that we are presented with a linear model $$Y_i = \beta_0 + \beta_1X_i + \epsilon_i,$$ where $\epsilon_i \texttt{~} \mathcal{N}(0,\sigma^2)$. We obtain predictions for $X_i$ by plugging ...
3
votes
0answers
32 views

Intepretting Linear Mixed Effects Model Output

I'm trying to reconcile the output of this MixedLM Model Output with my knowledge of OLS model outputs, e.g., MixedLM has Z-Stats vs. T-Stats for OLS. Model code for reference, but this is not my ...
1
vote
1answer
66 views

Obtaining the possible least squares solutions when $X^TX$ is not invertible

If $X^TX$ is not invertible, what is the set of solutions for the least squares estimator $\hat{\beta_1}$ in the below? $Y_i = \beta_0 +\beta_1(x_i-\bar{x}) +\epsilon_i$ I got as far as writing out ...
1
vote
1answer
60 views

Cointegration regression for stationary series

I have seven variables that are stationary at first difference. I am trying to figure out how variable X influences variable Y, conditional on Z, W, A, B, and C. Which would be more appropriate to ...
0
votes
0answers
32 views

Interpretation of Coefficients in Semilogarithmic Interaction Regression Model

The linear model for consideration is a binary interaction multiple regression model using OLS. The dependent variable is household net worth transformed with the inverse hyperbolic sine ...
1
vote
0answers
19 views

Questions about Endogeneity and IVs

I am doing the returns of education on earnings. I have education (number of years of edu) as my endogenous variable and I am using Father Education as an instrument. I am using SAS and I ran a 2sls. ...
0
votes
0answers
8 views

How to Observe OLS Effects Across Time Spans?

Forgive me if this should be in the web chat, I don't have the reputation yet. I have a data set that spans across roughly 10 years: ...
0
votes
1answer
109 views

Linear Regression: Why variance of β is high when $X^TX$ is singular

I know how to derive β when $X^TX$ has an inverse and on that condition, β is an unbiased estimate of β* with mean 0 and β* and variance $σ^2(X^TX)^{-1}$. But why the variance will become very high ...
0
votes
1answer
242 views

What is the general guideline for dropping dummy variables in a regression model?

For an ordinary least square (OLS) regression problem of the form $y = \beta + w_1 x_1 + \ldots + w_n x_n$, let's say I have 3 categorical variables. gender: male, female type: office, field, manager ...
0
votes
0answers
29 views

Show $T_0^2$ (t-statistic) is F-statistic

I am trying to show that $T_0^2 = F$ (but I just can't get it), where \begin{equation} T_0^2 = \left(\frac{\hat{\beta}}{\sqrt{\frac{\hat{\sigma}^2}{\sum_{t=1}^n (x_t-\bar{x})^2}}}\right)^2 = \frac{\...
1
vote
0answers
24 views

OLS difference between individual level and aggregate level estimates

I am new here, so please accept my apologies if this does not belong here. My question essentially concerns what appears to me a special case of the ecological fallacy. Consider a simple model that ...
2
votes
0answers
63 views

Least Squares fit of model - R

The data file (X in code thread below) contains the record of monthly data X[t] over a twenty year period. The data can be modelled by X[12j+i] = Mu + s[i] + Y[12j+i] where (i=1,...,12; j=1,...,k) ...
3
votes
0answers
35 views

Direction of Bias in OLS with systematic measurement error

I am looking at a study that wants to measure the effect of $x_t$ on $Y_t$, but the true values of the $x_t$ are not observed. Instead, what is observed is a minimum value for $x_t$. In effect, what ...
0
votes
2answers
96 views

Expression for $\hat{\beta}$ in simple linear regression

For simple linear regression, I have in my notes that $\hat{\beta} = \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sum(x_i-\bar{x})^2}=\frac{\sum(x_i-\bar{x})y_i}{\sum(x_i-\bar{x})^2}$ (was intended as a ...
0
votes
1answer
67 views

For OLS to be unbiased, do we need $x_i$ to be uncorrelated with $\epsilon_i$ or with $\epsilon_s$ for all $s$?

In some textbooks I've read, it is said that an assumption for OLS to be unbiased in the standard cross-sectional model $y_i=\alpha + \beta \cdot x_i +\epsilon_i$, we can use the assumption $E(\...
0
votes
0answers
72 views

What is the interpretation of the coefficient on an endogenous variable being higher in an TSLS regression compared to an OLS regression?

I ran an OLS regression with a suspected endogenous variable, obtained a coefficient of 14.06. I then obtained a instrument that has strong correlation with the endogenous variable, but questionable ...
16
votes
3answers
3k views

Why not use the “normal equations” to find simple least squares coefficients?

I saw this list here and couldn't believe there were so many ways to solve least squares. The "normal equations" on Wikipedia seemed to be a fairly straight forward way: $$ {\displaystyle {\begin{...
44
votes
3answers
3k views

Where does the misconception that Y must be normally distributed come from?

Seemingly reputable sources claim that the dependent variable must be normally distributed: Model assumptions: $Y$ is normally distributed, errors are normally distributed, $e_i \sim N(0,\sigma^2)...
0
votes
0answers
63 views

Nonlinear regression: improving parameter estimates

I'm running a nonlinear regression to estimate $\delta$ and $\alpha$ using the following model, where $X$, $Y$ and $Z$ are the variables: \begin{equation} Z = \left(\delta X^\alpha+(1-\delta)Y^\alpha\...
0
votes
0answers
72 views

ARDL bounds testing cointegration

I have a data set that contains one non-stationary variables that is stationary in first differences. Further, there are several other stationary variables. Since cointegration is not possible if ...
1
vote
1answer
37 views

OLS regression - sum of squared residuals?

I am studying an introductory econometrics course. $$ \Delta Z_t = \alpha+ \beta t + \gamma Z_{t-1} + e_t $$ Where $e_t \sim I(0)$ with $E(e_t)=0$ and $E(e^2_t)=\sigma_e^2$ My question: is $\...
0
votes
0answers
126 views

Parameter uncertainty for least-squares minimization

If I minimize the least squares error of a fitted function to find the values of the parameters, how does the Hessian matrix relate to the variance of the parameters? In my use case, I try using the ...