Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

What is an example to show Wald's test does not confirm significance at the same level for each variable?

Suppose that the true model is given by Y=0.3+X1+X2+X3. Assume we have 100 training examples where each covariate vector (X1, X2, X3) is randomly drawn from some distribution P and Y is generated ...
0
votes
1answer
7 views

Does endogeneity problem matter when proving the existence of association/causality relationship?

In social science field (particularly Finance and Operations Management), we usually need to prove or disprove hypotheses of the type: X are positively associated with Y. One of the typical method to ...
1
vote
0answers
8 views

What statistical test is most appropriate when my data consist of multiple series, each based on an individual sample?

I'm trying to determine the effect of an interferent $X$ on the measurement of a substance $Y$. Ultimately, I'm looking to predict $Y_{actual}$ within a confidence interval, given $Y_{observed}$ and a ...
0
votes
0answers
14 views

How do the Gauss-Markov Assumptions relate to the OLS Properties?

How do the 5 Gauss-Markov Assumptions below relate to the OLS properties of Biasedness, Consistency and Efficiency if they are violated?
1
vote
0answers
16 views

Can I use subset of a time series if OLS assumptions are failing?

My OLS model build on full data has heteroscedasticity of residuals. Is it allowed to use a subset of the time series to get around this issue? What are the possible implications?
0
votes
0answers
10 views

Calculating a formula for the asymptotic variance of a function of parameters?

Suppose I have a model $y=\beta_1+\beta_2x + u$ with an estimator $\hat{y}=x\hat{\beta}$. Also suppose I want to determine the average of 3 predictions of $\hat{y}$, called $\hat{y^*}$, and then ...
2
votes
2answers
26 views

Perfect multicollinearity with a cubic term in the model?

I'm trying to figure out why adding a cubic term in the model doesn't guarantee a perfect multicollinearity. If $X$ is known, then $X^3$ is known in both magnitude and sign and vice versa. It may not ...
2
votes
1answer
76 views

Question regarding the distribution of OLS estimators

In linear regression: Why does the sampling distribution of OLS estimators depend on the underlying distribution of errors? Thanks! (I am relatively new to statistics in general, so more simply ...
2
votes
0answers
23 views

FGLS vs OLS tradeoff

Hi, Can someone tell me what is the tradeoff between using OLS and FGLS? I know the conditions under which FGLS is more efficient than OLS. But what about robustness? It is usually said that the ...
0
votes
0answers
17 views

Estimating a logistic regression with OLS? [duplicate]

NB: This question is different from this one which assumes that we have computed the LHS of the regression equation with no issue. My question is about how to compute this LHS. Consider a simple ...
2
votes
1answer
23 views

2SLS and Control Functions (identity of estimator)

I would like to show the identity of the 2 Stage Least Squares estimator and the control function estimator. Assume a linear regression model $$y = X\beta + u$$ where $X =[X_1 \ X_2]$ is $n \times ...
0
votes
0answers
9 views

OLS and 2SLS normal equations

For a system of equations with $M=2$ endogenous variables, $ Y=\begin{bmatrix} y_1 & y_2 \end{bmatrix}$ and $K=3$ exogenous variables, $X=\begin{bmatrix} x_1 & x_2 & x_3 \end{bmatrix}$. ...
0
votes
0answers
13 views

Proof: intercept term (b0) becomes zero when both coefficients and covariates are centered [closed]

Trying to understand how in OLS, the intercept b0 term becomes zero when both the coefficients and covariates are centered (subtracting the sample mean).
0
votes
0answers
7 views

Can I use multiple regression (OLS) when I have panel and cross-section data as independent and dependent variables?

I am trying to analyse the impact of historical election results on current developmental performance, meaning I have panel data for various independent variables measured in an interval scale. I want ...
0
votes
0answers
28 views

OLS Time series regression with levels and first differences

I am currently working on my bachelors thesis and I am trying to perform an OLS time-series regression with the short-term interest rate as dependent and inflation expectations and an output-gap as ...
0
votes
0answers
26 views

Optimal value for multiple input

I run an experiment in which every second I record values of area, circularity, and elongation (there will be probably more variables in the future). I want to find in which second there are the ...
2
votes
0answers
25 views

Does $\operatorname{Var} (\varepsilon_i | X_i) = \operatorname{Var} (\varepsilon_j | X_j)$ follow from the first two assumptions of OLS estimator?

Let $$X_i=\begin{pmatrix} X_{i_1} \\ X_{i_2} \\ \vdots \\ X_{i_K} \\ \end{pmatrix}, \qquad \beta=\begin{pmatrix} \beta_1 \\ \beta_2 \\ \vdots \\ \beta_K\\ \end{pmatrix}$$ I'm learning OLS estimator'...
0
votes
0answers
5 views

Can class imbalance decrease the value of a coefficient of a dummy variable?

I modeled monthly electricity consumption as a function of summer and other regressors. summer is a dichotomous variable that is ...
1
vote
1answer
23 views

Is modeling location as a categorical variable in an OLS regression considered a fixed effect model?

My thesis advisor recommended me to model the monthly electricity consumption of households as a function of their state and multiple other regressors. In R, I turned the state variable into a ...
2
votes
2answers
46 views

Why can't we always have the zero-mean-condition assumption in linear regression?

I've read Zero conditional mean assumption (how can in not hold?). In linear regression, we assume the model follows $$y_i = \beta_0 + \beta_1 x_i + \epsilon_i $$ under the assumption that $\mathbb{...
2
votes
2answers
83 views

In an OLS regression, will excluding all data for a non-reference category of a dummy variable impact the other dummy level categories?

Say I have an OLS regression with a dummy variable level A, B, C and D, where A is the reference category. Will the estimated coefficient value and/or statistical significance of B or C change or be ...
0
votes
0answers
23 views

Can I transform a few features to polynomial in multi regression?

Let's say I have 3 features in my ols model as below. model = sm.ols(formula='y ~ a + b + c', data=df) Question 1. If I want to transform the feature 'a' as ...
0
votes
0answers
21 views

AR(1) process can be estimated using linear regression

Can the $AR(1)$ process represented as $$ x_t= ax_{t-1}+\epsilon_t$$ be estimated by regressing $x_t$ on its lagged value $x_{t-1}$.
1
vote
1answer
24 views

What is the process of obtaining Var(βhat) in simple linear regression?

I have just started statistics and we have used the estimation strategy OLS to obtain the parameter estimate of the independent variable for a simple linear regression model. As I understand it, my ...
1
vote
0answers
37 views

Neural Net Regression SSE Loss

Notation $y_i$ is observation $i$ of some response variable $Y$. $\hat{y}_i$ is the value of $y_i$ predicted by the regression. $\bar{y}$ is the average of all observations of the response variable....
0
votes
0answers
42 views

Unbiasedness of OLS estimator

Consider a very simple linear regression model with following assumptions: 1) No assumptions on how $x_i$ is generated (assume random design, not necessarily IID); 2) $\mathbb{E}[\epsilon_i|x_i]=0$, ...
1
vote
2answers
51 views

Should the existence of a closed-form solution inform the choice of robust regression method?

Suppose one has a linear least squares problem of the form \begin{align} \xi^* = \textrm{arg min} \ \sum_{i = 0}^n \left \lvert \ {\bf v}^T({\bf x}_i) \ \xi - c({\bf x}_i) \ \right \rvert^{\ 2} \in ...
0
votes
2answers
44 views

Relationship between SSE with GLS and OLS

I have been trying to derive if there is any relationship between the sum of squared residuals (SSE) from a model estimated with GLS, and the same model estimated with OLS. Professor Chung-Ming Kuan, ...
0
votes
0answers
20 views

Unbiased estimate of variance for 1d linear regression

I'm trying to show for the simple 1-D linear regression case that the estimate $\frac{1}{n-2}\sum_{i=1}^n \hat{\epsilon_i}^2$ is an unbiased estimate of $\sigma^2$, which is defined as the variance of ...
0
votes
0answers
30 views

LASSO Regression with noise

I know LASSO regression is useful to exclude redundant features, so can it be useful when you have noisy data? I explain better with this example: Suppose I generated a data set using an equation (e....
0
votes
0answers
31 views

confidence contours for linear model with multiple dependent variables

A book I have on regression analysis describes a technique for determining confidence contours of the parameters of a linear model $$ Y^{\textrm{model}} = f(\boldsymbol{x}, \boldsymbol{\theta}) = \...
0
votes
1answer
37 views

Graphical representation of Eigenvalue and EigenVector in Least Squares solution

Trying to understand the role eigenvalues and vectors play in least squares solutions Reading answers to this post, Relation between best fit line and eigenvector of maximum eigen value of an ...
0
votes
0answers
13 views

Coefficient sign flips after applying Fixed Effects

Good day everybody, I am currently writing my master thesis and I research whether family owned companies (FFF=1 if family owned, 0 otherwise) have a positive or negative effect on the performance of ...
1
vote
1answer
23 views

Test for difference in coefficients: same sample, same outcome, but different explanatory variable

I do Ordinary Least Squares regressions and want to test if the difference between two estimated coefficients is statistically significant. I use the same sample, the same outcome, and only the ...
1
vote
0answers
51 views

Can I add I(0) regressor to a regession of cointegrated variables I(1)?

My dependent variable, Y, is I(1) and is co-integrated with an I(1) independent variable X1. I understand I can estimate the regression Y~ X1 with OLS. Now can I include another I(0) independent ...
0
votes
0answers
18 views

Two different ways to write variance of OLS beta

I know that for OLS, we can write $var(\hat{\beta}) = \sigma^2 (X^{T}X)^{-1}$. Then for a the last variable $p$, we have $var(\hat{\beta}_p) = \frac{\sigma^2}{\langle x_p, x_p \rangle}$. However, we ...
2
votes
0answers
45 views

Problem with calculating a confidence interval

I'm working on Exercise 3.2 from Elements of Statistical Learning. It asks to find a $95\%$ confidence interval for a linear regression prediction (ordinary least squares are used) using two different ...
2
votes
1answer
41 views

What to do when error term are autocorrelated?

For my master thesis I estimated a linear regression. Durbin-Watson test shows, that the error terms are autocorrelated and thus, the estimator could be biased. So, what to do now? I found the ...
0
votes
0answers
19 views

How to check Gauß-Markov theorem after OLS estimation?

I have estimate a simple bivariate regression with OLS and want to proof if the estimator is unbiased. I found that the Gauß-Markov theorem consists of 4 Assumptions and the first 3 can be rewrite as ...
3
votes
1answer
41 views

MLE when variance of residuals is null (y is a linear combination of x)

Suppose I have the following model to be estimated via MLE assuming normal errors $y_{t}=x_{t} \beta +e_{t}$ with $e=N(0,\sigma^{2})$, where $y, x$ are matrixes and $\beta$ is a vector, so $\sigma^{2}$...
0
votes
0answers
12 views

Using least squares to estimate variance of latent variable

I am having trouble understanding why I can't use least squares to solve an overdetermined system of linear equations using $\bf{x} = (\bf{A}'\bf{A})^{-1} \bf{A}'\bf{b}$. The same model estimated ...
3
votes
1answer
96 views

Sparse linear poorly constrained least-squares problem

I have a somewhat simple linear problem. I have data $D$ (a vector with a few million elements), the parameter vector $X$ (a couple of thousands elements) and the design matrix $A$ which is extremely ...
0
votes
0answers
23 views

Gauss-Markov Theorem Verification

Can someone propose a simple test to verify Gauss-Markov theorem? Preferably, with Python. It tried to generate sample of Ridge regression and OLS estimates for data generated from linear function ...
1
vote
1answer
33 views

Can we apply Lindeberg CLT to simple OLS with non-stochastic regressors? [closed]

Consider the simple regression model $$ y_i = \beta x_i+\varepsilon_i $$ where $\{x_i\}_{i=1}^n$ is a sequence of real numbers. Assume $E[\varepsilon_i]=0$ and $E[\varepsilon_i^2]=\sigma^2$. My ...
0
votes
0answers
70 views

Absolute value of residuals in simple linear regression

In a simple linear regression model $$E(Y|X=x)=\beta_0+\beta_1x,$$ where the parameters $\beta_0, \beta_1$ are estimated via OLS as $$\hat{\beta}_1=\frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(X)}, \text{...
0
votes
1answer
13 views

What is the difference in meaning between the Pearson Coefficient and the error from a least squares regression line?

From what I understand, the Pearson Coefficient determines if there is a linear relationship between some data. However, doesn't the error from a least squares regression line also do basically the ...
0
votes
0answers
25 views

OLS reinterpretation (?)

I have read on a book written by a professor something similar to the following and want to check this statement on the forum, since it is the first time I have heard it. I have read that the OLS ...
2
votes
1answer
45 views

Standard error of the intercept in Frisch-Waugh theorem (de-meaned regression)

I am applying in Frisch-Waugh Theorem to partial out a set of fixed effects D and get the OLS estimates and standard errors of the remaining regressors ...
0
votes
0answers
19 views

logistic regression vs OLS - comparison of variable significance

I am modeling usage of a particular app like this: predicting week 3 engagement (number of days of the week the product is used) based on prior engagement (week 1 and week 2) and usage of particular ...
2
votes
1answer
72 views

Heteroscedasticity consistent (HC) standard error analysis and interaction effects in an OLS

I have made a model with several variables, and 8 of them interact with a dummy to find interaction effects. These are added stepwise, resulting in three models. Now, through a Breusch-Pagan test I ...