Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

2
votes
0answers
19 views

Show positive semi-definiteness of co-variance differences

I am having difficulty checking if $$ (A'D^{-1}A)(A'D^{-1}BD^{-1}A)^{-1}(A'D^{-1}A) - (A'A)(A'BA)^{-1}(A'A) \succeq 0 $$ where $B\succ 0$ is a square co-variance matrix and $D$ is a diagonal matrix ...
1
vote
2answers
38 views

Is it valid to iterate over every permutation of a regression specification and compute an “average significance?”

I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner. Let's say I run an OLS model and the results come back significant. There is discussion as to ...
0
votes
1answer
21 views

Derivation of Formula for linear regression with data points forming a circle

Suppose we have one dimensional data with following conditions.1. The value of this single feature lies in the closed interval [-a,a].2. The associated target value with each data also lies in the ...
0
votes
0answers
11 views

How to obtain a trend estimate using only ols?

I have data on $y$ and $t$ and I want to estimate the trend in $y$ that is supposedly given by relation $y=a(1-b e^{t/τ})$. I want to use linear regression on some transformed version of the model in ...
4
votes
1answer
300 views

OLS robust to outliers

I am facing the following problem: I have a training sample and estimate a model on that training sample. My model is simply OLS: $y_t = a + \beta x_t + \varepsilon_t$. The model is estimated on ...
12
votes
2answers
1k views

Gradient descent doesn't find solution to ordinary least squares on this dataset?

I have been studying linear regression and tried it on below set {(x,y)}, where x specified the area of house in square-feet, and y specified the price in dollars. This is the first example in Andrew ...
0
votes
0answers
14 views

How would I figure out - TSS of Y, Yi and XiYi from the following information?

I came across an analogous question when revising and have no clue how to approach it. The Given information is ΣXi= 20, ΣYi=40 Σ(Xi-x̅)²= 40, Σ(Xi-x̅)(Yi-nȳ)=20 and n=20. The question requires ...
5
votes
3answers
11k views

Difference between fixed effects dummies and fixed effects estimator?

I started to read about panel regression models. However, I am a bit confused about the different model specifications in the fixed effects model: Does a fixed effects panel regression always mean ...
0
votes
1answer
16 views

Cluster-robust standard errors in panel data analysis

In a simple panel data analysis with data on 64 firms over 8 years, I use cluster-robust standard errors (at the firm level) to evaluate significance of coefficients. I observe important differences ...
1
vote
0answers
52 views

Sequential Least Squares for Tikhonov Regularization [duplicate]

Given a Weighted Linear Least Squares problem where the cost function is given by: $$ J = { \left( x - H \Theta \right) }^{T} {C}^{-1} { \left( x - H \Theta \right) } $$ There is a Sequential ...
0
votes
0answers
21 views

Cannot replicate a 95% CI using: Beta ±1.96*SE [duplicate]

I am having a lot of difficulty figuring out why I cannot calculate a 95% CI by hand with one set of data; but I can using the same process with a different set of data. Drawing on OLS results; I am ...
0
votes
0answers
10 views

Comparing the output and fit of an OLS and a Tobit model, and comparing Tobit models

I am trying to estimate a quite complicated model (many variables with different structures), with a limited dependent variable, which ranges from 0-100% with about 45% of the sample having an ...
0
votes
0answers
31 views

Does standard error affected by the coefficient?

I make a comparison on ridge regression and OLS using simulation. As i set my correlation as 0.9, which is high, i expect the standard error of ridge regression to be low. However, it is not. ...
0
votes
0answers
9 views

Least squares and identificability condition

Let a discrete-time system (which is minimal) with input $u \in \mathbb{R}^m$ and state $x \in \mathbb{R}^n$ be $ x_{k+1} = [x^T_k \quad u^T_k]\begin{bmatrix} A^T \\ B^T \end{bmatrix} + v^T_k $ ...
0
votes
0answers
12 views

upward/downward bias of negative variable

If I have a variable that, considering some omitted factor, should have fallen by a higher amount than when it is not there - would that be a downward bias? I.e. the decrease is not large enough, so ...
1
vote
0answers
27 views

Degrees of freedom in OLS regresison vs Bootstrap

I understand that in OLS, the degrees of freedom for estimating the variance of the residuals is n-q-1. We loose q+1 degrees because they are "used" to analytically determine the q parameters and 1 ...
0
votes
0answers
11 views

How to motivate a POLS?

How would you justify the usage of Pooled OLS regression instead of Fixed effects? If I am calculating just correlation between two phenomena, may I get rid of these fixed effects? May I choose to ...
2
votes
1answer
43 views

Does $\hat{\beta}=(X′X)^{−1}X′ \ Y$ simplify further?

As far as I know $(AB)^{-1}=B^{-1}A^{-1}$ and matrix multiplication is associative so: $$(X′ X)^{−1}X′=(X^{-1})(X')^{-1}(X')=X^{-1}.$$ What am I doing wrong?
3
votes
1answer
32 views

Obtain global linear regression estimate from subsamples

I want to estimate $\widehat\beta$ in a simple linear regression with scikit. $$y = X \beta + \varepsilon$$ The problem is that the dimension of the complete $X$ is too large to fit into memory. Is ...
18
votes
2answers
3k views

Proof that F-statistic follows F-distribution

In light of this question : Proof that the coefficients in an OLS model follow a t-distribution with (n-k) degrees of freedom I would love to understand why $$ F = \frac{(\text{TSS}-\text{RSS})/(p-1)...
1
vote
1answer
21 views

How to obtain the inverse of the variance covariance matrix of GLS (Random Effects Model)

In the standard GLS set up how do you find the inverse of the variance covariance matrix? $$y _ { i t } = \beta _ { 0 } + x _ { i t } ^ { \prime } \beta + \alpha _ { i } + u _ { i t } \hspace{35pt} u ...
0
votes
1answer
18 views

Log-normalization of predictors

I have the following dependent and independent variables for my linear regression model. Since they are all in different scales (some of the are % others continuous variables), I was suggested to take ...
0
votes
1answer
19 views

Does a panel regression model make sense for my data?

This might be a bit of a newbish question, but I recently picked up a forecasting project at my job, and I'm trying to figure out whether it makes sense to run a panel regression like a Fixed Effects ...
0
votes
0answers
26 views

Why does the variance of the OLS estimator goes to zero as n increases while it is a random variable?

I have a question regarding the OLS estimator. In my class, the teacher asked us to critic this comment: “We write the unconditional variance of the OLS as $V(\hat{b}^{ols}) = 1/N 𝜎^2 E(X_i'X_i )^{−...
11
votes
4answers
3k views

Why are the Least-Squares and Maximum-Likelihood methods of regression not equivalent when the errors are not normally distributed?

Title says it all. I understand that the Least-Squares and Maximum-Likelihood will give the same result for regression coefficients if the model's errors are normally distributed. But, what happens if ...
0
votes
0answers
13 views

Dynamic Pooled OLS: does it make sense?

I have data on 25 countries for about 35 years. I want to estimate a POLS as I am not looking for a causal interpretation but just sttistical correlation between two phenomena. That's why I will not ...
0
votes
0answers
7 views

On dichotomizing a continuous variable and how it affects MSE of OLS

Recently I have learned about the practice of dichotomizing a continuous independent variable (or maybe even discretize it into more than 2 categories), and then run a predictive model (e.g. multiple ...
0
votes
1answer
11 views

Significance of the sum of the main effect and interaction term

Consider a simple linear regression with an interaction term: $Y=b_0 + b_1X +b_2Z+b_3XZ$ where $X$ is continuous and $Z$ is a dummy. I want to find out whether $X$ has a significant impact on $Y$ ...
0
votes
0answers
28 views

How do we compute OLS coefficients from Sum of Squares in multiple linear regression?

Suppose we have a variable y which is dependent on 2 variables say x1 and x2. Then I can understand how we can compute Sum of squares due to x1 and x2. For instance we may have Type I Sum of squares ...
0
votes
0answers
43 views

How to prove variance of OLS estimator in matrix form?

I am reading Wooldridge's Introductory Econometrics (2000), don't judge me, old version = cheap second hand book, and in the page P94 Theorem 3.2 of Multiple Regression Analysis, it says that: $$ Var(...
0
votes
0answers
8 views

LS classifier that is based on the sum of error squares criterion

Is it possible given a set of points from two classes to determine the LS classifier that is based on the sum of error squares criterion? I don't ask if it is efficient if it is possible and if yes ...
2
votes
0answers
58 views

Negative values for OLS variance

I am currently writing some code which performs regression and have noticed that when I calculate variance of $c\hat{\beta}$ I am sometimes on some datasets getting negative values. The variance is ...
0
votes
1answer
562 views

Standard deviation of least-squares standard error

I do a measurement where I collect a set of data and fit it to a linear model using ordinary least squares. From that I get a slope, b and the standard error of it, s. Now I repeat the measurement N ...
0
votes
1answer
41 views

Implications of strict exogeneity for OLS in time series

Zero Conditional Mean (ZCM), or Strict Exogeneity, is given by: $E[u|X]=0$ Equivalently, $E[u_t|X]=0, t=1,...,T$ Is it true that this implies: Zero Unconditional Mean: $E[u_t]=0, \forall t$ ...
2
votes
0answers
29 views

Comparison between coefficients of two fixed effects models with same sample

I have a problem. I have two models of OLS regressions with time and group fixed effects of a crosscountry paneldata analysis: $$Y_i = b_1 X_i + b_2 D_1 Z_i + b_3 D_2 Z_i\tag 1$$ and $$Y_i = b_1 ...
2
votes
0answers
28 views

Intuition behind White's Estimators/ Heteroscedasticity-consistent Standard Errors

For a medical study I am trying to understand the intuition behind heteroscedasticity-consistent standard errors. I know that it can be used, when in OLS regression residuals are heteroscedastic. By ...
0
votes
0answers
17 views

Comparing the mean of predicted values for a misspecified model against the mean of the observed values

I have a regression that I have run on average ratings for some products (dependent variable) and their characteristics (Model 1). I have reason to believe there is a prejudice against a specific set ...
2
votes
1answer
707 views

Non-linear least squares standard error calculation in R

I am using implementations of the Levenberg-Marquardt algorithm for non-linear least squares regression based on MINPACK-1 utilizing either the R function nlsLM() from minpack.lm or an implementation ...
1
vote
1answer
55 views

linear regression predicts lower than expected

I am trying to predict first term GPA for college students based on a number of incoming factors (high school gpa, placement test, year). This isn't the overall model just a simpler one. The first ...
1
vote
0answers
32 views

How to verify the “random sampling” Gauss-Markov Assumption with Stata (or anything else)?

According to the book I am using, Introductory Econometrics by J.M. Wooldridge, there are 5 Gauss-Markov assumptions necessary to obtain BLUE. However, by looking in other literature, there is one ...
1
vote
2answers
31 views

Linearization of multiplicative and exponential regression models and their OLS estimation

So far I have been under the impression that you can "linearize" multiplicative models of the form (1) $y=\alpha * \beta_1x_1 * \beta_2x_2 * \beta_3x_3 $ and exponential models of the form (2) $y=\...
0
votes
1answer
784 views

Formal errors from non-negative least-squares?

I am computing a standard linear regression subject to a positivity constraint using non-negative least squares (lsqnonneg in Matlab, actually). Is it possible to ...
1
vote
0answers
12 views

Simulating size distrotion with a VAR(1) model

Firstly, I am not asking for code, I would like the intution of how I would do this. I am testing for size distortion. I have estimated and VAR(1) model and I have the parameters. I want to ...
2
votes
1answer
422 views

Asymptotic variance of estimated AR(1) coefficient

Consider an $AR(1)$ process \begin{align*} y_{t} + a y_{t-1} &= e_{t} \end{align*} for $t=1,\ldots,N$, where $e_{t}$ is a white Gaussian noise with variance $\sigma^{2}$. How do I express the ...
0
votes
1answer
29 views

Without testing: when are error terms possibly homoscedastic?

I am facing the following study: In the 1980's, Tennessee conducted an experiment in which kindergarten students were randomly assigned to regular and ...
1
vote
0answers
23 views

Consistency of an OLS estimator in time series

In the ARMA(2,1) time series $y_t = \beta_0+\beta_1y_{t-1}+\beta_2y_{t-2}+ u_t + \phi_1u_{t-2}$ $u_t$ and $u_{t-2}$ are white noise shocks. The time series are stationary and ergodic. In the ...
0
votes
1answer
534 views

What is the expectation of the error term conditional on Y?

In an OLS model, we usually assume $(Y_i, X_i)$ are i.i.d., and $U|X$ has mean $0$ and variance $\sigma^2$, and then run a regression of $Y$ on $X$. But when we are running a regression of $X$ on $Y$ ...
2
votes
0answers
22 views

Residualizing in OLS

Consider a linear regression $$ y_i = A_i' \theta + B_i' \psi + \epsilon_i $$ There's a "trick" to find the parameters $\theta$ and $\psi$ in a stepwise fashion. First minimizing squared error wrt....
-1
votes
0answers
15 views

Beta hat marginal distribution in Strong OLS setting

If in OLS setting I assume: Epsilon|X is normally distributed (mean= 0; variance= sigma^2 * Identity matrix) We have that the estimated coefficients (each "beta hat j") is conditional on X normally ...
4
votes
1answer
608 views

Surface Fit Using Tensor Product of B-Splines

I am trying to teach myself surface fitting with splines using tensor products. I am trying to construct a toy example but I can't seem to get my example to work. I will try to explain the best I can ....