Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Filter by
Sorted by
Tagged with
1 vote
2 answers
18 views

r - Different estimates of Least Squares and Maximum Likelihood Estimates under non-normality

It is said that Least Squares estimates would differ from Maximume Likelihood estimates if the underlying data would be non-normal. This should be the reason why in linear regression LS estimates can ...
Anti's user avatar
  • 147
1 vote
1 answer
34 views

Why does `systemfit` yield different results for OLS and WLS under cross-equation restrictions?

I am following up on the question "Why does systemfit yield identical results for OLS and WLS?". It deals with estimating a system of linear equations ...
Richard Hardy's user avatar
0 votes
0 answers
8 views

Only first differencing the independent variable?

I am trying to study the relationship between income and preferences for redistribution. I have a panel dataset. I can simply the model (1): preferences = alpha + income + e. However, I'm also ...
chunguc1004's user avatar
1 vote
2 answers
58 views

Why does `systemfit` yield identical results for OLS and WLS?

I am estimating a system of seemingly unrelated regressions (SUR) using the systemfit package in R. Each of the equations has one unique regressor and one common ...
Richard Hardy's user avatar
2 votes
1 answer
36 views

Does OLS give the maximum likelihood estimation for a linear log model?

I'm fitting a model $y=a\times \log(x)+ b$ using standard scikit linear regression (wich uses OLS) and a transformation $x'=\log(x)$. My doubt is: the parameters I get for the model are the best one ...
Roger Danilo Figlie's user avatar
2 votes
2 answers
47 views

Test and Train RSS in OLS model

I encountered the following true/false question: Given a train sample with $\ N $ observations and OLS model fitted on that sample, the RSS of the train sample will be less than or equal to the ...
bm1125's user avatar
  • 177
1 vote
1 answer
30 views

How can I find what is driving the change between my updated model and the original model?

I'm working on updating a model of rents (which is currently simple OLS) for my employer who has a large national portfolio. By tweaking here and there and drawing in a large amount of exogenous data ...
wjb_hwe's user avatar
  • 11
3 votes
3 answers
112 views

Calculate $R^2$ given estimated coefficients and $N$ only

We have a simple regression equation $y=a+bx$, where $a,b$ were estimated via OLS -- we know these values. Suppose the number of observations $N=25$ is given. Is it true, that we cannot calculate $R^2$...
Vnature's user avatar
  • 325
0 votes
1 answer
43 views

Outliers in OLS

I'm running OLS regression in stata with 5-6 IVs (all dummy variables) and a DV (continuous). Does it make sense to check for outliers (e.g., I can't run scatter dv x1 x2) in this case? Also, is it ...
brian's user avatar
  • 65
2 votes
1 answer
49 views

Regression in sex differences multiple dependent variables

If I want to examine sex differences in three variables, lets say academic attainment, study motivation, and a variable that is categorical. How should can I fit these variables with OLS regression? ...
user409571's user avatar
0 votes
0 answers
20 views

What are some possible reasons my feature has the opposite direction of effect in a multivariate model vs bivariate tests [duplicate]

I'm working on a model of rent for my employer. To keep it interpretable we're using OLS. I've had great luck on feature engineering so far in terms of increasing R-squared and reducing AIC. However, ...
Nye307's user avatar
  • 11
4 votes
1 answer
222 views

Why does this matrix form of weighted least squares not match sklearn's weight?

I coded up the answer to this question and it turned out not to match: https://math.stackexchange.com/questions/1021812/matrix-form-for-weighted-least-squares The solutions are close, and I'm ...
ron burgundy's user avatar
0 votes
0 answers
5 views

Single model restrictions X Multivariate model restrictions

I cannot spin my head around how to implement linear restrictions into multivariate models. Or, at least, what is the basic intuition behind it. Let's start from the easy stuff I understand: ...
Athaeneus's user avatar
  • 165
0 votes
0 answers
19 views

Generated regressors: using estimated value and residuals as regressors for another model when R2 is low

Consider three variables, $y, x, z$. Variable $y$ is a linear function of $x$ and $x$ has a 'part' that is driven by $z$ and another that is not. For example, the volume traded of a certain stock ($y$)...
debrah's user avatar
  • 1
5 votes
3 answers
405 views

Do autocorrelated residuals cause OLS coefficients to be biased?

I see different answers everywhere. Intuitively, I would think if residuals are autocorrelated then there is some information that you are not incorporating into your model and is a sign of a biased ...
user2330624's user avatar
3 votes
1 answer
80 views

Ordinary Linear Regression with One Independent Variable

I am currently undertaking a project where I aim to explore the relationship between a single independent variable and a dependent variable. I have five questions that are answered on a 5-Point Likert ...
NutellaMonster's user avatar
0 votes
0 answers
19 views

Endogeneity problem in ols regression and causality

If in my model, the independent variables are uncorrelated with the error term, there is no endogeneity problem, and the residuals satisfy the OLS assumptions, can I say that this model identifies ...
Ivan's user avatar
  • 71
1 vote
1 answer
70 views

Implication for a perfect fit in OLS regression

If $ \hat{\beta} = (X'X)^{-1}X'y $ with $ X $ being an $ n \times k $ matrix, then as I understand it, as long as $ k \leq n $, $ X'X $ is invertible (as long as all other OLS assumptions are ...
Joe94's user avatar
  • 85
1 vote
0 answers
67 views

OLS and log-likelihood from scratch with Tensorflow

I'm trying to code ordinary least square regression from scratch using Tensorflow and calculate the log likelihood. The results, however, are very different from the ones I get from my baselevel model,...
saml's user avatar
  • 11
2 votes
0 answers
23 views

Variance of a single measurement

Say that I have a collection of $n$ data points: $x_i, y_i, i = 0, \ldots, n-1$, and $x_0 < x_1 < \cdots < x_{n-1}$. The $x_i$ are the independent data, and the $y_i$ are the dependent data (...
OrangeWombat's user avatar
1 vote
1 answer
56 views

What if anything is being fit in a one-sample t-test?

I am revisiting some basic concepts involving t-tests and ANOVAs, and got tripped up early. I wanted to apply the concept of lack-of-fit sum of squares to the single sample t-test but wonder how this ...
Buck Thorn's user avatar
0 votes
0 answers
53 views

Relating Moments to OLS

I am trying to see the relationship between OLS and Method of Moments. Moment Equation: For a discrete random variable and a continuous random variable centered around some point "c": $$E[(X-...
Uk rain troll's user avatar
0 votes
0 answers
21 views

Endogeneity Analysis without the access of raw data?

I currently have the correlation/covariance matrix for a set of variables, as well as the output from a regression analysis, but lack access to the underlying raw dataset. Given these constraints, ...
Harshavardhana Srinivasan's user avatar
0 votes
2 answers
40 views

Mean square in least squares problem

While following some code to for a least squares problem using gradient descent, the claim was that the functional to be minimized is the "mean square error", $E=\frac{1}{n}\sum_{i=0}^n(y_i-\...
user124910's user avatar
2 votes
2 answers
64 views

Is $E((\Sigma_i^n u_i)^2) = 0$ or $n\sigma^2$ (in OLS)?

Consider an OLS estimator, $$y= \beta_0 + \beta_1 x_i + u_i $$ I think $E((\Sigma_i^n \: u_i)^2) $ is equal to zero because simply $\Sigma_i^n \: u_i$ is always zero. But my professor in class showed ...
ragul-n's user avatar
  • 21
0 votes
0 answers
10 views

Distribution of the OLS estimator in a predictive regression model

I have a model of the following kind: $y_t = \alpha +\beta x_{t-1} + u_t$ $x_t = \rho x_{t-1} + v_t$ Where: $Cov(u_t, v_t) = \sigma_{uv}\neq 0$ $u_t \sim N(0, \sigma^2_u)$ $v_t \sim N(0, \sigma^2_v)$ ...
Giorgio's user avatar
2 votes
0 answers
23 views

Is there a transformation that could inverse the residuals in multiple OLS regression?

Let's say I have a partial residual plot that looks like this, where the residuals are predicted minus actuals. I would instead prefer for the residuals to be inversed, so that instead of ...
confused's user avatar
  • 3,253
2 votes
0 answers
58 views

OLS estimator and conditional variance weighting

I'm reading Counterfactuals and Causal Inference by Morgan and Winship. In chapter 6, they discuss OLS as a means of estimating the average treatment effect for a binary exposure $D$ (assuming all ...
Demetri Pananos's user avatar
0 votes
0 answers
36 views

Assumptions about data distribution for OLS [duplicate]

From all the content I have read, ordinary least squares (OLS) assumes that the errors are normally distributed i.i.d. I still cannot find any content that gives a bit more insight on what can be ...
spie227's user avatar
3 votes
2 answers
90 views

Instrumental variable as a control variable

I understand that instrumental variable is used to address endogeneity bias since there could be correlation between the variable of interest and the error term. Suppose now we want to see the ...
hiu's user avatar
  • 33
1 vote
1 answer
32 views

SEM: Robust estimation in Onyx

Does anyone know if there is a possibility for robust estimation (e. g. Yuan-Bentler or Satorra-Bentler) in Onyx, because the normal distribution assumption is not given? I only see the options ...
anonymoususer's user avatar
0 votes
0 answers
27 views

taylor approximation multivariate OLS coefficient

Say we have the following multivariate regression model: $ y = \beta_1 x_1 + \beta_2 x_2 + \varepsilon $ The OLS formula for the first coefficient looks like this $ \hat{\beta}_1 = \frac{Cov(\tilde{y}...
user9875321__'s user avatar
1 vote
1 answer
44 views

What is the intuition for estimating residuals when boosting linear regression models?

So basically the title is my question. lin-reg model: $$y_i = x^{T}_i\beta + \epsilon_i, i = 1,...,n$$ Initalize $\hat{\beta^{[0]}}$ and the number of iterations $m_{stop}$. Compute: $$u = y - X\hat{\...
BlankerHans's user avatar
2 votes
0 answers
60 views

Constrained least squares where at least one of two coefficients is zero

I have a linear model with a bunch of variables a number of linear constraints on these variables. I am currently using quadratic programming to solve this constrained least squares problem. However, ...
galpo's user avatar
  • 121
0 votes
0 answers
26 views

Panel vs Pooled OLS

My sample comprises of data on accounting performance of companies that had their IPOs between 2009-22. I want to examine if companies which had more foreign investor participation in their IPOs ...
roshnigarg's user avatar
0 votes
0 answers
29 views

Advantages of GLS Estimator for OLS in the Presence of Violated Spherical Assumption

Let be the linear model given by: $$y_i = x_i'\beta + \varepsilon_i$$ Using its matrix form, consider strictly exogenous assumption and spherical assumption, respectivelly: $$E[\varepsilon | X]=0, \...
user346624's user avatar
0 votes
0 answers
32 views

How does correlation give better model predictability?

Does correlation give better model predictability. In case of using regression models, typically OLS, how does it help with the model predictability and what are its limitations. Any articles or other ...
user402101's user avatar
2 votes
1 answer
47 views

Can I compute a VAR Model and then work on only one OLS equation?

Good morning, I'm trying to estimate a VAR model between six variables and one of them is the price of copper. What I'm interested in is only the equation of the copper prices and i'm running a VAR ...
Ricter's user avatar
  • 123
2 votes
1 answer
49 views

Use of weights in non-linear least square fitting

I would like to have your suggestions and help concerning my problem. I have images generated on a position sensitive detector. The signal for each pixels corresponds to the amount of 'particles' ...
toto's user avatar
  • 21
0 votes
0 answers
22 views

Calculating the error on the speed of light using least squares

I am attempting to calculate the speed of light in a fiber optic by pinging servers around the world, and I am trying to figure out which algorithm I should use to calculate my two parameters. I know ...
Declan Lynch's user avatar
0 votes
0 answers
31 views

Expression for Omitted Variable Bias in Particular Coefficient [duplicate]

Suppose the true model is $y_i=\beta_0+\beta_1x_{1i}+\beta_2x_{2i}+\epsilon_i$, but $x_2$ is not observable. If we run a regression instead on the model $y_i=\beta_0+\beta_1x_{1i}+\eta_{1i}$, what is ...
user624173's user avatar
1 vote
2 answers
83 views

Effectiveness of Using Moving Averages in Timeseries Regression Analysis with Noisy Data

I am working with a dataset where I suspect there is significant noise in both the dependent variable ($y_t^*$) and the independent variable ($x_t^*$). I am considering using moving averages to ...
The One's user avatar
  • 205
0 votes
0 answers
17 views

least squares in regression with covariate-dependent model [duplicate]

Classical least squares results in regression in statistics state that if $(Y, X)$ follow a model where $$\mathbb{E}[Y\mid X=x] = \alpha + \beta x,$$ we can estimate $\beta$ from a random sample ...
Albert Paradek's user avatar
4 votes
1 answer
143 views

What are the best metrics to compare an OLS model and Random Forest model to predict house prices? [duplicate]

I am working on an assignment where the objective is to predict housing prices. My initial approach involves using an Ordinary Least Squares model. Following this, I plan to make a Random Forest model ...
Tim's user avatar
  • 273
3 votes
1 answer
126 views

Weighted least squares for a linear model

Background I have a 2-dimensional dataset $\{y_i, x_i\}_{i=1}^N$ in the coordinates $y,x$. I'm trying to fit the dataset with the trivial model $$\tag{*}y=mx$$ where $m$ is a (scalar) parameter that ...
matteogost's user avatar
12 votes
2 answers
1k views

If OLS estimator minimizes MSE, how does James-Stein Estimator achieve a lower MSE?

OLS estimator solves the following minimization problem: $$\min ||y-X\beta||^2.$$ By taking the FOC, we obtain $\hat{\beta}$, which minimizes the objective function. But the James-Stein estimator ...
user404474's user avatar
8 votes
1 answer
409 views

Why not directly brute force sparsify the OLS estimator instead of using Lasso?

I have a question about the Lasso estimator. I understand that it is particularly useful in high-dimensional settings due to its sparsity-inducing properties. For instance, if the design matrix is ...
user405777's user avatar
0 votes
1 answer
17 views

How to interpret a regression with years on the LHS

I would like to know how I can interpret the coefficients in a regression when the dependent variable is years. For example, suppose I am interested in the year different cities received a new Apple ...
Cola's user avatar
  • 33
3 votes
0 answers
33 views

Confidence interval of least square estimator and dependence of parameters

I have data from a physics experiment, where we measure some quantity $y$ as a function of $x$ and $t$. In practice, I have access to $M$ values $x_i$ of $x$, $N$ values $t_j$ of $t$, and thus $M\...
Adam's user avatar
  • 203
2 votes
1 answer
45 views

Linear regression on 3D position measurements

I have 3D position measurements $x_i,y_i,z_i$ and their corresponding timestamps $t_i$ in a buffer. The time intervals are not equal between all timestamps. I would like to carry out linear regression,...
user120112's user avatar

1
2 3 4 5
57