Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Filter by
Sorted by
Tagged with
3 votes
3 answers
75 views

Do autocorrelated residuals cause OLS coefficients to be biased?

I see different answers everywhere. Intuitively, I would think if residuals are autocorrelated then there is some information that you are not incorporating into your model and is a sign of a biased ...
user2330624's user avatar
0 votes
0 answers
36 views

Ordinary Linear Regression with One Independent Variable

I am currently undertaking a project where I aim to explore the relationship between a single independent variable and a dependent variable. I have five questions that are answered on a 5-Point Likert ...
NutellaMonster's user avatar
0 votes
0 answers
18 views

Endogeneity problem in ols regression and causality

If in my model, the independent variables are uncorrelated with the error term, there is no endogeneity problem, and the residuals satisfy the OLS assumptions, can I say that this model identifies ...
Ivan's user avatar
  • 71
-1 votes
0 answers
21 views

heteroskedasticity

I am struggling to understand if it is normal that, when using robust HAC estimators in estimating the parameters of the market model (with a single index and a single dependent variable), the ...
Mattia's user avatar
  • 117
1 vote
1 answer
65 views

Implication for a perfect fit in OLS regression

If $ \hat{\beta} = (X'X)^{-1}X'y $ with $ X $ being an $ n \times k $ matrix, then as I understand it, as long as $ k \leq n $, $ X'X $ is invertible (as long as all other OLS assumptions are ...
Joe94's user avatar
  • 85
1 vote
0 answers
64 views

OLS and log-likelihood from scratch with Tensorflow

I'm trying to code ordinary least square regression from scratch using Tensorflow and calculate the log likelihood. The results, however, are very different from the ones I get from my baselevel model,...
saml's user avatar
  • 11
2 votes
0 answers
23 views

Variance of a single measurement

Say that I have a collection of $n$ data points: $x_i, y_i, i = 0, \ldots, n-1$, and $x_0 < x_1 < \cdots < x_{n-1}$. The $x_i$ are the independent data, and the $y_i$ are the dependent data (...
OrangeWombat's user avatar
0 votes
1 answer
53 views

What if anything is being fit in a one-sample t-test?

I am revisiting some basic concepts involving t-tests and ANOVAs, and got tripped up early. I wanted to apply the concept of lack-of-fit sum of squares to the single sample t-test but wonder how this ...
Buck Thorn's user avatar
0 votes
0 answers
52 views

Relating Moments to OLS

I am trying to see the relationship between OLS and Method of Moments. Moment Equation: For a discrete random variable and a continuous random variable centered around some point "c": $$E[(X-...
Uk rain troll's user avatar
0 votes
0 answers
21 views

Endogeneity Analysis without the access of raw data?

I currently have the correlation/covariance matrix for a set of variables, as well as the output from a regression analysis, but lack access to the underlying raw dataset. Given these constraints, ...
Harshavardhana Srinivasan's user avatar
0 votes
2 answers
40 views

Mean square in least squares problem

While following some code to for a least squares problem using gradient descent, the claim was that the functional to be minimized is the "mean square error", $E=\frac{1}{n}\sum_{i=0}^n(y_i-\...
user124910's user avatar
2 votes
2 answers
62 views

Is $E((\Sigma_i^n u_i)^2) = 0$ or $n\sigma^2$ (in OLS)?

Consider an OLS estimator, $$y= \beta_0 + \beta_1 x_i + u_i $$ I think $E((\Sigma_i^n \: u_i)^2) $ is equal to zero because simply $\Sigma_i^n \: u_i$ is always zero. But my professor in class showed ...
ragul-n's user avatar
  • 21
0 votes
0 answers
10 views

Distribution of the OLS estimator in a predictive regression model

I have a model of the following kind: $y_t = \alpha +\beta x_{t-1} + u_t$ $x_t = \rho x_{t-1} + v_t$ Where: $Cov(u_t, v_t) = \sigma_{uv}\neq 0$ $u_t \sim N(0, \sigma^2_u)$ $v_t \sim N(0, \sigma^2_v)$ ...
Giorgio's user avatar
2 votes
0 answers
23 views

Is there a transformation that could inverse the residuals in multiple OLS regression?

Let's say I have a partial residual plot that looks like this, where the residuals are predicted minus actuals. I would instead prefer for the residuals to be inversed, so that instead of ...
confused's user avatar
  • 3,243
2 votes
0 answers
54 views

OLS estimator and conditional variance weighting

I'm reading Counterfactuals and Causal Inference by Morgan and Winship. In chapter 6, they discuss OLS as a means of estimating the average treatment effect for a binary exposure $D$ (assuming all ...
Demetri Pananos's user avatar
0 votes
0 answers
36 views

Assumptions about data distribution for OLS [duplicate]

From all the content I have read, ordinary least squares (OLS) assumes that the errors are normally distributed i.i.d. I still cannot find any content that gives a bit more insight on what can be ...
spie227's user avatar
3 votes
2 answers
84 views

Instrumental variable as a control variable

I understand that instrumental variable is used to address endogeneity bias since there could be correlation between the variable of interest and the error term. Suppose now we want to see the ...
hiu's user avatar
  • 33
1 vote
1 answer
30 views

SEM: Robust estimation in Onyx

Does anyone know if there is a possibility for robust estimation (e. g. Yuan-Bentler or Satorra-Bentler) in Onyx, because the normal distribution assumption is not given? I only see the options ...
anonymoususer's user avatar
0 votes
0 answers
27 views

taylor approximation multivariate OLS coefficient

Say we have the following multivariate regression model: $ y = \beta_1 x_1 + \beta_2 x_2 + \varepsilon $ The OLS formula for the first coefficient looks like this $ \hat{\beta}_1 = \frac{Cov(\tilde{y}...
user9875321__'s user avatar
1 vote
1 answer
42 views

What is the intuition for estimating residuals when boosting linear regression models?

So basically the title is my question. lin-reg model: $$y_i = x^{T}_i\beta + \epsilon_i, i = 1,...,n$$ Initalize $\hat{\beta^{[0]}}$ and the number of iterations $m_{stop}$. Compute: $$u = y - X\hat{\...
BlankerHans's user avatar
2 votes
0 answers
59 views

Constrained least squares where at least one of two coefficients is zero

I have a linear model with a bunch of variables a number of linear constraints on these variables. I am currently using quadratic programming to solve this constrained least squares problem. However, ...
galpo's user avatar
  • 121
0 votes
0 answers
25 views

Panel vs Pooled OLS

My sample comprises of data on accounting performance of companies that had their IPOs between 2009-22. I want to examine if companies which had more foreign investor participation in their IPOs ...
roshnigarg's user avatar
0 votes
0 answers
29 views

Advantages of GLS Estimator for OLS in the Presence of Violated Spherical Assumption

Let be the linear model given by: $$y_i = x_i'\beta + \varepsilon_i$$ Using its matrix form, consider strictly exogenous assumption and spherical assumption, respectivelly: $$E[\varepsilon | X]=0, \...
user346624's user avatar
0 votes
0 answers
32 views

How does correlation give better model predictability?

Does correlation give better model predictability. In case of using regression models, typically OLS, how does it help with the model predictability and what are its limitations. Any articles or other ...
user402101's user avatar
2 votes
1 answer
42 views

Can I compute a VAR Model and then work on only one OLS equation?

Good morning, I'm trying to estimate a VAR model between six variables and one of them is the price of copper. What I'm interested in is only the equation of the copper prices and i'm running a VAR ...
Ricter's user avatar
  • 123
2 votes
1 answer
47 views

Use of weights in non-linear least square fitting

I would like to have your suggestions and help concerning my problem. I have images generated on a position sensitive detector. The signal for each pixels corresponds to the amount of 'particles' ...
toto's user avatar
  • 21
0 votes
0 answers
22 views

Calculating the error on the speed of light using least squares

I am attempting to calculate the speed of light in a fiber optic by pinging servers around the world, and I am trying to figure out which algorithm I should use to calculate my two parameters. I know ...
Declan Lynch's user avatar
0 votes
0 answers
31 views

Expression for Omitted Variable Bias in Particular Coefficient [duplicate]

Suppose the true model is $y_i=\beta_0+\beta_1x_{1i}+\beta_2x_{2i}+\epsilon_i$, but $x_2$ is not observable. If we run a regression instead on the model $y_i=\beta_0+\beta_1x_{1i}+\eta_{1i}$, what is ...
user624173's user avatar
1 vote
2 answers
80 views

Effectiveness of Using Moving Averages in Timeseries Regression Analysis with Noisy Data

I am working with a dataset where I suspect there is significant noise in both the dependent variable ($y_t^*$) and the independent variable ($x_t^*$). I am considering using moving averages to ...
The One's user avatar
  • 205
0 votes
0 answers
17 views

least squares in regression with covariate-dependent model [duplicate]

Classical least squares results in regression in statistics state that if $(Y, X)$ follow a model where $$\mathbb{E}[Y\mid X=x] = \alpha + \beta x,$$ we can estimate $\beta$ from a random sample ...
Albert Paradek's user avatar
4 votes
1 answer
142 views

What are the best metrics to compare an OLS model and Random Forest model to predict house prices? [duplicate]

I am working on an assignment where the objective is to predict housing prices. My initial approach involves using an Ordinary Least Squares model. Following this, I plan to make a Random Forest model ...
Tim's user avatar
  • 273
2 votes
1 answer
120 views

Weighted least squares for a linear model

Background I have a 2-dimensional dataset $\{y_i, x_i\}_{i=1}^N$ in the coordinates $y,x$. I'm trying to fit the dataset with the trivial model $$\tag{*}y=mx$$ where $m$ is a (scalar) parameter that ...
matteogost's user avatar
12 votes
2 answers
1k views

If OLS estimator minimizes MSE, how does James-Stein Estimator achieve a lower MSE?

OLS estimator solves the following minimization problem: $$\min ||y-X\beta||^2.$$ By taking the FOC, we obtain $\hat{\beta}$, which minimizes the objective function. But the James-Stein estimator ...
user404474's user avatar
8 votes
1 answer
406 views

Why not directly brute force sparsify the OLS estimator instead of using Lasso?

I have a question about the Lasso estimator. I understand that it is particularly useful in high-dimensional settings due to its sparsity-inducing properties. For instance, if the design matrix is ...
user405777's user avatar
0 votes
1 answer
16 views

How to interpret a regression with years on the LHS

I would like to know how I can interpret the coefficients in a regression when the dependent variable is years. For example, suppose I am interested in the year different cities received a new Apple ...
Cola's user avatar
  • 33
3 votes
0 answers
33 views

Confidence interval of least square estimator and dependence of parameters

I have data from a physics experiment, where we measure some quantity $y$ as a function of $x$ and $t$. In practice, I have access to $M$ values $x_i$ of $x$, $N$ values $t_j$ of $t$, and thus $M\...
Adam's user avatar
  • 203
2 votes
1 answer
41 views

Linear regression on 3D position measurements

I have 3D position measurements $x_i,y_i,z_i$ and their corresponding timestamps $t_i$ in a buffer. The time intervals are not equal between all timestamps. I would like to carry out linear regression,...
user120112's user avatar
3 votes
1 answer
84 views

OLS Forecasting Intervals

currently studying for my econometrics exam and struggling to understand the difference between these two forecasting intervals. xf are new observations added to the sample. Could someone please ...
Quack's user avatar
  • 33
1 vote
1 answer
84 views

How to show that OLS and MLE are the same for $\beta$

I have several questions regarding this proof: Shouldn't the $\propto$ be used instead of the $=$ when we leave out the $\frac{1}{\sqrt{2\pi\sigma²}}$ ? Is a maximization problem simply inverted to a ...
BlankerHans's user avatar
2 votes
1 answer
49 views

How do I handle outliers?

I'm calculating the beta coefficients for some stocks using a single-index linear model with the OLS method. I'm computing the betas at different return intervals to assess the interval effect on the ...
Mattia's user avatar
  • 117
0 votes
0 answers
27 views

OLS and Linear and Nonlinear Models

I understand what linear (w.r.t. coefficients) models are. For example, the power model $Y=k X^p$ is nonlinear..$Y= \beta ^X$ is also nonlinear, etc. Ordinary-least-squares (OLS) is one of the many ...
Brett Cooper's user avatar
2 votes
1 answer
114 views

Bayesian optimization for solving least squares

Bayesian optimization with Gaussian processes (GPs) is an effective minimization methodology when the evaluation of the function to minimize, say $f(a)$, is computationally expensive. Loosely speaking,...
altroware's user avatar
  • 206
4 votes
3 answers
159 views

Proof that quadratic regression splines are continuous at the knots?

The title says it all. For more info: I have a dependent and independent variable $y$ and $X$. I want to fit a square spline on the data given a single knot $k$. I can do that by fitting 2 separate ...
PaulG's user avatar
  • 1,245
3 votes
0 answers
112 views

Least Square Estimate and Latent Space

I'm currently studying regression coefficient w.r.t latent space in a paper Surprises in High-Dimensional Ridgeless Least Squares Interpolation by Trevor Hastie etc. This topic occurs in chapter 5.4 (...
jason 1's user avatar
  • 267
2 votes
1 answer
51 views

What are the implications on prediction if a regression model with heteoscedastic errors is considered for analysis?

I have a simple linear regression model, and I tried validating if the model fit for purposes. One of the tests is Bruesch Pagan test on the residuals but the result of the test shows that the error ...
Abdul Mateen Hashim's user avatar
2 votes
1 answer
40 views

One extreme outlier fitted value that is replaced by another when dropped

I have an OLS model with a very bad prediction score - when I decided to test for heteroskedasticity, it turned out my model's predictions include one incredibe outlier - it looks like my fitted ...
ErikHansen's user avatar
1 vote
1 answer
36 views

How to estimate the coefficients in OLS (all steps) [duplicate]

I'm a B.S. Math graduate who likes to (attempt) to teach myself statistics on my own time because I can't afford a masters degree. It really bothers me when I can't understand at a fundamental level ...
AdmiralMunson's user avatar
1 vote
0 answers
20 views

Does GEE (Generalized Estimating Equations) need normality of residuals for inference in case of "approx. Gaussian" response?

A quick question. My intention is to analyze some numerical data across several categories (treat this as ANOVA, if you wish, but I'm going to focus on simple effects) that are "just numeric"...
FatimaShannn23's user avatar
3 votes
1 answer
81 views

Dual form of the least square solution (ridge rigression)

I was reading this introductory material and on the 5th page, it describes the dual form of the least-square solution (with ridge regression) as $$A(aI + A^\top A)^{-1} = (aI + AA^\top)^{-1}A$$ for a $...
Alemu's user avatar
  • 125
7 votes
3 answers
315 views

OLS assumptions of uncorrelatedess

When dealing with data $(X,Y)$ that is not time series data, for example $X=\text{weight}$ and $Y=\text{height}$, we can use OLS to estimate the coefficients $b_1$ and $b_0$ of a linear regression ...
Brett Cooper's user avatar

1
2 3 4 5
57