Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Filter by
Sorted by
Tagged with
1
vote
0answers
25 views

Regression problem with “error in variables”

Suppose that there is a deterministic relation $y_t=ax_t$ where $x_t,y_t$ are real sequences or real functions and $a$ a constant. But only $X_t=x_t+e_t$ and $Y_t+u_t$ can be observed, with $e_t, u_t$ ...
0
votes
0answers
11 views

Change Score or Regressor Variable Method - Should I regress $Y_1$ over $X$ and $Y_0$ or $(Y_1-Y_0)$ over $X$

I have data about investment preferences 1 year before the Covid and during the Covid lockdown. Some changes appear using simple T-Test. I want to be able to assess if these changes are particularly ...
0
votes
0answers
18 views

Online estimation of a quadratic form

I have a functional form $y = x^T Q x + b^T x + c$, with $Q, b, c$ to be estimated, $x \in \mathbb{R}^n$ and $n$ varies around 10-20, depending on the problem. $x$ is sampled from a known Gaussian ...
0
votes
1answer
33 views

Why are coefficient and standard errors zero?

I am having a problem with OLS regression. I am doing a relative time model (leads and lags model) to prove the robustness of my Difference in differences model. Problem is that some of my ...
3
votes
1answer
67 views
+50

Asymptotic dist of an average involving OLS coefs?

Suppose that we have iid sample of size $n$. i.e., the random vector $(Y_{i}, X_{1i}, X_{2i}, X_{3i})$ is iid from $1,\ldots,n$. And suppose the following relationship is true: $$ Y_i = \beta_0 + \...
0
votes
0answers
14 views

OLS with seasonal components

Suppose the observations $y_t, x_t$ follow the model $$y_t=ax_t+e_t, \quad t=1,\dotsc,T, \quad \quad(1)$$ where the errors are i.i.d. with mean zero and finite variance, and $a$ constant. In addition, ...
1
vote
0answers
26 views

Given $N$ IID zero-meaned points on a plane, find the equation for the plane

Question: (1) Given $N$ IID zero-meaned 3-D (x, y, z) points on a plane, find the equation for the plane. The plane must pass through (0,0,0). (2) Solve problem (1), but now assume every point has ...
0
votes
0answers
33 views

A practical example where maximum likelihood correctly estimates an underlying parameter, but where least squares would fail?

Forgive my very limited understanding. I am trying to learn about maximum likelihood estimation, and how it differs from least-squares estimation. From reading a little, I understand that the two are ...
0
votes
0answers
19 views

Is this formulation of image classification using least squares appropriate?

I want to detect sidewalks in aerial images using a simple least squares classifier. This can be done using a more sophisticated method using a library such as neat-EO, but I would like to implement ...
3
votes
2answers
155 views

Why is the correlation between independent variables/regressor and residuals zero for OLS?

In page 4 of https://web.stanford.edu/~mrosenfe/soc_meth_proj3/matrix_OLS_NYU_notes.pdf, it states that the regressors have zero correlation with the residuals for OLS, but I don't think this is true. ...
0
votes
0answers
14 views

How does regularization affect the offset term and weights for OLS?

I have a couple of questions regarding regularization that I am confused on, and searches on this forum doesn't seem to answer my specific questions. When we regularize with $\lambda ||w||^2$ or $\...
-1
votes
1answer
24 views

How to determine economic significance of an interaction term of two continuous variables in an OLS regression? [closed]

Y= Intercept+ Beta1X1+ Beta2X2+ Beta3(X1X2)** I want to determine the economic significance of Beta3 when Y, X1, and X2 are continuous variables, preferably in terms of standard deviation units. Is ...
0
votes
0answers
23 views

Pearson correlation not distributed around zero when using nonnegative least squares (NNLS) in permutation test?

I am trying to explain artificially created fmri data in MATLAB [1] through a linear combination of w neural network layers. To compare those, I transformed the data into respresentational ...
0
votes
1answer
36 views

Bias-variance tradeoff question

From https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff, the derivation for the bias-variance decomposition of the squared error is $$ E[(y - \hat{f})^2] = Bias[\hat{f}]^2 + \sigma^2 + Var[\...
0
votes
0answers
14 views

Calculating group-level mean with varying group sizes for micro-macro data structure

My study is designed in such a way that I have individuals with varying numbers of observations of the predictors, but only one observation of the outcome. Like a multilevel model, but with the ...
0
votes
1answer
18 views

Regularized least squares with a black box predictive function

In regularized OLS, the regularization parameter is applied to the weights $$ \arg \min_w ||y - f(w)|| + \lambda ||w||^2 \\ f(w) = wx + b $$ Does it change the optimization at all if it was instead ...
1
vote
2answers
62 views

How would you minimize the sum of squares if the predictive function is a black box?

I'm solving an optimization problem, using the mean squared error: $$ \arg\min_{\mathcal{M}} ||y - \hat{y}|| $$ $y$ is the true value and $\hat{y}$ is obtained from some black box function. $\mathcal{...
0
votes
0answers
16 views

Resource for in depth “model checking” for multiple linear regression?

I'm looking for a book or resource that goes really in depth on the model checking and diagnostics for the multiple linear regression. Basically it goes into a lot of depth on how model assumption ...
1
vote
0answers
24 views

Does this exist: Recursive Least Absolute Deviation-Regression?

For least squares regression there exists the Recursive Least Squares algorithm which allows to find the least squares solution online. Does something similar exist for Least Absolute Deviation ...
0
votes
0answers
27 views

How is the p-value in an OLS regression in statsmodels calculated?

I ran an OLS regression using statsmodels. The summary is as follows. I am confused looking at the t-stat and the corresponding p-values. For 'var_1' since the t-stat lies beyond the 95% confidence ...
1
vote
0answers
18 views

Bivariate basis functions with span invariant to rotation about $z$-axis

Consider the following functions defined over $x,y\in\mathbb{R}$: $f_0(x,y)=1$ $f_1(x,y)=x$ $f_2(x,y)=y$ These functions form a basis with three-dimensional span (the set of all non-vertical planes) ...
1
vote
1answer
25 views

Can I estimate the COX regression using OLS

I am wondering if I can estimate the Cox regression $h(t,z)= h_0(t)exp(\beta_1 z_1+\dots+\beta_x z_x)$ model using OLS. The idea would be to rewrite the model with regards to the structure of a simple ...
0
votes
0answers
12 views

Proving Least-Squares Estimator is Unbiased [duplicate]

Im working my way through the text Mathematical Statistics with Applications 7th edition (Wackerly et al) and I am slightly confused by how they go about proving that the least-squares estimates are ...
0
votes
0answers
30 views

Pooled OLS vs Fixed Effects regression

For my thesis, I'm currently comparing the effect of institutional ownership on the idiosyncratic volatility of stocks. My current approach has been to do a simple regression and including multiple ...
0
votes
0answers
15 views

Least squares fit of a bivariate quadratic-linear product to an oriented point set

As explained in this question, a bivariate quadratic has 6 DoF (coefficients), and a bivariate cubic has 10 DoF, while a bivariate quadratic-linear product has 8 DoF. The quadratic or the cubic models ...
1
vote
0answers
28 views

Least-squares fit of explicit parabolic sheet to scattered data points [duplicate]

For a given set of data points $$\{(x_i, y_i, z_i)\}$$ there exists some $$f_{ABC}(x,y)=Ax^2+Bxy+Cy^2$$ that minimizes $$\sum_i(f_{ABC}(x_i,y_i)-z_i)^2$$ $A$, $B$, and $C$ can be found quickly ...
0
votes
0answers
21 views

What kind of regression to run when treatment occurs at different time periods?

I'm working on a regression analysis where I'm trying to estimate how a neighborhood's population growth varies as a function of the amount of land area dedicated to pocket parks (independent variable ...
20
votes
7answers
2k views

Why is using squared error the standard when absolute error is more relevant to most problems? [duplicate]

I recognize that parts of this topic have been discussed on this forum. Some examples: Is minimizing squared error equivalent to minimizing absolute error? Why squared error is more popular than the ...
1
vote
2answers
55 views

Help me prove CLT $\implies$ error term in OLS regression is distributed normally

How can I prove that the central limit theorem implies that error term in an OLS regression is normally distributed? (A statement to this effect has been mentioned in Econometrics p. 33 by Fumio ...
0
votes
1answer
14 views

What is the correct way to report an OLS model with undefined coefficients due to singularities?

I used the R function lm to generate an ordinary least squares model, and one coefficient is "not defined due to singularities". My understanding from reading the StackOverflow question here https://...
0
votes
0answers
52 views

Heteroskedasticity tests: heavy-tailedness of squared estimated errors

I have a time series model and obtain the following distribution of estimated errors: I suspect that the errors are heteroscedastic in the sense that their variance depends on the level of one or ...
1
vote
0answers
22 views

OLS with multicollinear data and possible polynomial effects - what feature selection techniques would work?

I am new to regression modelling, have a dataset (vegetation health over time for ~250 sites) that I would like to model using OLS as a function of 5 continuous variables: light, pollution, humidity, ...
0
votes
0answers
5 views

Panel Data: can not include quadratic variables in GLS or LME modelling

I have used the Pooled OLS technique in R to model my panel data (covering 3 years with 191 firms) and the technique has failed key diagnostic tests due to the presence of heteroskedasticity. So now I ...
1
vote
1answer
64 views

Why is the error square equivalent to variance of error?

I'm confused as to why is V($\hat β$) = E[$\ εε^t$] What's an intuitive explanation for this? Additionally, is this the reason why when we run Breusch Pagan test, we regress $ \ ε^2$ on independent ...
2
votes
1answer
24 views

Doubt on derivation of OLS estimators as unbiased estimators of Optimal Linear Predictors

I'm studying from C. Shalizi's lecture notes https://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/ . In the third chapter he introduces the optimal linear estimator of a random variable $Y$ conditioned to ...
1
vote
1answer
13 views

Nominal control variable: does it really have to be dummy?

I estimate some logistic and OLS models on big survey data (overfitting shouldn't be a problem) where I need to control for a country of a respondent. The country variable is coded as iso3n- 3 digits ...
4
votes
3answers
124 views

Proof that the mean of predicted values in OLS regression is equal to the mean of original values? [duplicate]

https://en.wikipedia.org/wiki/Pearson_correlation_coefficient#In_least_squares_regression_analysis I was reading this page and came across the fact that the mean of the predicted target values for an ...
3
votes
0answers
109 views

Distribution of OLS predictions

Suppose: $y = X\beta + \varepsilon$, with $\varepsilon \sim (0, \Omega) \Rightarrow y|X \sim (X \beta, \Omega)$ $\hat{\beta}_{ols} = (X'X)^{-1}X'Y = \beta +(X'X)^{-1}X'\varepsilon \sim (\beta, \Sigma)...
0
votes
1answer
31 views

Can you explain how OLS for non-linear data is working in statsmodels OLS implementation?

reference: https://www.statsmodels.org/stable/examples/notebooks/generated/ols.html#OLS-non-linear-curve-but-linear-in-parameters How is OLS in this implementation working? Is it finding the best ...
0
votes
1answer
38 views

multiplying regression beta (covariance) coefficients

I consider 3 variables $x$, $y$, $z$ and study the relationship of the beta coefficient from OLS regression of two variables connected via a third. Whilst it is not true that $$\beta_{(x,y)}\times\...
1
vote
0answers
33 views

Derivation of Variance of Variance estimator

so far I do understand how to show that $$ \hat\sigma^2 = \frac{e'e}{n-k} $$ is an unbiased variance estimator in the standard OLS model. However, I don't know how to show that the variance of the ...
0
votes
1answer
40 views

Understanding the measurement error in Levenberg-Marquardt algorithm for non-linear least square

I have been reading this pdf on LM algorithm and I have been having trouble understanding a few parts of it. For example, in the Introduction: where $σ_{y_i}$ is the measurement error for ...
1
vote
0answers
20 views

In typical regression, if Y | X is normal, is Y itself normal?

I am reading DeGroot and we made the assumption that Y | X is normal and each Y | X has the same variance. However, in deriving the sampling distributions of b0 and b1, he says that Y is normal. Do ...
1
vote
0answers
18 views

Is there a way to solve for $\sigma^2$ in linear regression only using least squares estimation?

If we only use least squares estimation, is there a way to solve for the variance of the conditional distribution of Y | X? Or do we have to use Maximum Likelihood Estimation? Additionally, do most ...
0
votes
0answers
17 views

Rationale for taking second derivative in least squares estimation?

I am reading DeGroot and he talks about how to derive the b0 and b1 coefficients using LS estimation. I understand everything except the last part where he talks about taking the second derivative of ...
1
vote
0answers
12 views

Generalisation of Absolute Deviation and Least Squares Deviation: Statistical Use?

This linked answer defines Least Absolute Deviation (LAD) and Least Squares Deviation (LSD), as follows: $$ LAD = \Sigma^n_{i=1} |y_i-f(x_i)| $$ And: $$ LSD = \Sigma^n_{i=1} (y_i-f(x_i))^2 $$ ...
6
votes
2answers
437 views

Does there exist a Frequentist or Non-Bayesian solution to Gull's Lighthouse Problem?

Does there exist a Frequentist or ODE or Non-Bayesian solution to Gull's Lighthouse Problem which is correctly modeled with cauchy distribution? See The Lighthouse Problem and Dave Harris' answer to ...
1
vote
1answer
27 views

Does Maximum Likelihood Estimation solve n < p problem in regression?

If we use Maximum Likelihood Estimation to estimate regression parameters (B and sigma), and if we have less observations (n) than predictors (p), can we bypass dimension reduction ? My understanding ...
0
votes
0answers
9 views

Least squares model selection criteria: standard error of point estimator vs variance of non-zero components in corresponding column of design matrix

I am performing ordinary least squares regression, and I would like some help with my model selection. One possible model is the following: The design matrix is denoted as $ \mathbf Z $ and the vector ...
0
votes
1answer
46 views

Unbiased estimator and biased error

I'm having some trouble relating unbiased estimators and bias error. By bias error, I mean the bias error we talk about when analyzing "bias-variance tradeoffs." Is this bias error and an unbiased ...

1
2 3 4 5
40