Questions tagged [least-squares]
Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.
974
questions with no upvoted or accepted answers
8
votes
0
answers
3k
views
Fitting a Logistic Regression via Brier Score or Mean Squared Error
Is there a name for a logistic regression model that has been fit using the Brier score (or equivalently the mean-squared error) rather than the cross-entropy?
I realise this isn't maximum-likelihood, ...
- 55.1k
8
votes
0
answers
195
views
Scope of non-linear least squares
edit: tl;dr: I can coerce a lot of optimization problems to take the form of a non-linear least squares problem, but does it make sense to do so?
Suppose we have some empirical data $P=\{(x_i', y_i')\...
- 323
7
votes
0
answers
211
views
Total least square intuition
I have yet to find a good intuitive explanation of TLS. Online resources tend to focus on the vertical vs. perpendicular square error pictures (I don't need to see perpendicular lines to understand ...
- 4,568
6
votes
0
answers
557
views
Derivation of a doubly robust estimator with clever covariate and inverse probability weighting
With notation: outcome $Y$, (binary) treatment $A$, and covariates $L$. In Hernan and Robins (2020) causal inference textbook:
To obtain a doubly robust estimate of the average causal effect, first
...
- 6,759
6
votes
0
answers
291
views
Confidence Interval for least squares estimator
There was a paper by Yasin-Abbasi-Yadkori https://arxiv.org/pdf/1102.2670.pdf titled Online Least Squares Estimation with Self-Normalized Processes. I am trying to give a brief context before asking ...
- 183
6
votes
1
answer
4k
views
What is a trust region reflective algorithm?
What is a trust region reflective algorithm?
I know (from the matlab help) that it is used for solving constrained optimization problems.
How is it different than the Levenberg-Marquardt algorithm ...
- 876
6
votes
0
answers
358
views
OLS standard error that corrects for autocorrelation but not heteroskedasticity
Question: By mapping the OLS regression into the GMM framework, write the formula for the standard error of the OLS regression coefficients that corrects for autocorrelation but not heteroskedasticity....
- 899
5
votes
0
answers
1k
views
HAC standard errors: small-sample correction
The Python package statsmodels provides a use_correction option when computing HAC standard errors for an OLS model, which ...
- 540
5
votes
0
answers
2k
views
What are the advantages of a random effects model versus a pooled OLS regression with cluster–robust standard errors?
Both models allow for explanatory variables that are time-invariant.
I had thought that the advantage of a random effects model might be related to the fact that random effects models mitigate ...
- 216
5
votes
0
answers
1k
views
Prediction with OLS better then prediction with lasso or ridge
I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation.
After that i wanted to compare the ...
- 341
5
votes
0
answers
123
views
Direction of Bias in OLS with systematic measurement error
I am looking at a study that wants to measure the effect of $x_t$ on $Y_t$, but the true values of the $x_t$ are not observed. Instead, what is observed is a minimum value for $x_t$. In effect, what ...
- 51
5
votes
0
answers
971
views
Choosing the basis functions in a linear regression
I have two random variables $X$ and $Y$ and I'm trying to model $\mathbb{E}[Y|X]$.
To this end, I'd like to pick a collection of functions $f_1, f_2 \dots f_n : \mathbb{R} \to \mathbb{R}$ and then ...
- 443
5
votes
0
answers
130
views
Derivation of Olsens LS Selectivity Correction
There are many estimation procedures that correct for sample selection. The most famous is Heckman's two-step selectivity correction (in two equations) that assumes bivariate normality of the error ...
- 51
5
votes
0
answers
508
views
What is least angle regression?
Conceptually, I don't understand what least angle regression Least Angle Regression (LARS) is and why it solves LASSO (pdf).
We know that LASSO is:
$$\arg \min_x {\left\| A x - y \right\|}_{2}^{2} +...
- 101
5
votes
0
answers
2k
views
Methods to best test lead/lag relationships
I was wondering if you can share your experiences on what you feel is the best method to test lead / lag relationships between I(1) time series variables (i.e stock prices) and advantages and ...
- 209
4
votes
0
answers
79
views
Proof of invariant angle between $Y$ and $\hat Y$ in $L^2$ regularisation
On this site is the following question which claims that the $L^2$ regularised OLS preserves the angle between $\hat Y$ and $Y$ irrespective of the value $\lambda$. I have not found any source that ...
- 345
4
votes
0
answers
158
views
Small sample OLS regression: how to conduct inference?
Consider a linear data generating process (DGP)
$$
y=X\beta+\varepsilon, \quad \varepsilon\sim i.i.D(0,\sigma^2)
$$
with some unknown distribution $D$ with zero mean and finite variance $\sigma^2$ (...
- 68.8k
4
votes
0
answers
991
views
What is the probability distribution and variance of the OLS estimate $s^2$ of the error variance $\sigma^2$ in linear regression?
Consider the standard linear regression model
$$
y = X \beta + \varepsilon,
$$
where the error $\varepsilon$ has fixed variance $\sigma^2$. We can make an unbiased estimate of the error variance in a ...
- 795
4
votes
0
answers
1k
views
Non-linear least squares covariance estimate
In many places I've seen this formula quoted as an estimate for the covariance matrix $C$ in a nonlinear least squares fit: $$C=\sigma^2H^{-1}$$
where $H$ is the Hessian matrix and $\sigma$ is ...
- 175
4
votes
0
answers
296
views
Distribution of OLS predictions
Suppose: $y = X\beta + \varepsilon$, with $\varepsilon \sim (0, \Omega) \Rightarrow y|X \sim (X \beta, \Omega)$
$\hat{\beta}_{ols} = (X'X)^{-1}X'Y = \beta +(X'X)^{-1}X'\varepsilon \sim (\beta, \Sigma)...
- 151
4
votes
0
answers
954
views
Robust regression for autocorrelation and heteroskedasticity - coefficients do not change, only standard errors change?
When using Newey-West robust standard errors to deal with heteroskedasticity and autocorrelation:
http://support.sas.com/kb/40/098.html
is it correct to state that the coefficients are not different ...
- 483
4
votes
0
answers
685
views
Can "non-parametric" tests be achieved with generalized linear models?
I recently read @Kodiologist's answer to a post here looking for clarification on the relation between GLMs and non-parametric tests. His answer is along the lines of "the approach is not non-...
- 64.2k
4
votes
0
answers
105
views
Only minimizers of quadratic penalized least squares can be linear
Question
Define the matrix $A \in \mathbb{R}^{m \times n}$, the vector $b \in \mathbb{R}^m$, and the function $\mathrm{pen}: \mathbb{R}^n \to \mathbb{R}$. Then, put the minimizer $$\hat{x} = \arg\...
- 2,882
4
votes
0
answers
428
views
Modern approaches to nonlinear regression which are available in R
I would like to fit a complex nonlinear regression model: basically, I have a complex computer code which has an input vector $\mathbf{x}$, a vector of calibration parameters $\boldsymbol{\theta}$ and ...
- 18.3k
4
votes
0
answers
914
views
MSE of an individual coefficient from ridge or lasso vs. OLS
Consider a multiple regression model
$$ y = X\beta + \varepsilon $$
with $K$ regressors in $X$.
If the model is correctly specified, the OLS estimator $\hat\beta_{OLS}=(X'X)^{-1}X'y$ will be the ...
- 68.8k
4
votes
0
answers
302
views
Sign and size of OLS bias for Tobit models
I have a question related to the sign and size of the OLS bias in the case of a Tobit model.
Consider the following model
(1) Sample of observations $\{X_i,Y_i\}_{i=1}^n$, i.i.d., $X_i$ is a vector ...
- 881
4
votes
0
answers
378
views
Best-fit plane through a set of lines?
A linear model estimates the best fit line from a set of points, often through minimization of the sum of squared residuals.
By analogy, is there any established method (possibly implemented in R) ...
- 1,043
4
votes
0
answers
226
views
Prediction Intervals for Incremental OLS regression
I am implementing incremental OLS regression algorithm where the data points arrive one at a time. As the regression parameters are determined by the formula, $(X'X)^{-1} X'y$ and the Sherman-Morrison ...
- 632
4
votes
0
answers
311
views
Techniques for scaling data matrix to avoid rank deficiency issues
I have a $n \times p$ matrix $A$ where $n$ is the number of observables and $p$ is the number of observations. $n \gg p$
In my code, I have done $[E,V] \,=\, eig(A)$ and doing a least squares ...
- 141
4
votes
0
answers
364
views
Regression model with heteroskedasticity in both variables
I've been learning (lurking) from this site for a while and I finally have a question I haven't seen answered yet.
I'm doing a flight test and trying to fit the resulting data to linear line. From a ...
- 149
4
votes
0
answers
1k
views
Relationship between regularized least squares and MLE
We know that the least square method is equivalent to the MLE for Gaussian distributed errors. What is the relationship (if any) between regularized (Tichonov regularization) least squares and MLE?
- 2,088
4
votes
0
answers
357
views
Estimating parameters of an unknown PID controller
Say that I have your standard PID controller at work. To keep it extremely simple imagine I have a target $x^*$ on the variable $x$. Then the controller is:
$y(t) = K_p ( x^* - x_t) + K_i \int_0^t (x^...
- 1,118
4
votes
1
answer
302
views
Simultaneity of price in sales modeling
The price of a product has signficant impact on the total sales. Hence modeling sales would give the incentive to include price as a regressor (amongst other variables). Suppose we would estimate this ...
- 353
4
votes
1
answer
679
views
Least squares robust to outliers
I have sparse overdefined system of linear equations.
For example I have n variables, m equations(m>n) and k equations from m are "bad" equations that represent outliers.
Is there any methods to ...
- 2,217
3
votes
1
answer
112
views
Kalman Filter to minimize weighted errors on the states: what's wrong with my derivation
I am thinking about how to implement a "weighted Kalman Filter". Note that the weights here are on the states. Basically the classical KF minimizes $\sum (x_i - \hat{x_i} )^2$ but I want to ...
- 230
3
votes
0
answers
34
views
Is there a transformation that could inverse the residuals in multiple OLS regression?
Let's say I have a partial residual plot that looks like this, where the residuals are predicted minus actuals.
I would instead prefer for the residuals to be inversed, so that instead of ...
- 3,253
3
votes
0
answers
73
views
OLS estimator and conditional variance weighting
I'm reading Counterfactuals and Causal Inference by Morgan and Winship. In chapter 6, they discuss OLS as a means of estimating the average treatment effect for a binary exposure $D$ (assuming all ...
- 38.6k
3
votes
0
answers
33
views
Confidence interval of least square estimator and dependence of parameters
I have data from a physics experiment, where we measure some quantity $y$ as a function of $x$ and $t$. In practice, I have access to $M$ values $x_i$ of $x$, $N$ values $t_j$ of $t$, and thus $M\...
- 203
3
votes
0
answers
121
views
Least Square Estimate and Latent Space
I'm currently studying regression coefficient w.r.t latent space in a paper Surprises in High-Dimensional Ridgeless Least Squares Interpolation by Trevor Hastie etc. This topic occurs in chapter 5.4 (...
- 311
3
votes
0
answers
20
views
Missing assumption in MA estimation
Assume we observe a $MA(1)$ process for which it is known that the mean is zero. Based on a series of length $3$ , we observe $Y_1 = 0, Y_2 = 1$ and $Y_3 = 0.5$. Find the least-squares estimate of $\...
- 435
3
votes
1
answer
68
views
Proof that p-value in OLS regression is symmetric
OLS regression is not symmetric, meaning that it produces different relationships if you flip the dependent and independent variables; however, it would seem odd if the p-values were different and ...
- 283
3
votes
1
answer
1k
views
Can I interpret control variable's coefficients in linear regression?
imagine I want to estimate a regression model (let's say OLS for simplicity's sake) using observational data. I include a number of controls that might confound the relationship. For example, I might ...
- 31
3
votes
0
answers
177
views
Why orthogonal polynomials lead to diagonal matrix $X^{T}X$ when estimating regression estimates?
We believe that the response variable $Y$ can be modeled in a following way:
$$Y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i^3 + \epsilon_i $$
where $\epsilon_i$ is independently ...
- 604
3
votes
0
answers
115
views
Estimating a linear system of simultaneous equations at once
Consider the following simultaneous system
$$
y_{1} =\beta _{1}y_{2}+\alpha _{1}z+u_{1} \\
y_{2} =\beta _{2}y_{1}+\alpha _{2}z+u_{2}
$$
where $y_{1}$, $y_{2}$ and $z$ are vectors of random variables ...
3
votes
1
answer
249
views
Estimating the errors in parameters in the ordinary least square
I am reading the book An Introduction to Error Analysis by John R. Taylor. In Ch8: Least-Squares Fitting, he has derived expressions for parameters $A$ and $B$ in fitting the line $A+Bx$ to the set of ...
- 623
3
votes
2
answers
29
views
terms in a simple linear least square model
I'm reading a textbook. In the chapter about least square regression I red that A simple linear least square model can be described as
\begin{equation}
Y = \alpha + \beta x + e
\end{equation}
where Y ...
- 409
3
votes
0
answers
451
views
Interpretation: Adding quadratic term makes linear term insignificant (OLS regression)
I'm conducting a multiple OLS regression. My main model contains a significant effect (p < .5) of x on y.
I want to test in a robustness check whether x is related to y in a curvilinear/quadratic ...
- 647
3
votes
0
answers
679
views
What is the effect of autocorrelation on R² in OLS
I was wondering. What would be the effect of the presence of autocorrelation on R² or R²_adjusted? Especially in a dataset used in multiple linear regression?
edit: So the correlation of each variable ...
- 31
3
votes
0
answers
77
views
Why is my QR decomposition updating code numerically off?
I apologize if this is the wrong place for this question; there are a number of potential points of failure each of which suggest either Math StackExchange or StackOverflow or here, but since the ...
- 1,373
3
votes
1
answer
79
views
Measuring the causal impact of a policy that is not binding
This may be a little tricky because it's difficult to explain but bear with me. Assume a new policy implemented in 2015 which is a new requirement for firms, let's say for instance that the ...
- 497