Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

18
votes
0answers
524 views

Bound for Arithmetic Harmonic mean inequality for matrices?

NOTE: This question has originally been posted in MSE, but it did not generate any interest. It was first posted there, because the question itself is a pure matrix-algebra question. Nevertheless, ...
6
votes
0answers
165 views

Scope of non-linear least squares

edit: tl;dr: I can coerce a lot of optimization problems to take the form of a non-linear least squares problem, but does it make sense to do so? Suppose we have some empirical data $P=\{(x_i', y_i')\...
5
votes
0answers
39 views

Is a visual estimate of homoscedasticity rigorous enough?

As part of my research in astronomy (quasar magnitudes at various wavelengths), I've been producing graphs such as the following: The bottom plot on each graph shows the distribution of the residuals ...
5
votes
0answers
255 views

What is least angle regression?

Conceptually, I don't understand what least angle regression Least Angle Regression (LARS) is and why it solves LASSO (pdf). We know that LASSO is: $$\arg \min_x {\left\| A x - y \right\|}_{2}^{2} +...
5
votes
0answers
1k views

Methods to best test lead/lag relationships

I was wondering if you can share your experiences on what you feel is the best method to test lead / lag relationships between I(1) time series variables (i.e stock prices) and advantages and ...
4
votes
0answers
109 views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
4
votes
0answers
240 views

Is the OLS estimator the UMVUE (assuming Normality)?

Suppose $$ \mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e} \, , \\ \mathbf{e} \sim \mathcal{N}(0,\mathbf{I}_P) \, . $$ We know that $\mathbf{\hat{b}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \...
4
votes
0answers
158 views

Choosing the basis functions in a linear regression

I have two random variables $X$ and $Y$ and I'm trying to model $\mathbb{E}[Y|X]$. To this end, I'd like to pick a collection of functions $f_1, f_2 \dots f_n : \mathbb{R} \to \mathbb{R}$ and then ...
4
votes
0answers
60 views

Only minimizers of quadratic penalized least squares can be linear

Question Define the matrix $A \in \mathbb{R}^{m \times n}$, the vector $b \in \mathbb{R}^m$, and the function $\mathrm{pen}: \mathbb{R}^n \to \mathbb{R}$. Then, put the minimizer $$\hat{x} = \arg\...
4
votes
0answers
176 views

Modern approaches to nonlinear regression which are available in R

I would like to fit a complex nonlinear regression model: basically, I have a complex computer code which has an input vector $\mathbf{x}$, a vector of calibration parameters $\boldsymbol{\theta}$ and ...
4
votes
0answers
490 views

MSE of an individual coefficient from ridge or lasso vs. OLS

Consider a multiple regression model $$ y = X\beta + \varepsilon $$ with $K$ regressors in $X$. If the model is correctly specified, the OLS estimator $\hat\beta_{OLS}=(X'X)^{-1}X'y$ will be the ...
4
votes
0answers
155 views

Sign and size of OLS bias for Tobit models

I have a question related to the sign and size of the OLS bias in the case of a Tobit model. Consider the following model (1) Sample of observations $\{X_i,Y_i\}_{i=1}^n$, i.i.d., $X_i$ is a vector ...
4
votes
0answers
116 views

Best-fit plane through a set of lines?

A linear model estimates the best fit line from a set of points, often through minimization of the sum of squared residuals. By analogy, is there any established method (possibly implemented in R) ...
4
votes
0answers
108 views

Total least square intuition

I have yet to find a good intuitive explanation of TLS. Online resources tend to focus on the vertical vs. perpendicular square error pictures (I don't need to see perpendicular lines to understand ...
4
votes
0answers
109 views

Derivation of Olsens LS Selectivity Correction

There are many estimation procedures that correct for sample selection. The most famous is Heckman's two-step selectivity correction (in two equations) that assumes bivariate normality of the error ...
4
votes
0answers
122 views

Adjust linear regression penalty for under/over-estimations

Basically I have a case where under-predictions are worse than over-predictions. Is there a way to penalize the linear regression model during training according to some predefined ratio? E.g. I want ...
4
votes
0answers
232 views

OLS standard error that corrects for autocorrelation but not heteroskedasticity

Question: By mapping the OLS regression into the GMM framework, write the formula for the standard error of the OLS regression coefficients that corrects for autocorrelation but not heteroskedasticity....
4
votes
0answers
205 views

Techniques for scaling data matrix to avoid rank deficiency issues

I have a $n \times p$ matrix $A$ where $n$ is the number of observables and $p$ is the number of observations. $n \gg p$ In my code, I have done $[E,V] \,=\, eig(A)$ and doing a least squares ...
4
votes
0answers
265 views

Regression model with heteroskedasticity in both variables

I've been learning (lurking) from this site for a while and I finally have a question I haven't seen answered yet. I'm doing a flight test and trying to fit the resulting data to linear line. From a ...
4
votes
0answers
840 views

Relationship between regularized least squares and MLE

We know that the least square method is equivalent to the MLE for Gaussian distributed errors. What is the relationship (if any) between regularized (Tichonov regularization) least squares and MLE?
4
votes
0answers
286 views

Estimating parameters of an unknown PID controller

Say that I have your standard PID controller at work. To keep it extremely simple imagine I have a target $x^*$ on the variable $x$. Then the controller is: $y(t) = K_p ( x^* - x_t) + K_i \int_0^t (x^...
3
votes
0answers
36 views

Dealing with autocorrelation using Generalized Least Squares

I have a time series data set where the auto correlation of the residuals follow an exponential decay. I was wondering how I should deal with this? I would like to fit a linear model and have tried ...
3
votes
0answers
32 views

Intepretting Linear Mixed Effects Model Output

I'm trying to reconcile the output of this MixedLM Model Output with my knowledge of OLS model outputs, e.g., MixedLM has Z-Stats vs. T-Stats for OLS. Model code for reference, but this is not my ...
3
votes
0answers
35 views

Direction of Bias in OLS with systematic measurement error

I am looking at a study that wants to measure the effect of $x_t$ on $Y_t$, but the true values of the $x_t$ are not observed. Instead, what is observed is a minimum value for $x_t$. In effect, what ...
3
votes
0answers
76 views

Can “non-parametric” tests be achieved with generalized linear models?

I recently read @Kodiologist's answer to a post here looking for clarification on the relation between GLMs and non-parametric tests. His answer is along the lines of "the approach is not non-...
3
votes
0answers
320 views

OLS standard errors vs GLS standard errors

I currently have a dataset with y and x that seems to experience heteroskedasticity. Are the results consistent/within expectations? (The standard error for x from my GLS is even higher than that of ...
3
votes
0answers
49 views

Rotation changes correlation - correction from OLS

Let $X,Y$ be real random variables with finite variances, and with no loss of generality assume $\mathbf{E}[X] = \mathbf{E}[Y]=0$. For simplicity, I will focus on the case $\mathrm{Var}X \neq \mathrm{...
3
votes
0answers
99 views

Linear regression with overlapping observations

Suppose we're doing univariate linear regression between X and Y. Let's say X are daily observations, and Y reflects how some variable changes 1 year into the future. So Y observations will be ...
3
votes
0answers
158 views

Efficiency of the OLS estimator

I have this linear regression: $y_{i}=\beta_{0} + \beta_{1}x_{i} + u_{i}$ with $i=\{1..n\}$. Say $\hat{\beta}_{1}$ is the OLS estimator of $\beta_{1}$. $\hat{\beta}_{1}$ is BLUE since the Gauss ...
3
votes
0answers
44 views

Does the residuals distrubution change the result of least-squares estimation?

Suppose I have the model: $\mathbf{y} = f(\mathbf{x}, \mathbf{\theta})+\mathbf{e}$ where $f$ is a nonlinear function, $\theta$ a parameter vector, $\mathbf{x}$ the independent variable vector and $\...
3
votes
0answers
66 views

small sample approach to simple linear regression with errors-in-variables (measurement errors)

I seek to estimate $b_1$ and $b_0$ from data of the form: $$y_i = b_1x_i + b_0 + e_i, \quad i\in\{0,1,...,N-1\}$$ given $\{y_i\}$ and $\{\tilde{x}_i\}$ where $\tilde{x}_i=x_i + n_i$ (i.e., error-in-...
3
votes
0answers
68 views

Multiple multivariate regression problem - auto correlated dependent variables

I have a classic multivariate regression problem, i.e. dependent variables are stored in matrix $Y$ having dimension $n \times p$. So $p$ observations come from the same respondent $i$ and we need to ...
3
votes
0answers
51 views

When to use weighting in least squares?

Normally when one talks about weighted least squares, the end-goal is to weight each point by its variance. However my question pertains to models which have multiple components. It will be easier ...
3
votes
0answers
572 views

Why use quantile regression instead of splitting the data in quantiles and calculating multiple linear regressions?

Why use quantile regression instead of splitting the data in quantiles and calculating multiple linear regressions? What are the advantages and disadvantages of these methods? As far as I understand ...
3
votes
0answers
77 views

Linear model with biased estimator

Consider a linear regression model. Suppose that the estimator $\hat{\beta}$ for the vector of the parameters of the model $\beta$ is, for some reasons, biased. As a consequence: $$E[\hat{\...
3
votes
0answers
411 views

How to choose between different options in partial least square regression?

There seem to be several methods of performing partial least square regression. For example in pls pacakge in R, following are available: ...
3
votes
0answers
135 views

Prediction Intervals for Incremental OLS regression

I am implementing incremental OLS regression algorithm where the data points arrive one at a time. As the regression parameters are determined by the formula, $(X'X)^{-1} X'y$ and the Sherman-Morrison ...
3
votes
0answers
136 views

Can I use OLS with robust standard errors to analyze proportions?

I have a response variable that is a proportion. It is not the outcome of a series of Bernoulli trials, so there is no numerator/denominator, just the proportion. I'd like to assess the relationship ...
3
votes
0answers
2k views

choosing $β_0$ and $β_1$ to minimize the residual sum of squares

I'm reading a book called An Introduction to Statistical Learning: with Applications in R, and I have a question in regards to the material inside. I understand that we can find the residual sum of ...
3
votes
0answers
385 views

How to approximate error on Chi-Squared Fit when bin counts are zero

I am using a galaxy image simulator that provides a 2D histogram that has the number of photons per pixel (bin) $N$. I am currently using a least-squares residual: $\sum_{bin}(f_{data}-f_{model})^2$ ...
3
votes
0answers
425 views

Which observation has the largest variance of the residual?

a) For which of the following observations (obs1, obs2, obs3) is the variance of the residual the largest? Which observation has the highest leverage? And which one the smallest? Explain why. b) ...
3
votes
0answers
324 views

Tobit or OLS question

My dependent variable is mostly continuous and positive, but has a modest number of zeros (10% of the sample). The results from tobit and ols are very similar. How can I formally compare the tobit and ...
3
votes
0answers
42 views

What theory of estimation does apply when estimation follows model selection?

Almost two decades ago, Chatfield JRSS(1995)[vol.158,p.441] wrote that ...
3
votes
0answers
71 views

Reducing the dimension of an embedding

Let $O \in \mathbb R^{p\times m}$ be a data matrix of observations. Suppose we are given a model $\mu : \mathbb R^n \rightarrow \mathbb R^m$ which is able to approximately fit the observations. Fix $...
3
votes
0answers
182 views

Is is correct to compare t-statistics of different pairs of cointegrated timeseries?

I am testing for cointegration all the pairs from a set of 100 stocks. I run an Ordinary Least Square Regression on each pair and then I test for the existence of unit roots in the residuals. I am ...
3
votes
0answers
363 views

Standard errors in weighted least squares

My data is a panel of countries by year. Suppose my main RHS variable is a country's GDP and my main LHS variable classifies countries by whether the country is a democracy. Is it desirably to ...
3
votes
0answers
691 views

Pregibon test for linearity vs. Ramsey's RESET test

Does every Ordinary Least Squares (OLS) regression model have to pass both the Pregibon Test for Linearity (sometimes called link-test) and the Ramsey RESET tests? I am working on an OLS model and it ...
3
votes
0answers
4k views

What to do when ovtest and linktest in Stata suggest model misspecification?

I have a sample that consists of 50 observations. The base model of the OLS-Regression with three control variables, two of them significant, has a $R^2=0.50$ and its F-Value is 7. Both ...
2
votes
0answers
19 views

Show positive semi-definiteness of co-variance differences

I am having difficulty checking if $$ (A'D^{-1}A)(A'D^{-1}BD^{-1}A)^{-1}(A'D^{-1}A) - (A'A)(A'BA)^{-1}(A'A) \succeq 0 $$ where $B\succ 0$ is a square co-variance matrix and $D$ is a diagonal matrix ...
2
votes
0answers
58 views

Negative values for OLS variance

I am currently writing some code which performs regression and have noticed that when I calculate variance of $c\hat{\beta}$ I am sometimes on some datasets getting negative values. The variance is ...